Running the webhook and Vault on different clusters

This section describes how to configure the webhook and Vault when the webhook runs on a different cluster from Vault, or if Vault runs outside Kubernetes.

Let’s suppose you have two different K8S clusters:

  • cluster1 contains vault-operator
  • cluster2 contains vault-secrets-webhook

Basically, you have to grant cluster2 access to the Vault running on cluster1. To achieve this, complete the following steps.

  1. Extract the cluster.certificate-authority-data and the cluster.server fields from your cluster2 kubeconfig file. You will need them in the externalConfig section of the cluster1 configuration. For example:

    kubectl config view -o yaml --minify=true --raw=true
    
  2. Decode the certificate from the cluster.certificate-authority-data field, for example::

    grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 --decode
    
  3. On cluster2, create a vault ServiceAccount and the vault-auth-delegator ClusterRoleBinding:

    kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
        

    Expected output:

    serviceaccount/vault created
        role.rbac.authorization.k8s.io/vault created
        role.rbac.authorization.k8s.io/leader-election-role created
        rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
        rolebinding.rbac.authorization.k8s.io/vault created
        clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
        

    You can use the vault ServiceAccount token as a token_reviewer_jwt in the auth configuration. To retrieve the token, run the following command:

    kubectl get secret $(kubectl get sa vault -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode
    
  4. In the vault.banzaicloud.com custom resource (for example, in this sample CR) of cluster1, define an externalConfig section. Fill the values of the kubernetes_ca_cert, kubernetes_host, and token_reviewer_jwt using the data collected in the previous steps.

      externalConfig:
        policies:
          - name: allow_secrets
            rules: path "secret/*" {
              capabilities = ["create", "read", "update", "delete", "list"]
              }
        auth:
          - type: kubernetes
            config:
              token_reviewer_jwt: <token-for-cluster2-service-account>
              kubernetes_ca_cert: |
                -----BEGIN CERTIFICATE-----
                <certificate-from-certificate-authority-data-on-cluster2>
                -----END CERTIFICATE-----            
              kubernetes_host: <cluster.server-field-on-cluster2>
            roles:
              # Allow every pod in the default namespace to use the secret kv store
              - name: default
                bound_service_account_names: ["default", "vault-secrets-webhook"]
                bound_service_account_namespaces: ["default", "vswh"]
                policies: allow_secrets
                ttl: 1h
    
  5. In a production environment, it is highly recommended to specify TLS config for your Vault ingress.

      # Request an Ingress controller with the default configuration
      ingress:
        # Specify Ingress object annotations here, if TLS is enabled (which is by default)
        # the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations
        # to support TLS backends
        annotations:
        # Override the default Ingress specification here
        # This follows the same format as the standard Kubernetes Ingress
        # See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#ingressspec-v1beta1-extensions
        spec:
          tls:
          - hosts:
            - vault-dns-name
            secretName: vault-ingress-tls-secret
    
  6. Deploy the Vault custom resource containing the externalConfig section to cluster1:

    kubectl apply -f your-proper-vault-cr.yaml
    
  7. After Vault started in cluster1, you can use the vault-secrets-webhook in cluster2 with the proper annotations. For example:

    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: hello-secrets
      template:
        metadata:
          labels:
            app.kubernetes.io/name: hello-secrets
          annotations:
            vault.security.banzaicloud.io/vault-addr: "https://vault-dns-name:443"
            vault.security.banzaicloud.io/vault-role: "default"
            vault.security.banzaicloud.io/vault-skip-verify: "true"
            vault.security.banzaicloud.io/vault-path: "kubernetes"
    

Authenticate the mutating-webhook with a cloud identity

You can use a cloud identity to authenticate the mutating-webhook against the external vault.

  1. Add your cloud authentication method in your external vault, for example, the Azure Auth Method.

  2. Configure your vault-secrets-webhook to use the this method. For example:

    env:
      VAULT_ADDR: https://external-vault.example.com
      VAULT_AUTH_METHOD: azure
      VAULT_PATH: azure
      VAULT_ROLE: default
    

For the VAULT_AUTH_METHOD env var, the following types are supported: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”