Running the webhook and Vault on different clusters

This section describes how to configure the webhook and Vault when the webhook runs on a different cluster from Vault, or if Vault runs outside Kubernetes.

Let’s suppose you have two different K8S clusters:

  • cluster1 contains vault-operator
  • cluster2 contains vault-secrets-webhook

Basically, you have to grant cluster2 access to the Vault running on cluster1. To achieve this, complete the following steps.

  1. Extract the cluster.certificate-authority-data and the cluster.server fields from your cluster2 kubeconfig file. You will need them in the externalConfig section of the cluster1 configuration. For example:

    kubectl config view -o yaml --minify=true --raw=true
  2. Decode the certificate from the cluster.certificate-authority-data field, for example::

    grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 --decode
  3. On cluster2, create a vault ServiceAccount and the vault-auth-delegator ClusterRoleBinding:

    kubectl kustomize | kubectl apply -f -

    Expected output:

    serviceaccount/vault created created created created created created

    You can use the vault ServiceAccount token as a token_reviewer_jwt in the auth configuration. To retrieve the token, run the following command:

    kubectl get secret $(kubectl get sa vault -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode
  4. In the custom resource (for example, in this sample CR) of cluster1, define an externalConfig section. Fill the values of the kubernetes_ca_cert, kubernetes_host, and token_reviewer_jwt using the data collected in the previous steps.

          - name: allow_secrets
            rules: path "secret/*" {
              capabilities = ["create", "read", "update", "delete", "list"]
          - type: kubernetes
              token_reviewer_jwt: <token-for-cluster2-service-account>
              kubernetes_ca_cert: |
                -----BEGIN CERTIFICATE-----
                -----END CERTIFICATE-----            
              kubernetes_host: <cluster.server-field-on-cluster2>
              # Allow every pod in the default namespace to use the secret kv store
              - name: default
                bound_service_account_names: ["default", "vault-secrets-webhook"]
                bound_service_account_namespaces: ["default", "vswh"]
                policies: allow_secrets
                ttl: 1h
  5. In a production environment, it is highly recommended to specify TLS config for your Vault ingress.

      # Request an Ingress controller with the default configuration
        # Specify Ingress object annotations here, if TLS is enabled (which is by default)
        # the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations
        # to support TLS backends
        # Override the default Ingress specification here
        # This follows the same format as the standard Kubernetes Ingress
        # See:
          - hosts:
            - vault-dns-name
            secretName: vault-ingress-tls-secret
  6. Deploy the Vault custom resource containing the externalConfig section to cluster1:

    kubectl apply -f your-proper-vault-cr.yaml
  7. After Vault started in cluster1, you can use the vault-secrets-webhook in cluster2 with the proper annotations. For example:

      replicas: 1

Authenticate the mutating-webhook with a cloud identity

You can use a cloud identity to authenticate the mutating-webhook against the external vault.

  1. Add your cloud authentication method in your external vault, for example, the Azure Auth Method.

  2. Configure your vault-secrets-webhook to use the this method. For example:

      VAULT_AUTH_METHOD: azure
      VAULT_PATH: azure
      VAULT_ROLE: default

For the VAULT_AUTH_METHOD env var, the following types are supported: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”