Deploy the webhook

Deploy the mutating webhook

You can deploy the Vault Secrets Webhook using Helm. Note that:

• The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
• The deployed RBAC objects contain the necessary permissions fo running the webhook.

Prerequisites

• The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
• The chart requires Helm 3.
• To interact with Vault (for example, for testing), the Vault command line client must be installed on your computer.
• You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.

Deploy the webhook

1. Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:

   kubectl create namespace vault-infra
   kubectl label namespace vault-infra name=vault-infra
   

2. Deploy the vault-secrets-webhook chart. If you want to customize the Helm chart, see the list of vault-secrets-webhook Helm chart values.

   helm upgrade --install --wait vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook --namespace vault-infra
   

   Expected output:

   Release "vault-secrets-webhook" does not exist. Installing it now.
   NAME: vault-secrets-webhook
   LAST DEPLOYED: Fri Jul 14 15:42:36 2023
   NAMESPACE: vault-infra
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

   For further details, see the webhook’s Helm chart repository.

3. Check that the pods are running:

   kubectl get pods --namespace vault-infra
   

   Expected output:

   NAME                                     READY   STATUS    RESTARTS   AGE
   vault-secrets-webhook-58b97c8d6d-qfx8c   1/1     Running   0          22s
   vault-secrets-webhook-58b97c8d6d-rthgd   1/1     Running   0          22s
   

4. If you already have the Vault CLI installed, write a secret into Vault:

   vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
   

   Expected output:

   Key              Value
   ---              -----
   created_time     2020-11-04T11:39:01.863988395Z
   deletion_time    n/a
   destroyed        false
   version          1
   

5. Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:

   kubectl apply -f - <<"EOF"
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vault-test
   spec:
     replicas: 1
     selector:
       matchLabels:
         app.kubernetes.io/name: vault
     template:
       metadata:
         labels:
           app.kubernetes.io/name: vault
         annotations:
           vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200
           vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"
           vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate
           vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optional, the name of the Secret where the Vault CA cert is, if not defined it is not mounted
           vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication
           vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"
       spec:
         serviceAccountName: default
         containers:
         - name: alpine
           image: alpine
           command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
           env:
           - name: AWS_SECRET_ACCESS_KEY
             value: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
   EOF
   

   Expected output:

   deployment.apps/vault-test created
   

6. Check the mutated deployment.

   kubectl describe deployment vault-test
   

   The output should look similar to the following:

   Name:                   vault-test
   Namespace:              default
   CreationTimestamp:      Wed, 04 Nov 2020 12:44:18 +0100
   Labels:                 <none>
   Annotations:            deployment.kubernetes.io/revision: 1
   Selector:               app.kubernetes.io/name=vault
   Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
   StrategyType:           RollingUpdate
   MinReadySeconds:        0
   RollingUpdateStrategy:  25% max unavailable, 25% max surge
   Pod Template:
     Labels:           app.kubernetes.io/name=vault
     Annotations:      vault.security.banzaicloud.io/vault-addr: https://vault:8200
                       vault.security.banzaicloud.io/vault-agent: false
                       vault.security.banzaicloud.io/vault-path: kubernetes
                       vault.security.banzaicloud.io/vault-role: default
                       vault.security.banzaicloud.io/vault-skip-verify: false
                       vault.security.banzaicloud.io/vault-tls-secret: vault-tls
     Service Account:  default
     Containers:
      alpine:
       Image:      alpine
       Port:       <none>
       Host Port:  <none>
       Command:
         sh
         -c
         echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000
       Environment:
         AWS_SECRET_ACCESS_KEY:  vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
       Mounts:                   <none>
     Volumes:                    <none>
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   OldReplicaSets:  <none>
   NewReplicaSet:   vault-test-55c569f9 (1/1 replicas created)
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  29s   deployment-controller  Scaled up replica set vault-test-55c569f9 to 1
   

   As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.

Deploy the webhook from a private registry

If you are getting the x509: certificate signed by unknown authority app=vault-secrets-webhook error when the webhook is trying to download the manifest from a private image registry, you can:

• Build a docker image where the CA store of the OS layer of the image contains the CA certificate of the registry.
• Alternatively, you can disable certificate verification for the registry by using the REGISTRY_SKIP_VERIFY=“true” environment variable in the deployment of the webhook.

Deploy in daemon mode

vault-env by default replaces itself with the original process of the Pod after reading the secrets from Vault, but with the vault.security.banzaicloud.io/vault-env-daemon: "true" annotation this behavior can be changed. So vault-env can change to daemon mode, so vault-env starts the original process as a child process and remains in memory, and renews the lease of the requested Vault token and of the dynamic secrets (if requested any) until their final expiration time.

You can find a full example using MySQL dynamic secrets in the Bank-Vaults project’s Vault Operator repository:

# Deploy MySQL first as the Vault storage backend and our application will request dynamic secrets for this database as well:
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=your-root-password --set mysqlDatabase=vault --set mysqlUser=vault --set mysqlPassword=secret --set 'initializationFiles.app-db\.sql=CREATE DATABASE IF NOT EXISTS app;'

# Deploy the vault-operator and the vault-secrets-webhook
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
helm upgrade --namespace vault-infra --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
helm upgrade --namespace vault-infra --install vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook

# Create a Vault instance with MySQL storage and a configured dynamic database secrets backend
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-mysql-ha.yaml

# Deploy the example application requesting dynamic database credentials from the above Vault instance
kubectl apply -f deploy/test-dynamic-env-vars.yaml
kubectl logs -f deployment/hello-secrets