This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
Secret injection webhook
How the webhook works - overview
Kubernetes secrets are the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption eventually ends up as a Kubernetes secret. However, despite their name, Kubernetes secrets are not secure, since they are only base64 encoded.
The secret injection webhook of Bank-Vaults is a mutating webhook that bypasses the Kubernetes secrets mechanism and injects the secrets retrieved from Vault directly into the Pods. Specifically, the mutating admission webhook injects (in a very non-intrusive way) an executable into containers of Deployments and StatefulSets. This executable can request secrets from Vault through special environment variable definitions.
An important and unique aspect of the webhook is that it is a daemonless solution (although if you need it, you can deploy the webhook in daemon mode as well).
Why is this more secure than using Kubernetes secrets or any other custom sidecar container?
Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk or in etcd - not even temporarily. All secrets are stored in memory, and are only visible to the process that requested them. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.
If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.
The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
# or
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-key-secret
key: AWS_SECRET_ACCESS_KEY
# or
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
configMapKeyRef:
name: aws-key-configmap
key: AWS_SECRET_ACCESS_KEY
The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:
envFrom:
- secretRef:
name: aws-key-secret
# or
- configMapRef:
name: aws-key-configmap
Secret and ConfigMap examples
Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them.
The secret value should contain a base64 encoded template string referencing the vault path you want to insert.
Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.
apiVersion: v1
kind: Secret
metadata:
name: aws-key-secret
data:
AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ==
type: Opaque
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-key-configmap
data:
AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
For further examples and use cases, see Configuration examples and scenarios.
1 - Deploy the webhook
Deploy the mutating webhook
You can deploy the Vault Secrets Webhook using Helm. Note that:
- The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
- The deployed RBAC objects contain the necessary permissions fo running the webhook.
Prerequisites
- The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
- The chart requires Helm 3.
- To interact with Vault (for example, for testing), the Vault command line client must be installed on your computer.
- You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.
Deploy the webhook
-
Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
-
Deploy the vault-secrets-webhook chart. If you want to customize the Helm chart, see the list of vault-secrets-webhook Helm chart values.
helm upgrade --install --wait vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook --namespace vault-infra
Expected output:
Release "vault-secrets-webhook" does not exist. Installing it now.
NAME: vault-secrets-webhook
LAST DEPLOYED: Fri Jul 14 15:42:36 2023
NAMESPACE: vault-infra
STATUS: deployed
REVISION: 1
TEST SUITE: None
For further details, see the webhook’s Helm chart repository.
-
Check that the pods are running:
kubectl get pods --namespace vault-infra
Expected output:
NAME READY STATUS RESTARTS AGE
vault-secrets-webhook-58b97c8d6d-qfx8c 1/1 Running 0 22s
vault-secrets-webhook-58b97c8d6d-rthgd 1/1 Running 0 22s
-
If you already have the Vault CLI installed, write a secret into Vault:
vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
Expected output:
Key Value
--- -----
created_time 2020-11-04T11:39:01.863988395Z
deletion_time n/a
destroyed false
version 1
-
Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:
kubectl apply -f - <<"EOF"
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-test
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: vault
template:
metadata:
labels:
app.kubernetes.io/name: vault
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200
vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"
vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optional, the name of the Secret where the Vault CA cert is, if not defined it is not mounted
vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication
vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"
spec:
serviceAccountName: default
containers:
- name: alpine
image: alpine
command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
EOF
Expected output:
deployment.apps/vault-test created
-
Check the mutated deployment.
kubectl describe deployment vault-test
The output should look similar to the following:
Name: vault-test
Namespace: default
CreationTimestamp: Wed, 04 Nov 2020 12:44:18 +0100
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
Selector: app.kubernetes.io/name=vault
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/name=vault
Annotations: vault.security.banzaicloud.io/vault-addr: https://vault:8200
vault.security.banzaicloud.io/vault-agent: false
vault.security.banzaicloud.io/vault-path: kubernetes
vault.security.banzaicloud.io/vault-role: default
vault.security.banzaicloud.io/vault-skip-verify: false
vault.security.banzaicloud.io/vault-tls-secret: vault-tls
Service Account: default
Containers:
alpine:
Image: alpine
Port: <none>
Host Port: <none>
Command:
sh
-c
echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000
Environment:
AWS_SECRET_ACCESS_KEY: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: vault-test-55c569f9 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 29s deployment-controller Scaled up replica set vault-test-55c569f9 to 1
As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.
Deploy the webhook from a private registry
If you are getting the x509: certificate signed by unknown authority app=vault-secrets-webhook error when the webhook is trying to download the manifest from a private image registry, you can:
- Build a docker image where the CA store of the OS layer of the image contains the CA certificate of the registry.
- Alternatively, you can disable certificate verification for the registry by using the REGISTRY_SKIP_VERIFY=“true” environment variable in the deployment of the webhook.
Deploy in daemon mode
vault-env by default replaces itself with the original process of the Pod after reading the secrets from Vault, but with the vault.security.banzaicloud.io/vault-env-daemon: "true" annotation this behavior can be changed. So vault-env can change to daemon mode, so vault-env starts the original process as a child process and remains in memory, and renews the lease of the requested Vault token and of the dynamic secrets (if requested any) until their final expiration time.
You can find a full example using MySQL dynamic secrets in the Bank-Vaults project’s Vault Operator repository:
# Deploy MySQL first as the Vault storage backend and our application will request dynamic secrets for this database as well:
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=your-root-password --set mysqlDatabase=vault --set mysqlUser=vault --set mysqlPassword=secret --set 'initializationFiles.app-db\.sql=CREATE DATABASE IF NOT EXISTS app;'
# Deploy the vault-operator and the vault-secrets-webhook
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
helm upgrade --namespace vault-infra --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
helm upgrade --namespace vault-infra --install vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook
# Create a Vault instance with MySQL storage and a configured dynamic database secrets backend
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-mysql-ha.yaml
# Deploy the example application requesting dynamic database credentials from the above Vault instance
kubectl apply -f deploy/test-dynamic-env-vars.yaml
kubectl logs -f deployment/hello-secrets
2 - Configuration examples and scenarios
The following examples show you how to configure the mutating webhook to best suit your environment.
The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
# or
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-key-secret
key: AWS_SECRET_ACCESS_KEY
# or
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
configMapKeyRef:
name: aws-key-configmap
key: AWS_SECRET_ACCESS_KEY
The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:
envFrom:
- secretRef:
name: aws-key-secret
# or
- configMapRef:
name: aws-key-configmap
Secret and ConfigMap examples
Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them.
The secret value should contain a base64 encoded template string referencing the vault path you want to insert.
Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.
apiVersion: v1
kind: Secret
metadata:
name: aws-key-secret
data:
AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ==
type: Opaque
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-key-configmap
data:
AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
Prerequisites for inline injection to work
Vault needs to be properly configured for mutation to function; namely
externalConfig.auth and externConfig.roles (from the perspective of the
vault operator CR) need to be properly configured. If you’re not using the vault
operator then you must make sure that your Vault configuration for Kubernetes
auth methods are properly configured. This configuration is outside the scope
of this document. If you use the operator for managing Vault in your cluster, see the Vault operator documentation.
Inject secret into resources
The webhook can inject into any kind of resources, even into CRDs, for example:
apiVersion: mysql.example.github.com/v1
kind: MySQLCluster
metadata:
name: "my-cluster"
spec:
caBundle: "vault:pki/cert/43138323834372136778363829719919055910246657114#ca"
Inline mutation
The webhook also supports inline mutation when your secret needs to be replaced somewhere inside a string.
apiVersion: v1
kind: Secret
metadata:
name: aws-key-secret
data:
config.yaml: >
foo: bar
secret: ${vault:secret/data/mysecret#supersecret}
type: Opaque
This works also for ConfigMap resources when configMapMutation: true is set in the webhook’s Helm chart.
You can specify the version of the injected Vault secret as well in the special reference, the format is: vault:PATH#KEY_OR_TEMPLATE#VERSION
Example:
env:
- name: AWS_SECRET_ACCESS_KEY
value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY#2
Define multiple inline-secrets in resources
You can also inject multiple secrets under the same key in a Secret/ConfigMap/Object. This means that you can use multiple Vault paths in a value, for example:
apiVersion: v1
kind: ConfigMap
metadata:
name: sample-configmap
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault.default:8200"
vault.security.banzaicloud.io/vault-role: "default"
vault.security.banzaicloud.io/vault-tls-secret: vault-tls
vault.security.banzaicloud.io/vault-path: "kubernetes"
data:
aws-access-key-id: "vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID"
aws-access-template: "vault:secret/data/accounts/aws#AWS key in base64: ${.AWS_ACCESS_KEY_ID | b64enc}"
aws-access-inline: "AWS_ACCESS_KEY_ID: ${vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY}"
This example also shows how a CA certificate (created by the operator) can be used with the vault.security.banzaicloud.io/vault-tls-secret: vault-tls annotation to validate the TLS connection in case of a non-Pod resource.
Request a Vault token
There is a special vault:login reference format to request a working Vault token into an environment variable to be later consumed by your application:
env:
- name: VAULT_TOKEN
value: vault:login
Read a value from Vault
Values starting with "vault:" issue a read (HTTP GET) request towards the Vault API, this can be also used to request a dynamic database username/password pair for MySQL:
NOTE: This feature takes advantage of secret caching since we need to access the my-role endpoint twice, but in the background, it is written only once in Vault:
env:
- name: MYSQL_USERNAME
value: "vault:database/creds/my-role#username"
- name: MYSQL_PASSWORD
value: "vault:database/creds/my-role#password"
- name: REDIS_URI
value: "redis://${vault:database/creds/my-role#username}:${vault:database/creds/my-role#password}@127.0.0.1:6739"
Write a value into Vault
Values starting with ">>vault:" issue a write (HTTP POST/PUT) request towards the Vault API, some secret engine APIs should be written instead of reading from like the Password Generator for HashiCorp Vault:
env:
- name: MY_SECRET_PASSWORD
value: ">>vault:gen/password#value"
Or with Transit Secret Engine which is a fairly complex example since we are using templates when rendering the response and send data in the write request as well, the format is: vault:PATH#KEY_OR_TEMPLATE#DATA
Example:
env:
- name: MY_SECRET_PASSWORD
value: ">>vault:transit/decrypt/mykey#${.plaintext | b64dec}#{"ciphertext":"vault:v1:/DupSiSbX/ATkGmKAmhqD0tvukByrx6gmps7dVI="}"
Templating in values
Templating is also supported on the secret sourced from Vault (in the key part, after the first #), in the very same fashion as in the Vault configuration and external configuration with all the Sprig functions (this is supported only for Pods right now):
env:
- name: DOCKER_USERNAME
value: "vault:secret/data/accounts/dockerhub#My username on DockerHub is: ${title .DOCKER_USERNAME}"
In this case, an init-container will be injected into the given Pod. This container copies the vault-env binary into an in-memory volume and mounts that Volume to every container which has an environment variable definition like that. It also changes the command of the container to run vault-env instead of your application directly. When vault-env starts up, it connects to Vault to checks the environment variables. (By default, vault-env uses the Kubernetes Auth method, but you can also configure other authentication methods for the webhook.) The variables that have a reference to a value stored in Vault (vault:secret/....) are replaced with that value read from the Secret backend. After this, vault-env immediately executes (with syscall.Exec()) your process with the given arguments, replacing itself with that process (in non-daemon mode).
With this solution none of your Secrets stored in Vault will ever land in Kubernetes Secrets, thus in etcd.
vault-env was designed to work in Kubernetes in the first place, but nothing stops you to use it outside Kubernetes as well. It can be configured with the standard Vault client’s environment variables (because there is a standard Go Vault client underneath).
Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, so it requests a Vault token based on the Service Account of the container it is injected into.
Kubernetes 1.12 introduced a feature called APIServer dry-run which became beta as of 1.13. This feature requires some changes in webhooks with side effects. Vault mutating admission webhook is dry-run aware.
Mutate data from Vault and replace it in Kubernetes Secret
You can mutate Secrets (and ConfigMaps) as well if you set annotations and define proper Vault path in the data section:
apiVersion: v1
kind: Secret
metadata:
name: sample-secret
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
vault.security.banzaicloud.io/vault-role: "default"
vault.security.banzaicloud.io/vault-skip-verify: "true"
vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvY2tlcnJlcG8vI0RPQ0tFUl9SRVBPX1VTRVIiLCJwYXNzd29yZCI6InZhdWx0OnNlY3JldC9kYXRhL2RvY2tlcnJlcG8vI0RPQ0tFUl9SRVBPX1BBU1NXT1JEIiwiYXV0aCI6ImRtRjFiSFE2YzJWamNtVjBMMlJoZEdFdlpHOWphMlZ5Y21Wd2J5OGpSRTlEUzBWU1gxSkZVRTlmVlZORlVqcDJZWFZzZERwelpXTnlaWFF2WkdGMFlTOWtiMk5yWlhKeVpYQnZMeU5FVDBOTFJWSmZVa1ZRVDE5UVFWTlRWMDlTUkE9PSJ9fX0=
In the example above the secret type is kubernetes.io/dockerconfigjson and the webhook can get credentials from Vault.
The base64 encoded data contain vault path in case of username and password for docker repository and you can create it with commands:
kubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"
Use charts without explicit container.command and container.args
The Webhook can determine the container’s ENTRYPOINT and CMD with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication) you don’t need to do anything, but if the registry requires authentication the credentials have to be available in the Pod’s imagePullSecrets section.
Some examples (apply cr.yaml from the operator samples first):
helm upgrade --install mysql stable/mysql \
--set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
--set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
--set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls
Registry access
You can also specify a default secret being used by the webhook for cases where a pod has no imagePullSecrets specified. To make this work you have to set the environment variables DEFAULT_IMAGE_PULL_SECRET and DEFAULT_IMAGE_PULL_SECRET_NAMESPACE when deploying the vault-secrets-webhook. Have a look at the values.yaml of the
vault-secrets-webhook helm chart to see how this is done.
Note:
- If your EC2 nodes have the ECR instance role, the webhook can request an ECR access token through that role automatically, instead of an explicit
imagePullSecret - If your workload is running on GCP nodes, the webhook automatically authenticates to GCR.
Using a private image repository
# Docker Hub
kubectl create secret docker-registry dockerhub --docker-username=${DOCKER_USERNAME} --docker-password=$DOCKER_PASSWORD
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=dockerhub" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="private-repo/mysql"
# GCR
kubectl create secret docker-registry gcr \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat ~/json-key-file.json)"
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=gcr" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="gcr.io/your-repo/mysql"
# ECR
TOKEN=`aws ecr --region=eu-west-1 get-authorization-token --output text --query authorizationData[].authorizationToken | base64 --decode | cut -d: -f2`
kubectl create secret docker-registry ecr \
--docker-server=https://171832738826.dkr.ecr.eu-west-1.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}"
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=ecr" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="171832738826.dkr.ecr.eu-west-1.amazonaws.com/mysql" --set-string imageTag=5.7
Mount all keys from Vault secret to env
This feature is very similar to Kubernetes’ standard envFrom: construct, but instead of a Kubernetes Secret/ConfigMap, all its keys are mounted from a Vault secret using the webhook and vault-env.
You can set the Vault secret to mount using the vault.security.banzaicloud.io/vault-env-from-path annotation.
Compared to the original environment variable definition in the Pod env construct, the only difference is that you won’t see the actual environment variables in the definition, because they are dynamic, and are based on the contents of the Vault secret’s, just like envFrom:.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-secrets
spec:
selector:
matchLabels:
app.kubernetes.io/name: hello-secrets
template:
metadata:
labels:
app.kubernetes.io/name: hello-secrets
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: vault-tls
vault.security.banzaicloud.io/vault-env-from-path: "secret/data/accounts/aws"
spec:
initContainers:
- name: init-ubuntu
image: ubuntu
command: ["sh", "-c", "echo AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID && echo initContainers ready"]
containers:
- name: alpine
image: alpine
command: ["sh", "-c", "echo AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
Since vault-env v1.21.1 (which is the default since vault-secrets-webhook v1.21.0) you can specify the version of
the injected Vault secret as well, the format is: PATH#VERSION
Example:
annotations:
vault.security.banzaicloud.io/vault-env-from-path: "secret/data/accounts/aws#1"
Authenticate the webhook to Vault
By default, the webhook uses Kubernetes ServiceAccount-based authentication in Vault. Use the vault.security.banzaicloud.io/vault-auth-method annotation to request different authentication types from the following supported types: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”.
Note: GCP IAM authentication (gcp-iam) only allows for authentication with the ‘default’ service account of the caller, and a new token is generated at every request.
Note: Azure MSI authentication (azure) a new token is generated at every request.
The following deployment - if running on a GCP instance - will automatically receive a signed JWT token from the metadata server of the cloud provider, and use it to authenticate against Vault. The same goes for vault-auth-method: "aws-ec2", when running on an EC2 node with the right instance-role.
apiVersion: apps/v1
kind: Deployment
metadata:
name: vault-env-gcp-auth
spec:
selector:
matchLabels:
app.kubernetes.io/name: vault-env-gcp-auth
template:
metadata:
labels:
app.kubernetes.io/name: vault-env-gcp-auth
annotations:
# These annotations enable Vault GCP GCE auth, see:
# https://developer.hashicorp.com/vault/docs/auth/gcp#gce-login
vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: vault-tls
vault.security.banzaicloud.io/vault-role: "my-role"
vault.security.banzaicloud.io/vault-path: "gcp"
vault.security.banzaicloud.io/vault-auth-method: "gcp-gce"
spec:
containers:
- name: alpine
image: alpine
command:
- "sh"
- "-c"
- "echo $MYSQL_PASSWORD && echo going to sleep... && sleep 10000"
env:
- name: MYSQL_PASSWORD
value: vault:secret/data/mysql#MYSQL_PASSWORD
4 - Using Vault Agent Templating in the mutating webhook
With Bank-Vaults you can use Vault Agent to handle secrets that expire, and supply them to applications that read their configurations from a file.
When to use vault-agent
- You have an application or tool that requires to read its configuration from a file.
- You wish to have secrets that have a TTL and expire.
- You have no issues with running your application with a sidecar.
Note: If you need to revoke tokens, or use additional secret backends, see Using consul-template in the mutating webhook.
Workflow
- Your pod starts up, the webhook will inject one container into the pods lifecycle.
- The sidecar container is running Vault, using the vault agent that accesses Vault using the configuration specified inside a configmap and writes a configuration file based on a pre configured template (written inside the same configmap) onto a temporary file system which your application can use.
Prerequisites
This document assumes the following.
-
You have a working Kubernetes cluster which has:
-
You have a working knowledge of Kubernetes.
-
You can apply Deployments or PodSpec’s to the cluster.
-
You can change the configuration of the mutating webhook.
Use Vault TTLs
If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Vault Agent to execute a command when it writes a new configuration file using the command attribute. The following is a basic example which uses the Kubernetes authentication method.
Configuration
To configure the webhook, you can either:
Enable vault agent in the webhook
For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-agent-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Vault Agent.
Defaults via environment variables
| Variable | Default | Explanation |
| VAULT_IMAGE | hashicorp/vault:latest | The vault image to use for the sidecar container |
| VAULT_IMAGE_PULL_POLICY | IfNotPresent | The pull policy for the vault agent container |
| VAULT_ADDR | https://127.0.0.1:8200 | Kubernetes service Vault endpoint URL |
| VAULT_TLS_SECRET | "" | Supply a secret with the vault TLS CA so TLS can be verified |
| VAULT_AGENT_SHARE_PROCESS_NAMESPACE | Kubernetes version <1.12 default off, 1.12 or higher default on | ShareProcessNamespace override |
PodSpec annotations
| Annotation | Default | Explanation |
| vault.security.banzaicloud.io/vault-addr | Same as VAULT_ADDR above | "" |
| vault.security.banzaicloud.io/vault-tls-secret | Same as VAULT_TLS_SECRET above | "" |
| vault.security.banzaicloud.io/vault-agent-configmap | "" | A configmap name which holds the vault agent configuration |
| vault.security.banzaicloud.io/vault-agent-once | False | Do not run vault-agent in daemon mode, useful for kubernetes jobs |
| vault.security.banzaicloud.io/vault-agent-share-process-namespace | Same as VAULT_AGENT_SHARE_PROCESS_NAMESPACE above | "" |
| vault.security.banzaicloud.io/vault-agent-cpu | 100m | Specify the vault-agent container CPU resource limit |
| vault.security.banzaicloud.io/vault-agent-memory | 128Mi | Specify the vault-agent container memory resource limit |
| vault.security.banzaicloud.io/vault-agent-cpu-request | 100m | Specify the vault-agent container CPU resource request |
| vault.security.banzaicloud.io/vault-agent-cpu-limit | 100m | Specify the vault-agent container CPU resource limit (Overridden by vault-agent-cpu) |
| vault.security.banzaicloud.io/vault-agent-memory-request | 128Mi | Specify the vault-agent container memory resource request |
| vault.security.banzaicloud.io/vault-agent-memory-limit | 128Mi | Specify the vault-agent container memory resource limit (Overridden by vault-agent-cpu) |
| vault.security.banzaicloud.io/vault-configfile-path | /vault/secrets | Mount path of Vault Agent rendered files |
5 - Using consul-template in the mutating webhook
With Bank-Vaults you can use Consul Template as an addition to vault-env to handle secrets that expire, and supply them to applications that read their configurations from a file.
When to use consul-template
- You have an application or tool that must read its configuration from a file.
- You wish to have secrets that have a TTL and expire.
- You do not wish to be limited on which vault secrets backend you use.
- You can also expire tokens/revoke tokens (to do this you need to have a ready/live probe that can send a HUP to consul-template when the current details fail).
Workflow
The following shows the general workflow for using Consul Template:
- Your pod starts up. The webhook injects an init container (running vault agent) and a sidecar container (running consul-template) into the pods lifecycle.
- The vault agent in the init container logs in to Vault and retrieves a Vault token based on the configured VAULT_ROLE and Kubernetes Service Account.
- The consul-template running in the sidecar container logs in to Vault using the Vault token and writes a configuration file based on a pre-configured template in a configmap onto a temporary file system which your application can use.
Prerequisites
This document assumes the following.
-
You have a working Kubernetes cluster which has:
-
You have a working knowledge of Kubernetes.
-
You can apply Deployments or PodSpec’s to the cluster.
-
You can change the configuration of the mutating webhook.
Use Vault TTLs
If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Consul Template to execute a command when it writes a new configuration file using the command attribute. The following is a basic example (adapted from here).
Configuration
To configure the webhook, you can either:
Enable Consul Template in the webhook
For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-ct-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Consul Template.
Defaults via environment variables
| Variable | default | Explanation |
| VAULT_IMAGE | hashicorp/vault:latest | the vault image to use for the init container |
| VAULT_ENV_IMAGE | ghcr.io/bank-vaults/vault-env:latest | the vault-env image to use |
| VAULT_CT_IMAGE | hashicorp/consul-template:0.32.0 | the consul template image to use |
| VAULT_ADDR | https://127.0.0.1:8200 | Kubernetes service Vault endpoint URL |
| VAULT_SKIP_VERIFY | “false” | should vault agent and consul template skip verifying TLS |
| VAULT_TLS_SECRET | "" | supply a secret with the vault TLS CA so TLS can be verified |
| VAULT_AGENT | “true” | enable the vault agent |
| VAULT_CT_SHARE_PROCESS_NAMESPACE | Kubernetes version <1.12 default off, 1.12 or higher default on | ShareProcessNamespace override |
PodSpec annotations
| Annotation | default | Explanation |
| vault.security.banzaicloud.io/vault-addr | Same as VAULT_ADDR above | |
| vault.security.banzaicloud.io/vault-role | default | The Vault role for Vault agent to use |
| vault.security.banzaicloud.io/vault-path | auth/<method type> | The mount path of the method |
| vault.security.banzaicloud.io/vault-skip-verify | Same as VAULT_SKIP_VERIFY above | |
| vault.security.banzaicloud.io/vault-tls-secret | Same as VAULT_TLS_SECRET above | |
| vault.security.banzaicloud.io/vault-agent | Same as VAULT_AGENT above | |
| vault.security.banzaicloud.io/vault-ct-configmap | "" | A configmap name which holds the consul template configuration |
| vault.security.banzaicloud.io/vault-ct-image | "" | Specify a custom image for consul template |
| vault.security.banzaicloud.io/vault-ct-once | false | do not run consul-template in daemon mode, useful for kubernetes jobs |
| vault.security.banzaicloud.io/vault-ct-pull-policy | IfNotPresent | the Pull policy for the consul template container |
| vault.security.banzaicloud.io/vault-ct-share-process-namespace | Same as VAULT_CT_SHARE_PROCESS_NAMESPACE above | |
| vault.security.banzaicloud.io/vault-ct-cpu | “100m” | Specify the consul-template container CPU resource limit |
| vault.security.banzaicloud.io/vault-ct-memory | “128Mi” | Specify the consul-template container memory resource limit |
| vault.security.banzaicloud.io/vault-ignore-missing-secrets | “false” | When enabled will only log warnings when Vault secrets are missing |
| vault.security.banzaicloud.io/vault-env-passthrough | "" | Comma seprated list of VAULT_* related environment variables to pass through to main process. E.g.VAULT_ADDR,VAULT_ROLE. |
| vault.security.banzaicloud.io/vault-ct-secrets-mount-path | “/vault/secret” | Mount path of Consul template rendered files |
8 - Injecting consul-template into the Prometheus operator for Vault metrics
To get vault metrics into Prometheus you need to log in to Vault to get access to a native Vault endpoint that provides the metrics.
Workflow
- The webhook injects
vault-agent as an init container, based on the Kubernetes Auth role configuration prometheus-operator-prometheus. - The vault-agent grabs a token with the policy of
prometheus-operator-prometheus. consul-template runs as a sidecar, and uses the token from the previous step to retrieve a new token using the Token Auth role prometheus-metrics which has the policy prometheus-metrics applied to it.- Prometheus can now use this second token to read the Vault Prometheus endpoint.
The trick here is that Prometheus runs with the SecurityContext UID of 1000 but the default consul-template image is running under the UID of 100. This is because of a Dockerfile Volume that is configured which dockerd mounts as 100 (/consul-template/data).
Subsequently using this consul-template means it will never start, so we need to ensure we do not use this declared volume and change the UID using a custom Dockerfile and entrypoint.
Prerequisites
This document assumes you have a working Kubernetes cluster which has a:
-
You have a working Kubernetes cluster which has:
-
You have the CoreOS Prometheus Operator installed and working.
-
You have a working knowledge of Kubernetes.
-
You can apply Deployments or PodSpec’s to the cluster.
-
You can change the configuration of the mutating webhook.
Configuration
Custom consul-template image; docker-entrypoint.sh
#!/bin/dumb-init /bin/sh
set -ex
# Note above that we run dumb-init as PID 1 in order to reap zombie processes
# as well as forward signals to all processes in its session. Normally, sh
# wouldn't do either of these functions so we'd leak zombies as well as do
# unclean termination of all our sub-processes.
# CONSUL_DATA_DIR is exposed as a volume for possible persistent storage.
# CT_CONFIG_DIR isn't exposed as a volume but you can compose additional config
# files in there if you use this image as a base, or use CT_LOCAL_CONFIG below.
CT_DATA_DIR=/consul-template/config
CT_CONFIG_DIR=/consul-template/config
# You can also set the CT_LOCAL_CONFIG environment variable to pass some
# Consul Template configuration JSON without having to bind any volumes.
if [ -n "$CT_LOCAL_CONFIG" ]; then
echo "$CT_LOCAL_CONFIG" > "$CT_CONFIG_DIR/local-config.hcl"
fi
# If the user is trying to run consul-template directly with some arguments, then
# pass them to consul-template.
if [ "${1:0:1}" = '-' ]; then
set -- /bin/consul-template "$@"
fi
# If we are running Consul, make sure it executes as the proper user.
if [ "$1" = '/bin/consul-template' ]; then
# Set the configuration directory
shift
set -- /bin/consul-template \
-config="$CT_CONFIG_DIR" \
"$@"
# Check the user we are running as
current_user="$(id -un)"
if [ "${current_user}" == "root" ]; then
# Run under the right user
set -- gosu consul-template "$@"
fi
fi
exec "$@"
Dockerfile
FROM hashicorp/consul-template:0.32.0
ADD build/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN apk --no-cache add shadow && \
usermod -u 1000 consul-template && \
chown -Rc consul-template:consul-template /consul-template/
USER consul-template:consul-template
ConfigMap
---
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/name: prometheus
prometheus: consul-template
name: prometheus-consul-template
data:
config.hcl: |
vault {
ssl {
ca_cert = "/vault/tls/ca.crt"
}
grace = "5m"
retry {
backoff = "1s"
}
}
template {
destination = "/vault/secrets/vault-token"
command = "/bin/sh -c '/usr/bin/curl -s http://127.0.0.1:9090/-/reload'"
contents = <<-EOH
{{with secret "/auth/token/create/prometheus-metrics" "policy=prometheus-metrics" }}{{.Auth.ClientToken}}{{ end }}
EOH
wait {
min = "2s"
max = "60s"
}
}
Vault CR snippets
Set the vault image to use:
---
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
name: "vault"
spec:
size: 2
image: hashicorp/vault:1.14.1
Our Vault config for telemetry:
# A YAML representation of a final vault config file.
# See https://developer.hashicorp.com/vault/docs/configuration for more information.
config:
telemetry:
prometheus_retention_time: 30s
disable_hostname: true
Disable statsd:
# since we are running Vault 1.1.0 with the native Prometheus support, we do not need the statsD exporter
statsdDisabled: true
Vault externalConfig policies:
externalConfig:
policies:
- name: prometheus-operator-prometheus
rules: |
path "auth/token/create/prometheus-metrics" {
capabilities = ["read", "update"]
}
- name: prometheus-metrics
rules: path "sys/metrics" {
capabilities = ["list", "read"]
}
auth:
auth:
- type: token
roles:
- name: prometheus-metrics
allowed_policies:
- prometheus-metrics
orphan: true
- type: kubernetes
roles:
- name: prometheus-operator-prometheus
bound_service_account_names: prometheus-operator-prometheus
bound_service_account_namespaces: mynamespace
policies: prometheus-operator-prometheus
ttl: 4h
Prometheus Operator Snippets
prometheusSpec
prometheusSpec:
# https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
podMetadata:
annotations:
vault.security.banzaicloud.io/vault-ct-configmap: "prometheus-consul-template"
vault.security.banzaicloud.io/vault-role: prometheus-operator-prometheus
vault.security.banzaicloud.io/vault-ct-image: "mycustomimage:latest"
secrets:
- etcd-client-tls
- vault-tls
Prometheus CRD ServiceMonitor
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: vault
app.kubernetes.io/instance: prometheus-operator
name: prometheus-operator-vault
spec:
endpoints:
- bearerTokenFile: /vault/secrets/vault-token
interval: 30s
params:
format: ['prometheus']
path: /v1/sys/metrics
port: api-port
scheme: https
tlsConfig:
caFile: /etc/prometheus/secrets/vault-tls/ca.crt
certFile: /etc/prometheus/secrets/vault-tls/server.crt
keyFile: /etc/prometheus/secrets/vault-tls/server.key
insecureSkipVerify: true
selector:
matchLabels:
app.kubernetes.io/name: vault
vault_cr: vault
10 - Running the webhook and Vault on different clusters
This section describes how to configure the webhook and Vault when the webhook runs on a different cluster from Vault, or if Vault runs outside Kubernetes.
Let’s suppose you have two different K8S clusters:
cluster1 contains vault-operatorcluster2 contains vault-secrets-webhook
Basically, you have to grant cluster2 access to the Vault running on cluster1. To achieve this, complete the following steps.
-
Extract the cluster.certificate-authority-data and the cluster.server fields from your cluster2 kubeconfig file. You will need them in the externalConfig section of the cluster1 configuration. For example:
kubectl config view -o yaml --minify=true --raw=true
-
Decode the certificate from the cluster.certificate-authority-data field, for example::
grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 --decode
-
On cluster2, create a vault ServiceAccount and the vault-auth-delegator ClusterRoleBinding:
kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
Expected output:
serviceaccount/vault created
role.rbac.authorization.k8s.io/vault created
role.rbac.authorization.k8s.io/leader-election-role created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
rolebinding.rbac.authorization.k8s.io/vault created
clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
You can use the vault ServiceAccount token as a token_reviewer_jwt in the auth configuration. To retrieve the token, run the following command:
kubectl get secret $(kubectl get sa vault -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode
-
In the vault.banzaicloud.com custom resource (for example, in this sample CR) of cluster1, define an externalConfig section. Fill the values of the kubernetes_ca_cert, kubernetes_host, and token_reviewer_jwt using the data collected in the previous steps.
externalConfig:
policies:
- name: allow_secrets
rules: path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
auth:
- type: kubernetes
config:
token_reviewer_jwt: <token-for-cluster2-service-account>
kubernetes_ca_cert: |
-----BEGIN CERTIFICATE-----
<certificate-from-certificate-authority-data-on-cluster2>
-----END CERTIFICATE-----
kubernetes_host: <cluster.server-field-on-cluster2>
roles:
# Allow every pod in the default namespace to use the secret kv store
- name: default
bound_service_account_names: ["default", "vault-secrets-webhook"]
bound_service_account_namespaces: ["default", "vswh"]
policies: allow_secrets
ttl: 1h
-
In a production environment, it is highly recommended to specify TLS config for your Vault ingress.
# Request an Ingress controller with the default configuration
ingress:
# Specify Ingress object annotations here, if TLS is enabled (which is by default)
# the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations
# to support TLS backends
annotations:
# Override the default Ingress specification here
# This follows the same format as the standard Kubernetes Ingress
# See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#ingressspec-v1beta1-extensions
spec:
tls:
- hosts:
- vault-dns-name
secretName: vault-ingress-tls-secret
-
Deploy the Vault custom resource containing the externalConfig section to cluster1:
kubectl apply -f your-proper-vault-cr.yaml
-
After Vault started in cluster1, you can use the vault-secrets-webhook in cluster2 with the proper annotations. For example:
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: hello-secrets
template:
metadata:
labels:
app.kubernetes.io/name: hello-secrets
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault-dns-name:443"
vault.security.banzaicloud.io/vault-role: "default"
vault.security.banzaicloud.io/vault-skip-verify: "true"
vault.security.banzaicloud.io/vault-path: "kubernetes"
Authenticate the mutating-webhook with a cloud identity
You can use a cloud identity to authenticate the mutating-webhook against the external vault.
-
Add your cloud authentication method in your external vault, for example, the Azure Auth Method.
-
Configure your vault-secrets-webhook to use the this method. For example:
env:
VAULT_ADDR: https://external-vault.example.com
VAULT_AUTH_METHOD: azure
VAULT_PATH: azure
VAULT_ROLE: default
For the VAULT_AUTH_METHOD env var, the following types are supported: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”