Bank-Vaults is a Vault swiss-army knife: a K8s operator, Go client with automatic token renewal, automatic configuration, multiple unseal options and more. A CLI tool to init, unseal and configure Vault (auth methods, secret engines). Direct secret injection into Pods.
Bank-Vaults provides the following tools for Hashicorp Vault to make its use easier and more automated:
A CLI tool to automatically initialize, unseal, and configure Vault with authentication methods and secret engines.
A Go client wrapper for the official Vault client with automatic token renewal, built-in Kubernetes support, and a dynamic database credential provider.
The package also includes Helm charts for installing the various components, and a collection of scripts to support advanced features (for example, dynamic SSH).
If you need help using Bank-Vaults, see the Support page for ways to contact us.
1 - Getting started
Bank-Vaults is a swiss-army knife with multiple manifestations, so the first steps depend on what you want to achieve.
Check one of the following guides to get an overview:
With the help of these charts you can run a HA Vault instance with automatic initialization, unsealing, and external configuration which would otherwise be a tedious manual operation. Also secrets from Vault can be injected into your Pods directly as environment variables (without using Kubernetes Secrets). These charts can be used easily for development purposes as well.
Note: Starting with Bank-Vaults version 1.6.0, only Helm 3 is supported. If you have installed the chart with Helm 2 and now you are trying to upgrade with Helm3, see the Bank-Vaults 1.6.0 release notes for detailed instructions.
Deploy a local Vault operator
This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets.
Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:
Wait a few seconds, then check the operator and the vault pods:
kubectl get pods
Expected output:
NAME READY STATUS RESTARTS AGE
vault-0 3/3 Running 0 10s
vault-configurer-6c545cb6b4-dmvb5 1/1 Running 0 10s
vault-operator-788559bdc5-kgqkg 1/1 Running 0 23s
Configure your Vault client to access the Vault instance running in the vault-0 pod.
Port-forward into the pod:
kubectl port-forward vault-0 8200&
Set the address of the Vault instance.
exportVAULT_ADDR=https://127.0.0.1:8200
Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):
Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true
Check that you can access the vault:
vault status
Expected output:
Key Value
--- -----
Seal Type shamir
Initialized trueSealed falseTotal Shares 5Threshold 3Version 1.5.4
Cluster Name vault-cluster-27ecd0e6
Cluster ID ed5492f3-7ef3-c600-aef3-bd77897fd1e7
HA Enabled false
To authenticate to Vault, you can access its root token by running:
exportVAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root}| base64 --decode)
Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.
Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).
For other configuration examples of the Vault CustomResource, see the YAML files in the operator/deploy directory of the project (we use these for testing). After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:
kubectl get pods --namespace vault-infra
NAME READY STATUS RESTARTS AGE
vault-secrets-webhook-58b97c8d6d-qfx8c 1/1 Running 0 22s
vault-secrets-webhook-58b97c8d6d-rthgd 1/1 Running 0 22s
Write a secret into Vault (the Vault CLI must be installed on your computer):
vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:
Check the mutated deployment.
kubectl describe deployment vault-test
The output should look similar to the following:
Name:vault-testNamespace:defaultCreationTimestamp:Wed, 04 Nov 2020 12:44:18 +0100Labels:<none>Annotations: deployment.kubernetes.io/revision:1Selector:app.kubernetes.io/name=vaultReplicas:1desired | 1 updated | 1 total | 1 available | 0 unavailableStrategyType:RollingUpdateMinReadySeconds:0RollingUpdateStrategy:25% max unavailable, 25% max surgePod Template:Labels:app.kubernetes.io/name=vaultAnnotations: vault.security.banzaicloud.io/vault-addr:https://vault:8200vault.security.banzaicloud.io/vault-agent:falsevault.security.banzaicloud.io/vault-path:kubernetesvault.security.banzaicloud.io/vault-role:defaultvault.security.banzaicloud.io/vault-skip-verify:falsevault.security.banzaicloud.io/vault-tls-secret:vault-tlsService Account:defaultContainers:alpine:Image:alpinePort:<none>Host Port:<none>Command:sh-cecho $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000Environment:AWS_SECRET_ACCESS_KEY:vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEYMounts:<none>Volumes:<none>Conditions:Type Status Reason---- ------ ------Available True MinimumReplicasAvailableProgressing True NewReplicaSetAvailableOldReplicaSets:<none>NewReplicaSet:vault-test-55c569f9 (1/1 replicas created)Events:Type Reason Age From Message---- ------ ---- ---- -------Normal ScalingReplicaSet 29s deployment-controller Scaled up replica set vault-test-55c569f9 to 1
As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.
Install the CLI tool
On macOs, you can directly install the CLI from Homebrew:
brew install banzaicloud/tap/bank-vaults
Alternatively, fetch the source code and compile it using go get:
go get github.com/bank-vaults/bank-vaults/cmd/bank-vaults
go get github.com/bank-vaults/bank-vaults/cmd/vault-env
Docker images
If you want to build upon our Docker images, you can find them on Docker Hub:
To deploy vault into a custom namespace (not into default), you have to deploy the vault CustomResource to the custom namespace. Also, you have to use the custom namespace in the following fields:
In addition to the standard Vault configuration, the operator and CLI can continuously configure Vault using an external YAML/JSON configuration. That way you can configure Vault declaratively using your usual automation tools and workflow.
The following sections describe the configuration sections you can use.
2.1 - Fully or partially purging unmanaged configuration in Vault
Bank-Vaults gives you a full control over Vault in a declarative style by removing any unmanaged configuration.
By enabling purgeUnmanagedConfig you keep Vault configuration up-to-date.
So if you added a policy using Bank-Vaults then removed it from the configuration,
Bank-Vaults will remove it from Vault too. In other words, if you enabled purgeUnmanagedConfig
then any changes not in Bank-Vaults configuration will be removed (including manual changes).
Note:
This feature is destructive, so be carful when you enable it especially for the first time
because it could delete all data in your Vault. We recommend you to test it a non-production environment first.
This feature is disabled by default and it needs to be enabled explicitly in your configuration.
Mechanism
Bank-Vaults handles unmanaged configuration by simply comparing what in Bank-Vaults configuration (the desired state)
and what’s already in Vault (the actual state), then it removes any differences that are not in Bank-Vaults
configuration.
Fully purge unmanaged configuration
You can remove all unmanaged configuration by enabling the purge option as following:
purgeUnmanagedConfig:enabled:true
Partially purge unmanaged configuration
You can also enable the purge feature for some of the config by excluding any config that
you don’t want to purge its unmanaged config.
It could be done by explicitly exclude the Vault configuration that you don’t want to mange:
This will remove any unmanaged or manual changes in Vault but it will leave secrets untouched.
So if you enabled a new secret engine manually (and it’s not in Bank-Vaults configuration),
Bank-Vaults will not remove it.
auth:- type:aws# Make the auth provider visible in the web ui# See https://www.vaultproject.io/api/system/auth.html#config for more# information.options:listing_visibility:"unauth"config:access_key:VKIAJBRHKH6EVTTNXDHAsecret_key:vCtSM8ZUEQ3mOFVlYPBQkf2sO6F/W7a5TVzrl3Ojiam_server_id_header_value:vault-dev.example.com# consider setting this to the Vault server's DNS namecrossaccountrole:# Add cross account number and role to assume in the cross account# https://www.vaultproject.io/api/auth/aws/index.html#create-sts-role- sts_account:12345671234sts_role:arn:aws:iam::12345671234:role/crossaccountroleroles:# Add roles for AWS instances or principals# See https://www.vaultproject.io/api/auth/aws/index.html#create-role- name:dev-role-iambound_iam_principal_arn:arn:aws:iam::123456789012:role/dev-vaultpolicies:allow_secretsperiod:1h- name:cross-account-rolebound_iam_principal_arn:arn:aws:iam::12345671234:role/crossaccountrolepolicies:allow_secretsperiod:1h
auth:- type:gcp# Make the auth provider visible in the web ui# See https://www.vaultproject.io/api/system/auth.html#config for more# information.options:listing_visibility:"unauth"config:# Credentials context is service account's key. Can download when you create a key for service account. # No need to manually create it. Just paste the json context as multiline yaml.credentials:-| {
"type": "service_account",
"project_id": "PROJECT_ID",
"private_key_id": "KEY_ID",
"private_key": "-----BEGIN PRIVATE KEY-----.....-----END PRIVATE KEY-----\n",
"client_email": "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com",
"client_id": "CLIENT_ID",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT%40PROJECT_ID.iam.gserviceaccount.com"
}roles:# Add roles for gcp service account# See https://www.vaultproject.io/api/auth/gcp/index.html#create-role- name:user-roletype:iamproject_id:PROJECT_IDpolicies:"readonly_secrets"bound_service_accounts:"USER_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"- name:admin-roletype:iamproject_id:PROJECT_IDpolicies:"allow_secrets"bound_service_accounts:"ADMIN_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
GitHub auth method
Create team mappings in Vault which can be used later on for the
GitHub authentication.
auth:- type:github# Make the auth provider visible in the web ui# See https://www.vaultproject.io/api/system/auth.html#config for more# information.options:listing_visibility:"unauth"config:organization:banzaicloudmap:# Map the banzaicloud GitHub team on to the dev policy in Vaultteams:dev:dev# Map my username (bonifaido) to the root policy in Vaultusers:bonifaido:allow_secrets
Use the Kubernetes auth method to authenticate with Vault
using a Kubernetes Service Account Token.
auth:- type:kubernetes# If you want to configure with specific kubernetes service account instead of default service account# https://www.vaultproject.io/docs/auth/kubernetes.html# config:# token_reviewer_jwt: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9....# kubernetes_ca_cert: |# -----BEGIN CERTIFICATE-----# ...# -----END CERTIFICATE-----# kubernetes_host: https://192.168.64.42:8443# Allows creating roles in Vault which can be used later on for the Kubernetes based# authentication.# See https://www.vaultproject.io/docs/auth/kubernetes.html#creating-a-role for# more information.roles:# Allow every pod in the default namespace to use the secret kv store- name:defaultbound_service_account_names:defaultbound_service_account_namespaces:defaultpolicies:allow_secretsttl:1h
To start an LDAP test server, run: docker run -it –rm -p 389:389 -e LDAP_TLS=false –name ldap osixia/openldap
To start an LDAP admin server, run: docker run -it –rm -p 6443:443 –link ldap:ldap -e PHPLDAPADMIN_LDAP_HOSTS=ldap -e PHPLDAPADMIN_LDAP_CLIENT_TLS=false osixia/phpldapadmin
auth:- type:ldapdescription:LDAP directory auth.# add mount options# See https://www.vaultproject.io/api/system/auth.html#config for more# information.options:listing_visibility:"unauth"config:url:ldap://localhostbinddn:"cn=admin,dc=example,dc=org"bindpass:"admin"userattr:uiduserdn:"ou=users,dc=example,dc=org"groupdn:"ou=groups,dc=example,dc=org"groups:# Map the banzaicloud dev team on GitHub to the dev policy in Vaultdevelopers:policies:allow_secrets# Map myself to the allow_secrets policy in Vaultusers:bonifaido:groups:developerspolicies:allow_secrets
2.4 - Plugins
To register a new plugin in Vault’s plugin catalog,
set the plugin_directory option in the Vault server configuration to the directory where the plugin binary
is located. Also, for some plugins readOnlyRootFilesystem Pod Security Policy should be disabled to allow RPC
communication between plugin and Vault server via Unix socket. For details,
see the Hashicorp Go plugin documentation.
You can configure Secrets Engines in Vault.
The Key-Value, Database, and SSH values are tested, but the configuration is free form, so probably others work as well.
AWS
The AWS secrets engine generates AWS access credentials
dynamically based on IAM policies.
This plugin stores database credentials dynamically based on configured roles for the
MySQL/MariaDB database.
secrets:- type:databasedescription:MySQL Database secret engine.configuration:config:- name:my-mysqlplugin_name:"mysql-database-plugin"connection_url:"{{username}}:{{password}}@tcp(127.0.0.1:3306)/"allowed_roles:[pipeline]username:"${env `ROOT_USERNAME`}"# Example how to read environment variablespassword:"${env `ROOT_PASSWORD`}"roles:- name:pipelinedb_name:my-mysqlcreation_statements:"GRANT ALL ON *.* TO '{{name}}'@'%' IDENTIFIED BY '{{password}}';"default_ttl:"10m"max_ttl:"24h"
Only external groups are supported at the moment through the use of group-aliases.
For supported authentication backends (for example JWT, which automatically matches those aliases
to groups returned by the backend) the configuration files for the groups and group-aliases
need to be parsed after the authentication backend has been mounted. Ideally they should be in the same file
to avoid of errors.
This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets.
Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:
Wait a few seconds, then check the operator and the vault pods:
kubectl get pods
Expected output:
NAME READY STATUS RESTARTS AGE
vault-0 3/3 Running 0 10s
vault-configurer-6c545cb6b4-dmvb5 1/1 Running 0 10s
vault-operator-788559bdc5-kgqkg 1/1 Running 0 23s
Configure your Vault client to access the Vault instance running in the vault-0 pod.
Port-forward into the pod:
kubectl port-forward vault-0 8200&
Set the address of the Vault instance.
exportVAULT_ADDR=https://127.0.0.1:8200
Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):
Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true
Check that you can access the vault:
vault status
Expected output:
Key Value
--- -----
Seal Type shamir
Initialized trueSealed falseTotal Shares 5Threshold 3Version 1.5.4
Cluster Name vault-cluster-27ecd0e6
Cluster ID ed5492f3-7ef3-c600-aef3-bd77897fd1e7
HA Enabled false
To authenticate to Vault, you can access its root token by running:
exportVAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root}| base64 --decode)
Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.
Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).
For other configuration examples of the Vault CustomResource, see the YAML files in the operator/deploy directory of the project (we use these for testing). After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:
Backing up the storage backend to prevent data loss, is not handled by the Vault operator. We recommend using Velero for backups.
Pod anti-affinity
If you want to setup pod anti-affinity, you can set podAntiAffinity vault with a topologyKey value.
For example, you can use failure-domain.beta.kubernetes.io/zone to force K8S deploy vault on multi AZ.
Delete a resource created by the operator
If you manually delete a resource that the Bank-Vaults operator has created (for example, the Ingress resource), the operator automatically recreates it every 30 seconds. If it doesn’t, then something went wrong, or the operator is not running. In this case, check the logs of the operator.
4.1 - Running Vault with external end to end encryption
This document assumes you have a working Kubernetes cluster which has a:
Working install of Vault.
That you have a working knowledge of Kubernetes.
A working install of helm
A working knowledge of Kubernetes ingress
A valid external (www.example.com) SSL certificate, verified by your provider as a Kubernetes secret.
Background
The bank-vaults operator takes care of creating and maintaining internal cluster communications but if you wish to use your vault install
outside of your Kubernetes cluster what is the best way to maintain a secure state. Creating a standard Ingress object will reverse proxy
these requests to your vault instance but this is a hand off between the external SSL connection and the internal one. This might not be acceptable
under some circumstances, for example, if you have to adhere to strict security standards.
Workflow
Here we will create a separate TCP listener for vault using a custom SSL certificate on an external domain of your choosing. We will then
install a unique ingress-nginx controller allowing SSL pass through. SSL Pass through comes with a performance hit, so you would not use this
on a production website or ingress-controller that has a lot of traffic.
---apiVersion:"vault.banzaicloud.com/v1alpha1"kind:"Vault"metadata:name:"vault"namespace:secretsspec:size:2image:vault:1.1.2bankVaultsImage:banzaicloud/bank-vaults:0.4.16# A YAML representation of a final vault config file.# See https://www.vaultproject.io/docs/configuration/ for more information.config:listener:- tcp:address:"0.0.0.0:8200"tls_cert_file:/vault/tls/server.crttls_key_file:/vault/tls/server.key- tcp:address:"0.0.0.0:8300"tls_cert_file:/etc/ingress-tls/tls.crttls_key_file:/etc/ingress-tls/tls.keyapi_addr:https://vault:8200cluster_addr:https://vault:8201ui:true
CR Service:
# Specify the Service's type where the Vault Service is exposedserviceType:ClusterIPservicePorts:api-port:8200cluster-port:8201ext-api-port:8300ext-clu-port:8301
# Request an Ingress controller with the default configurationingress:annotations:kubernetes.io/ingress.allow-http:"false"kubernetes.io/ingress.class:"nginx-vault"nginx.ingress.kubernetes.io/force-ssl-redirect:"true"nginx.ingress.kubernetes.io/ssl-passthrough:"true"nginx.ingress.kubernetes.io/backend-protocol:"HTTPS"nginx.ingress.kubernetes.io/whitelist-source-range:"127.0.0.1"spec:rules:- host:vault.example.comhttp:paths:- path:/backend:serviceName:vaultservicePort:8300
4.2 - Using templates for injecting dynamic configuration
Background
When configuring a Vault object via the externalConfig property, sometimes it’s convenient (or necessary) to inject settings that are only known at runtime, for example:
secrets that you don’t want to store in source control
dynamic resources managed elsewhere
computations based on multiple values (string or arithmetic operations).
For these cases, the operator supports parameterized templating. The vault-configurer component evaluates the templates and injects the rendered configuration into Vault.
This templating is based on Go templates, extended by Sprig, with some custom functions available specifically for bank-vaults (for example, to decrypt strings using the AWS Key Management Service or the Cloud Key Management Service of the Google Cloud Platform).
Using templates
To avoid confusion and potential parsing errors (and interference with other templating tools like Helm), the templates don’t use the default delimiters that Go templates use ({{ and }}). Instead, use the following characters:
${ for the left delimiter
} for the right one.
To quote parameters being passed to functions, surround them with backticks (`) instead.
For example, to call the env function, you can use this in your manifest:
password:"${ env `MY_ENVIRONMENT_VARIABLE` }"
In this case, vault-configurer evaluates the value of MY_ENVIRONMENT_VARIABLE at runtime (assuming it was properly injected), and sets the result as the value of the password field.
In addition to the default functions in Go templates, you can also use Sprig functions in your configuration.
CAUTION:
Use only functions that return a string, otherwise the generated configuration is invalid.
Custom functions
To provide functionality that’s more Kubernetes-friendly and cloud-native, bank-vaults provides a few additional functions not available in Sprig or Go. The functions and their parameters (in the order they should go in the function) are documented below.
awskms
Takes a base64-encoded, KMS-encrypted string and returns the decrypted string. Additionally, the function takes an optional second parameter for any encryption context that might be required for decrypting. If any encryption context is required, the function will take any number of additional parameters, each of which should be a key-value pair (separated by a = character), corresponding to the full context.
Note: This function assumes that the vault-configurer pod has the appropriate AWS IAM credentials and permissions to decrypt the given string. You can inject the AWS IAM credentials by using Kubernetes secrets as environment variables, an EC2 instance role, kube2iam, or EKS IAM roles, and so on.
Parameter
Type
Required
encodedString
Base64-encoded string
Yes
encryptionContext
Variadic list of strings
No
For example:
password:'${ awskms (env `ENCRYPTED_DB_CREDS`) }'
You can also nest functions in the template, for example:
Takes a base64-encoded string, encrypted with a Google Cloud Platform (GCP) symmetric key and returns the decrypted string.
Note: This function assumes that the vault-configurer pod has the appropriate GCP IAM credentials and permissions to decrypt the given string. You can inject the GCP IAM credentials by using Kubernetes secrets as environment variables, or they can be acquired via a service account authentication, and so on.
Parameter
Type
Required
encodedString
Base64-encoded string
Yes
projectId
String
Yes
location
String
Yes
keyRing
String
Yes
key
String
Yes
blob
Reads the content of a blob from disk (file) or from cloud blob storage services (object storage) at the given URL and returns it. This assumes that the path exists, and is readable by vault-configurer.
Valid values for the URL parameters are listed below, for more fine grained options check the documentation of the underlying library:
file:///path/to/dir/file
s3://my-bucket/object?region=us-west-1
gs://my-bucket/object
azblob://my-container/blob
Note: This function assumes that the vault-configurer pod has the appropriate rights to access the given cloud service. For details, see the awskms and gcpkms functions.
Reads the content of a file from disk at the given path and returns it. This assumes that the file exists, it’s mounted, and readable by vault-configurer.
Parameter
Type
Required
path
String
Yes
accessor
Looks up the accessor id of the given auth path and returns it. This function is only useful in policies that use templated policies, to generalize the <mount accessor> field.
The Bank-Vaults operator updates the statefulset. (It does not take the HA leader into account in HA scenarios, but this has never caused any issues so far.)
4.4 - Operator Configuration for Functioning Webhook Secrets Mutation
You can find several examples of the vault operator CR manifest in
the project repository. The following examples use only this vanilla CR
to demonstrate some main points about how to properly configure the
operator for secrets mutations to function.
This document does not attempt to explain every possible scenario with respect to
the CRs in the aforementioned directory, but instead attempts to explain at
a high level the important aspects of the CR, so that you can determine how best to configure your operator.
Main points
Some important aspects of the operator and its configuration with respect to secrets
mutation are:
The vault operator instantiates:
the vault configurer pod(s),
the vault pod(s),
the vault-secrets-webhook pod(s).
The vault configurer:
unseals the vault,
configures vault with policies, roles, and so on.
vault-secrets-webhook does nothing more than:
monitors cluster for resources with specific annotations for secrets injection, and
integrates with vault API to answer secrets requests from those resources for
requested secrets.
For pods using environment secrets, it injects a binary vault-env into pods
and updates ENTRYPOINT to run vault-env CMD instead of CMD.
vault-env intercepts requests for env secrets requests during runtime of
pod and upon such requests makes vault API call for requested secret
injecting secret into environment variable so CMD works with proper
secrets.
Vault
the secrets workhorse
surfaces a RESTful API for secrets management
CR configuration properties
This section goes over some important properties of the CR and their purpose.
Vault’s service account
This is the serviceaccount where Vault will be running. The Configurer runs
in the same namespace and should have the same service account. The operator
assigns this serviceaccount to Vault.
# Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is runningserviceAccount:vault
caNamespaces
In order for vault communication to be encrypted, valid TLS certificates need to
be used. The following property automatically creates TLS certificate secrets for
the namespaces specified here. Notice that this is a list, so you can
specify multiple namespaces per line, or use the splat or wildcard asterisk to
specify all namespaces:
# Support for distributing the generated CA certificate Secret to other namespaces.# Define a list of namespaces or use ["*"] for all namespaces.caNamespaces:- "*"
Vault Config
The following is simply a YAML representation (as the comment says) for the
Vault configuration you want to run. This is the configuration that vault
configurer uses to configure your running Vault:
# A YAML representation of a final vault config file.config:api_addr:https://vault:8200cluster_addr:https://${.Env.POD_NAME}:8201listener:tcp:address:0.0.0.0:8200# Commenting the following line and deleting tls_cert_file and tls_key_file disables TLStls_cert_file:/vault/tls/server.crttls_key_file:/vault/tls/server.keystorage:file:path:"${ .Env.VAULT_STORAGE_FILE }"ui:truecredentialsConfig:env:""path:""secretName:""etcdSize:0etcdVersion:""externalConfig:policies:- name:allow_secretsrules:path "secret/*" {capabilities = ["create", "read", "update", "delete", "list"]}auth:- type:kubernetesroles:# Allow every pod in the default namespace to use the secret kv store- name:defaultbound_service_account_names:- external-secrets- vault- dexbound_service_account_namespaces:- external-secrets- vault- dex- auth-system- loki- grafanapolicies:- allow_secretsttl:1h# Allow mutation of secrets using secrets-mutation annotation to use the secret kv store- name:secretsmutationbound_service_account_names:- vault-secrets-webhookbound_service_account_namespaces:- vault-secrets-webhookpolicies:- allow_secretsttl:1h
externalConfig
The externalConfig portion of this CR example correlates to Kubernetes
configuration as specified by .auth[].type.
This YAML representation of configuration is flexible enough to work with any
auth methods available to Vault as documented in the Vault documentation.
For now, we’ll stick with this kubernetes configuration.
externalConfig.purgeUnmanagedConfig
Delete any configuration that in Vault but not in externalConfig. For more details please check
Purge unmanaged configuration
externalConfig.policies
Correlates 1:1 to the creation of the specified policy in conjunction with Vault
policies.
Vault does not respect inline secrets serviceaccount annotations, so the
namespace of any serviceaccount annotations for secrets are irrelevant to
getting inline secrets mutations functioning.
Instead, the serviceaccount of the vault-secrets-webhook pod(s) should be
used to configure the bound_service_account_names and
bound_service_account_namespaces for inline secrets to mutate.
Pod serviceaccounts, however, are respected so
bound_service_account_namespaces and bound_service_account_names for
environment mutations must identify such of the running pods.
Note: There are two roles specified in the YAML example above: one for pods, and one for inline secrets mutations. While this was not strictly required, it makes for cleaner implementation.
5 - Mutating Webhook
How the webhook works - overview
Kubernetes secrets are the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption eventually ends up as a Kubernetes secret. However, despite their name, Kubernetes secrets are not exactly secure, since they are only base64 encoded.
The mutating webhook of Bank-Vaults is a solution that bypasses the Kubernetes secrets mechanism and injects the secrets retrieved from Vault directly into the Pods. Specifically, the mutating admission webhook injects (in a very non-intrusive way) an executable into containers of Deployments and StatefulSets. This executable can request secrets from Vault through special environment variable definitions.
An important and unique aspect of the webhook is that it is a daemonless solution (although if you need it, you can deploy the webhook in daemon mode as well).
Why is this more secure than using Kubernetes secrets or any other custom sidecar container?
Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk or in etcd - not even temporarily. All secrets are stored in memory, and are only visible to the process that requested them. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.
If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.
The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.
Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them.
The secret value should contain a base64 encoded template string referencing the vault path you want to insert.
Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.
kubectl get pods --namespace vault-infra
NAME READY STATUS RESTARTS AGE
vault-secrets-webhook-58b97c8d6d-qfx8c 1/1 Running 0 22s
vault-secrets-webhook-58b97c8d6d-rthgd 1/1 Running 0 22s
Write a secret into Vault (the Vault CLI must be installed on your computer):
vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:
Check the mutated deployment.
kubectl describe deployment vault-test
The output should look similar to the following:
Name:vault-testNamespace:defaultCreationTimestamp:Wed, 04 Nov 2020 12:44:18 +0100Labels:<none>Annotations: deployment.kubernetes.io/revision:1Selector:app.kubernetes.io/name=vaultReplicas:1desired | 1 updated | 1 total | 1 available | 0 unavailableStrategyType:RollingUpdateMinReadySeconds:0RollingUpdateStrategy:25% max unavailable, 25% max surgePod Template:Labels:app.kubernetes.io/name=vaultAnnotations: vault.security.banzaicloud.io/vault-addr:https://vault:8200vault.security.banzaicloud.io/vault-agent:falsevault.security.banzaicloud.io/vault-path:kubernetesvault.security.banzaicloud.io/vault-role:defaultvault.security.banzaicloud.io/vault-skip-verify:falsevault.security.banzaicloud.io/vault-tls-secret:vault-tlsService Account:defaultContainers:alpine:Image:alpinePort:<none>Host Port:<none>Command:sh-cecho $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000Environment:AWS_SECRET_ACCESS_KEY:vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEYMounts:<none>Volumes:<none>Conditions:Type Status Reason---- ------ ------Available True MinimumReplicasAvailableProgressing True NewReplicaSetAvailableOldReplicaSets:<none>NewReplicaSet:vault-test-55c569f9 (1/1 replicas created)Events:Type Reason Age From Message---- ------ ---- ---- -------Normal ScalingReplicaSet 29s deployment-controller Scaled up replica set vault-test-55c569f9 to 1
As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.
Deploy the webhook from a private registry
If you are getting the x509: certificate signed by unknown authority app=vault-secrets-webhook error when the webhook is trying to download the manifest from a private image registry, you can:
Build a docker image where the CA store of the OS layer of the image contains the CA certificate of the registry.
Alternatively, you can disable certificate verification for the registry by using the REGISTRY_SKIP_VERIFY=“true” environment variable in the deployment of the webhook.
Deploy in daemon mode
vault-env by default replaces itself with the original process of the Pod after reading the secrets from Vault, but with the vault.security.banzaicloud.io/vault-env-daemon: "true" annotation this behavior can be changed. So vault-env can change to daemon mode, so vault-env starts the original process as a child process and remains in memory, and renews the lease of the requested Vault token and of the dynamic secrets (if requested any) until their final expiration time.
# Deploy MySQL first as the Vault storage backend and our application will request dynamic secrets for this database as well:helm upgrade --install mysql stable/mysql --set mysqlRootPassword=your-root-password --set mysqlDatabase=vault --set mysqlUser=vault --set mysqlPassword=secret --set 'initializationFiles.app-db\.sql=CREATE DATABASE IF NOT EXISTS app;'# Deploy the vault-operator and the vault-secerts-webhookkubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook
# Create a Vault instance with MySQL storage and a configured dynamic database secrets backendkubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-mysql-ha.yaml
# Deploy the example application requesting dynamic database credentials from the above Vault instancekubectl apply -f deploy/test-dynamic-env-vars.yaml
kubectl logs -f deployment/hello-secrets
5.2 - Configuration examples and scenarios
The following examples show you how to configure the mutating webhook to best suit your environment.
The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.
Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them.
The secret value should contain a base64 encoded template string referencing the vault path you want to insert.
Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.
Vault needs to be properly configured for mutation to function; namely
externalConfig.auth and externConfig.roles (from the perspective of the
vault operator CR) need to be properly configured. If you’re not using the vault
operator then you must make sure that your Vault configuration for Kubernetes
auth methods are properly configured. This configuration is outside the scope
of this document. If you use the operator for managing Vault in your cluster, see the Vault operator documentation.
Inject secret into resources
The webhook can inject into any kind of resources, even into CRDs, for example:
You can also inject multiple secrets under the same key in a Secret/ConfigMap/Object. This means that you can use multiple Vault paths in a value, for example:
apiVersion:v1kind:ConfigMapmetadata:name:sample-configmapannotations:vault.security.banzaicloud.io/vault-addr:"https://vault.default:8200"vault.security.banzaicloud.io/vault-role:"default"vault.security.banzaicloud.io/vault-tls-secret:vault-tlsvault.security.banzaicloud.io/vault-path:"kubernetes"data:aws-access-key-id:"vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID"aws-access-template:"vault:secret/data/accounts/aws#AWS key in base64: ${.AWS_ACCESS_KEY_ID | b64enc}"aws-access-inline:"AWS_ACCESS_KEY_ID: ${vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY}"
This example also shows how a CA certificate (created by the operator) can be used with the vault.security.banzaicloud.io/vault-tls-secret: vault-tls annotation to validate the TLS connection in case of a non-Pod resource.
Request a Vault token
There is a special vault:login reference format to request a working Vault token into an environment variable to be later consumed by your application:
NOTE: This feature takes advantage of secret caching since we need to access the my-role endpoint twice, but in the background, it is written only once in Vault:
Values starting with ">>vault:" issue a write (HTTP POST/PUT) request towards the Vault API, some secret engine APIs should be written instead of reading from like the Password Generator for HashiCorp Vault:
Or with Transit Secret Engine which is a fairly complex example since we are using templates when rendering the response and send data in the write request as well, the format is: vault:PATH#KEY_OR_TEMPLATE#DATA
Templating is also supported on the secret sourced from Vault (in the key part, after the first #), in the very same fashion as in the Vault configuration and external configuration with all the Sprig functions (this is supported only for Pods right now):
env:- name:DOCKER_USERNAMEvalue:"vault:secret/data/accounts/dockerhub#My username on DockerHub is: ${title .DOCKER_USERNAME}"
In this case, an init-container will be injected into the given Pod. This container copies the vault-env binary into an in-memory volume and mounts that Volume to every container which has an environment variable definition like that. It also changes the command of the container to run vault-env instead of your application directly. When vault-env starts up, it connects to Vault to checks the environment variables. (By default, vault-env uses the Kubernetes Auth method, but you can also configure other authentication methods for the webhook.) The variables that have a reference to a value stored in Vault (vault:secret/....) are replaced with that value read from the Secret backend. After this, vault-env immediately executes (with syscall.Exec()) your process with the given arguments, replacing itself with that process (in non-daemon mode).
With this solution none of your Secrets stored in Vault will ever land in Kubernetes Secrets, thus in etcd.
vault-env was designed to work in Kubernetes in the first place, but nothing stops you to use it outside Kubernetes as well. It can be configured with the standard Vault client’s environment variables (because there is a standard Go Vault client underneath).
Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, so it requests a Vault token based on the Service Account of the container it is injected into.
Kubernetes 1.12 introduced a feature called APIServer dry-run which became beta as of 1.13. This feature requires some changes in webhooks with side effects. Vault mutating admission webhook is dry-run aware.
Mutate data from Vault and replace it in Kubernetes Secret
You can mutate Secrets (and ConfigMaps) as well if you set annotations and define proper Vault path in the data section:
In the example above the secret type is kubernetes.io/dockerconfigjson and the webhook can get credentials from Vault.
The base64 encoded data contain vault path in case of username and password for docker repository and you can create it with commands:
Use charts without explicit container.command and container.args
The Webhook can determine the container’s ENTRYPOINT and CMD with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication) you don’t need to do anything, but if the registry requires authentication the credentials have to be available in the Pod’s imagePullSecrets section.
Some examples (apply cr.yaml from the operator samples first):
You can also specify a default secret being used by the webhook for cases where a pod has no imagePullSecrets specified. To make this work you have to set the environment variables DEFAULT_IMAGE_PULL_SECRET and DEFAULT_IMAGE_PULL_SECRET_NAMESPACE when deploying the vault-secrets-webhook. Have a look at the values.yaml of the
vault-secrets-webhook helm chart to see how this is done.
Note:
If your EC2 nodes have the ECR instance role, the webhook can request an ECR access token through that role automatically, instead of an explicit imagePullSecret
If your workload is running on GCP nodes, the webhook automatically authenticates to GCR.
Future improvements:
on Azure/Alibaba get a credential dynamically with the specific SDK (for AWS ECR and GCP GCR this is already done)
This feature is very similar to Kubernetes’ standard envFrom:construct, but instead of a Kubernetes Secret/ConfigMap, all its keys are mounted from a Vault secret using the webhook and vault-env.
You can set the Vault secret to mount using the vault.security.banzaicloud.io/vault-env-from-path annotation.
Compared to the original environment variable definition in the Pod env construct, the only difference is that you won’t see the actual environment variables in the definition, because they are dynamic, and are based on the contents of the Vault secret’s, just like envFrom:.
By default, the webhook uses Kubernetes ServiceAccount-based authentication in Vault. Use the vault.security.banzaicloud.io/vault-auth-method annotation to request different authentication types from the following supported types: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”.
Note: GCP IAM authentication (gcp-iam) only allows for authentication with the ‘default’ service account of the caller, and a new token is generated at every request.
Note: Azure MSI authentication (azure) a new token is generated at every request.
The following deployment - if running on a GCP instance - will automatically receive a signed JWT token from the metadata server of the cloud provider, and use it to authenticate against Vault. The same goes for vault-auth-method: "aws-ec2", when running on an EC2 node with the right instance-role.
The ServiceAccount in the objects namespace to use, useful for non-pod resources
vault.security.banzaicloud.io/vault-namespace
""
The Vault Namespace secrets will be pulled from. This annotation sets the VAULT_NAMESPACE environment variable. More information on namespaces within Vault can be found here
vault.security.banzaicloud.io/run-as-non-root
"false"
When enabled will add runAsNonRoot: true to the securityContext of all injected containers
vault.security.banzaicloud.io/run-as-user
"0"
Set the UID (runAsUser) for all injected containers. The default value of "0" means that no modifications will be made to the securityContext of injected containers.
vault.security.banzaicloud.io/run-as-group
"0"
Set the GID (runAsGroup) for all injected containers. The default value of "0" means that no modifications will be made to the securityContext of injected containers.
vault.security.banzaicloud.io/readonly-root-fs
"false"
When enabled will add readOnlyRootFilesystem: true to the securityContext of all injected containers
5.4 - Using Vault Agent Templating in the mutating webhook
With Bank-Vaults you can use Vault Agent to handle secrets that expire, and supply them to applications that read their configurations from a file.
When to use vault-agent
You have an application or tool that requires to read its configuration from a file.
You wish to have secrets that have a TTL and expire.
You have no issues with running your application with a sidecar.
Your pod starts up, the webhook will inject one container into the pods lifecycle.
The sidecar container is running Vault, using the vault agent that accesses Vault using the configuration specified inside a configmap and writes a configuration file based on a pre configured template (written inside the same configmap) onto a temporary file system which your application can use.
If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Vault Agent to execute a command when it writes a new configuration file using the command attribute. The following is a basic example which uses the Kubernetes authentication method.
For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-agent-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Vault Agent.
Defaults via environment variables
Variable
default
Explanation
VAULT_IMAGE
vault:latest
the vault image to use for the sidecar container
VAULT_IMAGE_PULL_POLICY
IfNotPresent
The pull policy for the vault agent container
VAULT_ADDR
https://127.0.0.1:8200
Kubernetes service Vault endpoint URL
VAULT_TLS_SECRET
""
supply a secret with the vault TLS CA so TLS can be verified
VAULT_AGENT_SHARE_PROCESS_NAMESPACE
Kubernetes version <1.12 default off, 1.12 or higher default on
5.5 - Using consul-template in the mutating webhook
With Bank-Vaults you can use Consul Template as an addition to vault-env to handle secrets that expire, and supply them to applications that read their configurations from a file.
When to use consul-template
You have an application or tool that must read its configuration from a file.
You wish to have secrets that have a TTL and expire.
You do not wish to be limited on which vault secrets backend you use.
You can also expire tokens/revoke tokens (to do this you need to have a ready/live probe that can send a HUP to consul-template when the current details fail).
Workflow
The following shows the general workflow for using Consul Template:
Your pod starts up. The webhook injects an init container (running vault agent) and a sidecar container (running consul-template) into the pods lifecycle.
The vault agent in the init container logs in to Vault and retrieves a Vault token based on the configured VAULT_ROLE and Kubernetes Service Account.
The consul-template running in the sidecar container logs in to Vault using the Vault token and writes a configuration file based on a pre-configured template in a configmap onto a temporary file system which your application can use.
If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Consul Template to execute a command when it writes a new configuration file using the command attribute. The following is a basic example (adapted from here).
For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-ct-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Consul Template.
Defaults via environment variables
Variable
default
Explanation
VAULT_IMAGE
vault:latest
the vault image to use for the init container
VAULT_ENV_IMAGE
banzaicloud/vault-env:latest
the vault-env image to use
VAULT_CT_IMAGE
hashicorp/consul-template:latest
the consul template image to use
VAULT_ADDR
https://127.0.0.1:8200
Kubernetes service Vault endpoint URL
VAULT_SKIP_VERIFY
“false”
should vault agent and consul template skip verifying TLS
VAULT_TLS_SECRET
""
supply a secret with the vault TLS CA so TLS can be verified
VAULT_AGENT
“true”
enable the vault agent
VAULT_CT_SHARE_PROCESS_NAMESPACE
Kubernetes version <1.12 default off, 1.12 or higher default on
ShareProcessNamespace override
PodSpec annotations
Annotation
default
Explanation
vault.security.banzaicloud.io/vault-addr
Same as VAULT_ADDR above
vault.security.banzaicloud.io/vault-role
default
The Vault role for Vault agent to use
vault.security.banzaicloud.io/vault-path
auth/<method type>
The mount path of the method
vault.security.banzaicloud.io/vault-skip-verify
Same as VAULT_SKIP_VERIFY above
vault.security.banzaicloud.io/vault-tls-secret
Same as VAULT_TLS_SECRET above
vault.security.banzaicloud.io/vault-agent
Same as VAULT_AGENT above
vault.security.banzaicloud.io/vault-ct-configmap
""
A configmap name which holds the consul template configuration
vault.security.banzaicloud.io/vault-ct-image
""
Specify a custom image for consul template
vault.security.banzaicloud.io/vault-ct-once
false
do not run consul-template in daemon mode, useful for kubernetes jobs
The transit secrets engine handles cryptographic functions on data in-transit, mainly to encrypt data from applications while still storing that encrypted data in some primary data store. Vault doesn’t store the data sent to the secrets engine, it can also be viewed as “cryptography as a service” or “encryption as a service”. For details about transit encryption, see the official documentation.
After completing the previous steps, the webhook will mutate pods that have at least one environment variable with a value which is encrypted by Vault. For example (in the last line of the example):
apiVersion:apps/v1kind:Deploymentmetadata:name:vault-testspec:replicas:1selector:matchLabels:app.kubernetes.io/name:vaulttemplate:metadata:labels:app.kubernetes.io/name:vaultannotations:vault.security.banzaicloud.io/vault-addr:"https://vault:8200"# optional, the address of the Vault service, default values is https://vault:8200vault.security.banzaicloud.io/vault-role:"default"# optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"vault.security.banzaicloud.io/vault-skip-verify:"false"# optional, skip TLS verification of the Vault server certificatevault.security.banzaicloud.io/vault-tls-secret:"vault-tls"# optinal, the name of the Secret where the Vault CA cert is, if not defined it is not mountedvault.security.banzaicloud.io/vault-agent:"false"# optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authenticationvault.security.banzaicloud.io/vault-path:"kubernetes"# optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"vault.security.banzaicloud.io/transit-key-id:"my-key"# required if encrypted data was found; transit key id that created beforespec:serviceAccountName:defaultcontainers:- name:alpineimage:alpinecommand:["sh","-c","echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]env:- name:AWS_SECRET_ACCESS_KEY# Value based on encrypted key that stored in Vault, so value from this example# not the same as you can get after `encrypt`value:vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==
5.7 - Monitoring the Webhook with Grafana and Prometheus
To monitor the webhook with Prometheus and Grafana, complete the following steps.
Prerequisites
An already deployed and configured mutating webhook. For details, see Mutating Webhook.
Select the previously created Data Source to feed this dashboard.
5.8 - Injecting consul-template into the Prometheus operator for Vault metrics
To get vault metrics into Prometheus you need to log in to Vault to get access to a native Vault endpoint that provides the metrics.
Workflow
The webhook injects vault-agent as an init container, based on the Kubernetes Auth role configuration prometheus-operator-prometheus.
The vault-agent grabs a token with the policy of prometheus-operator-prometheus.
consul-template runs as a sidecar, and uses the token from the previous step to retrieve a new token using the Token Auth role prometheus-metrics which has the policy prometheus-metrics applied to it.
Prometheus can now use this second token to read the Vault Prometheus endpoint.
The trick here is that Prometheus runs with the SecurityContext UID of 1000 but the default consul-template image is running under the UID of 100. This is because of a Dockerfile Volume that is configured which dockerd mounts as 100 (/consul-template/data).
Subsequently using this consul-template means it will never start, so we need to ensure we do not use this declared volume and change the UID using a custom Dockerfile and entrypoint.
Prerequisites
This document assumes you have a working Kubernetes cluster which has a:
#!/bin/dumb-init /bin/sh
set -ex
# Note above that we run dumb-init as PID 1 in order to reap zombie processes# as well as forward signals to all processes in its session. Normally, sh# wouldn't do either of these functions so we'd leak zombies as well as do# unclean termination of all our sub-processes.# CONSUL_DATA_DIR is exposed as a volume for possible persistent storage.# CT_CONFIG_DIR isn't exposed as a volume but you can compose additional config# files in there if you use this image as a base, or use CT_LOCAL_CONFIG below.CT_DATA_DIR=/consul-template/config
CT_CONFIG_DIR=/consul-template/config
# You can also set the CT_LOCAL_CONFIG environment variable to pass some# Consul Template configuration JSON without having to bind any volumes.if[ -n "$CT_LOCAL_CONFIG"];thenecho"$CT_LOCAL_CONFIG" > "$CT_CONFIG_DIR/local-config.hcl"fi# If the user is trying to run consul-template directly with some arguments, then# pass them to consul-template.if["${1:0:1}"='-'];thenset -- /bin/consul-template "$@"fi# If we are running Consul, make sure it executes as the proper user.if["$1"='/bin/consul-template'];then# Set the configuration directoryshiftset -- /bin/consul-template \
-config="$CT_CONFIG_DIR"\
"$@"# Check the user we are running ascurrent_user="$(id -un)"if["${current_user}"=="root"];then# Run under the right userset -- gosu consul-template "$@"fifiexec"$@"
# A YAML representation of a final vault config file.# See https://www.vaultproject.io/docs/configuration/ for more information.config:telemetry:prometheus_retention_time:30sdisable_hostname:true
Disable statsd:
# since we are running Vault 1.1.0 with the native Prometheus support, we do not need the statsD exporterstatsdDisabled:true
5.9 - Comparison of Banzai Cloud and HashiCorp mutating webhook for Vault
Legend
✅: Implemented
o: Planned/In-progress
Feature
Banzai Cloud Webhook
HashiCorp Webhook
Automated Vault and K8S setup
✅ (operator)
vault-agent/consul-template sidecar injection
✅
✅
Direct env var injection
✅
Injecting into K8S Secrets
✅
Injecting into K8S ConfigMaps
✅
Injecting into K8S CRDs
✅
Sidecar-less dynamic secrets
✅
CSI Driver
o
Native Kubernetes sidecar
o
5.10 - Running the webhook and Vault on different clusters
This section describes how to configure the webhook and Vault when the webhook runs on a different cluster from Vault, or if Vault runs outside Kubernetes.
Let’s suppose you have two different K8S clusters:
cluster1 contains vault-operator
cluster2 contains vault-secrets-webhook
Basically, you have to grant cluster2 access to the Vault running on cluster1. To achieve this, complete the following steps.
Extract the cluster.certificate-authority-data and the cluster.server fields from your cluster2 kubeconfig file. You will need them in the externalConfig section of the cluster1 configuration. For example:
You can use the vault ServiceAccount token as a token_reviewer_jwt in the auth configuration. To retrieve the token, run the following command:
kubectl get secret $(kubectl get sa vault -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}'| base64 --decode
In the vault.banzaicloud.com custom resource (for example, https://github.com/bank-vaults/bank-vaults/raw/main/operator/deploy/cr.yaml) of cluster1, define an externalConfig section. Fill the values of the kubernetes_ca_cert, kubernetes_host, and token_reviewer_jwt using the data collected in the previous steps.
externalConfig:policies:- name:allow_secretsrules:path "secret/*" {capabilities = ["create", "read", "update", "delete", "list"]}auth:- type:kubernetesconfig:token_reviewer_jwt:<token-for-cluster2-service-account>kubernetes_ca_cert:| -----BEGIN CERTIFICATE-----
<certificate-from-certificate-authority-data-on-cluster2>
-----END CERTIFICATE-----kubernetes_host:<cluster.server-field-on-cluster2>roles:# Allow every pod in the default namespace to use the secret kv store- name:defaultbound_service_account_names:["default","vault-secrets-webhook"]bound_service_account_namespaces:["default","vswh"]policies:allow_secretsttl:1h
In a production environment, it is highly recommended to specify TLS config for your Vault ingress.
# Request an Ingress controller with the default configurationingress:# Specify Ingress object annotations here, if TLS is enabled (which is by default)# the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations# to support TLS backendsannotations:# Override the default Ingress specification here# This follows the same format as the standard Kubernetes Ingress# See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#ingressspec-v1beta1-extensionsspec:tls:- hosts:- vault-dns-namesecretName:vault-ingress-tls-secret
Deploy the Vault custom resource containing the externalConfig section to cluster1:
kubectl apply -f your-proper-vault-cr.yaml
After Vault started in cluster1, you can use the vault-secrets-webhook in cluster2 with the proper annotations. For example:
For VAULT_AUTH_METHOD env var, these types: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure” are supported.
6 - Unseal keys
Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt data, allowing access to Vault. (From the official Vault documentation)
Vault starts in an uninitialized state, which means it has to be initialized with an initial set of parameters. The response to the init request is the root token and unseal keys. After that, Vault becomes initialized, but remains in a sealed state. A sealed state is a state in which no secrets can reach or leave Vault until a person, possibly more people than one, unseals it with the required number of unseal keys.
Vault data and the unseal keys live together: if you delete a Vault instance installed by the operator, or if you delete the Helm chart, all your data and the unseal keys to that initialized state should remain untouched. Read more about it in the official documentation.
The Bank-Vaults Init and Unseal process
Bank-Vaults runs in an endless loop and does the following:
Bank-Vaults checks if Vault is initialized. If yes, it continues to step 2, otherwise Bank-Vaults:
Calls Vault init, which returns the root token and the configured number of unseal keys.
Encrypts the received token and keys with the configured KMS key.
Stores the encrypted token and keys in the cloud provider’s object storage.
Flushes the root token and keys from its memory with explicit GC as soon as possible.
Bank-Vaults checks if Vault is sealed. If it isn’t, it continues to step 3, otherwise Bank-Vaults:
Reads the encrypted unseal keys from the cloud provider’s object storage.
Decrypts the unseal keys with the configured KMS key.
Unseals Vault with the decrypted unseal keys.
Flushes the keys from its memory with explicit GC as soon as possible.
Reads the encrypted root token from the cloud provider’s object storage.
Decrypts the root token with the configured KMS key.
Applies the parsed configuration on the Vault API
Flushes the root token from its memory with explicit GC as soon as possible.
Repeats from the second step after the configured time period.
Keys stored by Bank-Vaults
Bank-Vaults stores the following keys:
vault-root, which is the Vault’s root token
vault-unseal-N, where N is a number, starting at 0 up to the maximum defined minus 1, e.g. 5 unseal keys will be vault-unseal-0 up to including vault-unseal-4
HashiCorp recommends to revoke root tokens after the initial set up of Vault has been completed.
To unseal Vault, the vault-root token is not needed and can be removed from the storage if it was put there via the --init call to bank-vaults.
Decrypting root token
AWS
To use the KMS-encrypted root token with Vault CLI:
Required CLI tools:
aws
Steps:
Download and decrypt the root token (and the unseal keys, but that is not mandatory) into a file on your local file system:
There is a Kubernetes Secret backed unseal storage in Bank-Vaults, you should be aware of that Kubernetes Secrets are base64 encoded only if you are not using a EncryptionConfiguration in your Kubernetes cluster.
VAULT_NAME="vault"exportVAULT_TOKEN=$(kubectl get secrets ${VAULT_NAME}-unseal-keys -o jsonpath={.data.vault-root}| base64 -d)
Migrate unseal keys between cloud providers
If you need to move your Vault instance from one provider or an external managed Vault, you have to store those the unseal keys and a root token in the Bank-Vaults format.
All examples assume that you have created files holding the root-token and the 5 unseal keys in plaintext:
Bank-Vaults tries to automate as much as possible for handling TLS certificates.
The vault-operator automates the creation and renewal of TLS certificates for Vault.
The vault Helm Chart automates only the creation of TLS certificates for Vault via Sprig.
Both the operator and the chart generate a Kubernetes Secret holding the TLS certificates, this is named ${VAULT_CR_NAME}-tls. For most examples in the Bank-Vaults project repository, the name of the secret is vault-tls.
The Secret data keys are:
ca.crt
server.crt
server.key
Note: The operator doesn’t overwrite this Secret if it already exists, so you can provide this certificate in any other way, for example using cert-manager or by simply placing it there manually.
Operator custom TLS settings
The following attributes influence the TLS settings of the operator. The ca.crt key is mandatory in existingTlsSecretName, otherwise the Bank-Vaults components can’t verify the Vault server certificate.
CANamespaces
The list of namespaces where the generated CA certificate for Vault should be distributed. Use ["*"] for all namespaces.
Default value: []
ExistingTLSSecretName
The name of the secret that contains a TLS server certificate, key, and the corresponding CA certificate. The secret must be in the kubernetes.io/tls type secret keys + ca.crt key format. If the attribute is set, the operator uses the certificate already set in the secret, otherwise it generates a new one.
The ca.crt key is mandatory, otherwise the Bank-Vaults components can’t verify the Vault server certificate.
Default value: ""
TLSAdditionalHosts
A list hostnames or IP addresses to add to the SAN on the automatically generated TLS certificate.
Default value: []
TLSExpiryThreshold
The expiration threshold of the Vault TLS certificate in Go Duration format.
Default value: 168h
Using the generated custom TLS certificate with vault-operator
To use an existing secret which contains the TLS certificate, define existingTlsSecretName in the Vault custom resource.
Generate custom certificates with CFSSL
If you don’t want to use the certificates generated by Helm or the Bank-Vaults operator, the easiest way to create a custom certificate for Bank-Vaults is using CFSSL.
When using existing unseal keys, you need to make sure to kms encrypt these with the proper EncryptionContext.
If this is not done, the invocation of bank-vaults will trigger an InvalidCiphertextException from AWS KMS.
An example how to encrypt the keys (specify --profile and --region accordingly):
From this point on copy the encrypted files to the appropriate S3 bucket.
As an additional security measure make sure to turn on encryption of the S3 bucket before uploading the files.
Alibaba Cloud
A CLI example how to run bank-vaults based Vault unsealing on Alibaba Cloud:
You can configure the vault-operator to create backups of the Vault cluster with Velero.
Prerequisites
The Velero CLI must be installed on your computer.
To create Persistent Volume (PV) snapshots, you need access to an object storage. The following example uses an Amazon S3 bucket called bank-vaults-velero in the Stockholm region.
Install Velero
To configure the vault-operator to create backups of the Vault cluster, complete the following steps.
Note: The Vault CR in cr-raft.yaml has a special flag called veleroEnabled. This is useful for file-based Vault storage backends (file, raft), see the Velero documentation:
# Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:# https://velero.io/docs/v1.2.0/hooks/veleroEnabled:true
Create a backup with the Velero CLI or with the predefined Velero Backup CR:
Check that mesh is configured with mTLS turned on which applies to all applications in the cluster in Istio-enabled namespaces. You can change this if you would like to use another policy.
Now your cluster is properly running on Istio with mTLS enabled globally.
Install the Bank-Vaults components
You are recommended to create a separate namespace for Bank-Vaults called vault-system. You can enable Istio sidecar injection here as well, but Kubernetes won’t be able to call back the webhook properly since mTLS is enabled (and Kubernetes is outside of the Istio mesh). To overcome this, apply a PERMISSIVE Istio authentication policy to the vault-secrets-webhook Service itself, so Kubernetes can call it back without Istio mutual TLS authentication.
10.1 - Scenario 1 - Vault runs outside, the application inside the mesh
In this scenario, Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled.
Provision a Vault instance with the Bank-Vaults operator in a separate namespace:
kubectl create namespace vault
Apply the RBAC and CR files to the cluster to create a Vault instance in the vault namespace with the operator:
kubectl apply -f rbac.yaml -f cr-istio.yaml
kubectl get pods -n vault
Expected output:
NAME READY STATUS RESTARTS AGE
vault-0 3/3 Running 0 22h
vault-configurer-6458cc4bf-6tpkz 1/1 Running 0 22h
If you are writing your own Vault CR make sure that istioEnabled: true is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly.
The vault-secrets-webhook can’t inject Vault secrets into initContainers in an Istio-enabled namespace when the STRICT authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase when initContainers are running the Pod doesn’t have a sidecar yet.
If you wish to inject into initContainers as well, you need to apply a PERMISSIVE authentication policy in the vault namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).
In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:
Create a namespace first for the application and enable Istio sidecar injection:
Bank-Vaults offers a lot of alternatives for encrypting and storing the unseal-keys and the root-token for Vault. One of the encryption technics is the HSM - Hardware Security Module. HSM offers an industry-standard way to encrypt your data in on-premise environments.
You can use a Hardware Security Module (HSM) to generate and store the private keys used by Bank-Vaults. Some articles still point out the speed of HSM devices as their main selling point, but an average PC can do more cryptographic operations. Actually, the main benefit is from the security point of view. An HSM protects your private keys and handles cryptographic operations, which allows the encryption of protected information without exposing the private keys (they are not extractable). Bank-Vaults currently supports the PKCS11 software standard to communicate with an HSM. Fulfilling compliance requirements (for example, PCI DSS) is also a great benefit of HSMs, so from now on you can achieve that with Bank-Vaults.
Implementation in Bank-Vaults
To support HSM devices for encrypting unseal-keys and root-tokens, Bank-Vaults:
implements an encryption/decryption Service named hsm in the bank-vaults CLI,
the bank-vaults Docker image now includes the SoftHSM (for testing) and the OpenSC tooling,
the operator is aware of HSM and its nature.
The HSM offers an encryption mechanism, but the unseal-keys and root-token have to be stored somewhere after they got encrypted. Currently there are two possible solutions for that:
Some HSM devices can store a limited quantity of arbitrary data (like Nitrokey HSM), and Bank-Vaults can store the unseal-keys and root-token here.
If the HSM does not support that, Bank-Vaults uses the HSM to encrypt the unseal-keys and root-token, then stores them in Kubernetes Secrets. We believe that it is safe to store these keys in Kubernetes Secrets in encrypted format.
Bank-Vaults offers the ability to use the pre-created the cryptographic encryption keys on the HSM device, or generate a key pair on the fly if there isn’t any with the specified label in the specified slot.
Since Bank-Vaults is written in Go, it uses the github.com/miekg/pkcs11 wrapper to pull in the PKCS11 library, to be more precise the p11 high-level wrapper .
Supported HSM solutions
Bank-Vaults currently supports the following HSM solutions:
AWS CloudHSM supports the PKCS11 API as well, so it probably works, though it needs a custom Docker image.
11.1 - NitroKey HSM support (OpenSC)
Nitrokey HSM is a USB HSM device based on the OpenSC project. We are using NitroKey to develop real hardware-based HSM support for Bank-Vaults. This device is not a cryptographic accelerator, only key generation and the private key operations (sign and decrypt) are supported. Public key operations should be done by extracting the public key and working on the computer, and this is how it is implemented in Bank-Vaults. It is not possible to extract private keys from NitroKey HSM, the device is tamper-resistant.
This device supports only RSA based encryption/decryption, and thus this is implemented in Bank-Vaults currently. It supports ECC keys as well, but only for sign/verification operations.
To start using a NitroKey HSM, complete the following steps.
Install OpenSC and initialize the NitroKey HSM stick:
pkcs11-tool --list-objects
Using slot 0 with a present token (0x0)Public Key Object; RSA 2048 bits
label: bank-vaults
ID: a9548075b20243627e971873826ead172e932359
Usage: encrypt, verify, wrap
Access: none
pkcs15-tool --list-keys
Using reader with a card: Nitrokey Nitrokey HSM
Private RSA Key [bank-vaults] Object Flags : [0x03], private, modifiable
Usage : [0x0E], decrypt, sign, signRecover
Access Flags : [0x1D], sensitive, alwaysSensitive, neverExtract, local ModLength : 2048 Key ref : 1(0x01) Native : yes
Auth ID : 01 ID : a9548075b20243627e971873826ead172e932359
MD:guid : a6b2832c-1dc5-f4ef-bb0f-7b3504f67015
If you are testing the HSM on macOS, setup minikube. Otherwise, continue with the next step.
Configure the operator to use NitroKey HSM for unsealing.
You must adjust the unsealConfig section in the vault-operator configuration, so the operator can communicate with OpenSC HSM devices correctly. Adjust your configuration based on the following snippet:
# This example relies on an OpenSC HSM (NitroKey HSM) device initialized and plugged in to the Kubernetes Node.unsealConfig:hsm:# OpenSC daemon is needed in this case to communicate with the devicedaemon:true# The HSM SO module path (opensc is built into the bank-vaults image)modulePath:/usr/lib/opensc-pkcs11.so# For OpenSC slotId is the preferred way instead of tokenLabel# (OpenSC appends/prepends some extra stuff to labels)slotId:0pin:banzai# This can be specified in the BANK_VAULTS_HSM_PIN environment variable as well, from a SecretkeyLabel:bank-vaults
On macOS where you run Docker in VMs you need to do some extra steps before mounting your HSM device to Kubernetes.
Complete the following steps to mount NitroKey into the minikube Kubernetes cluster:
Make sure that the Oracle VM VirtualBox Extension Pack for USB 2.0 support is installed.
VBoxManage list extpacks
Remove the HSM device from your computer if it is already plugged in.
Specify VirtualBox as the VM backend for Minikube.
minikube config set vm-driver virtualbox
Create a minikube cluster with the virtualbox driver and stop it, so you can modify the VM.
minikube start
minikube stop
Enable USB 2.0 support for the minikube VM.
VBoxManage modifyvm minikube --usbehci on
Find the vendorid and productid for your Nitrokey HSM device.
VBoxManage list usbhost
VENDORID=0x20a0
PRODUCTID=0x4230
Now your minikube Kubernetes cluster has access to the HSM device through USB.
Kubernetes node setup
Some HSM vendors offer network daemons to enhance the reach of their HSM equipment to different servers. Unfortunately, there is no networking standard defined for PKCS11 access and thus currently Bank-Vaults has to be scheduled to the same node where the HSM device is attached directly (if not using a Cloud HSM).
Since the HSM is a hardware device connected to a physical node, Bank-Vaults has to find its way to that node. To make this work, create an HSM extended resource on the Kubernetes nodes for which the HSM device is plugged in. Extended resources must be advertised in integer amounts, for example, a Node can advertise four HSM devices, but not 4.5.
You need to patch the node to specify that it has an HSM device as a resource.
Because of the integer constraint and because all Bank-Vaults related Pods has to land on a Node where an HSM resource is available we need to advertise two units for 1 device, one will be allocated by each Vault Pod and one by the Configurer. If you would like to run Vault in HA mode - multiple Vault instances in different nodes - you will need multiple HSM devices plugged into those nodes, having the same key and slot setup.
From now on, you can request the nitrokey.com/hsm resource in the PodSpec
Include the nitrokey.com/hsm resource in your PodSpec:
# If using the NitroKey HSM example, that resource has to be part of the resource scheduling request.resources:hsmDaemon:requests:cpu:100mmemory:64Minitrokey.com/hsm:1limits:cpu:200mmemory:128Minitrokey.com/hsm:1
Check the logs that unsealing uses the NitroKey HSM device. Run the following command:
kubectl logs -f vault-0 bank-vaults
The output should be something like:
time="2020-03-04T13:32:29Z"level=info msg="HSM Information {CryptokiVersion:{Major:2 Minor:20} ManufacturerID:OpenSC Project Flags:0 LibraryDescription:OpenSC smartcard framework LibraryVersion:{Major:0 Minor:20}}"time="2020-03-04T13:32:29Z"level=info msg="HSM Searching for slot in HSM slots [{ctx:0xc0000c0318 id:0}]"time="2020-03-04T13:32:29Z"level=info msg="found HSM slot 0 in HSM by slot ID"time="2020-03-04T13:32:29Z"level=info msg="HSM TokenInfo {Label:bank-vaults (UserPIN)\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ManufacturerID:www.CardContact.de Model:PKCS#15 emulated SerialNumber:DENK0200074 Flags:1037 MaxSessionCount:0 SessionCount:0 MaxRwSessionCount:0 RwSessionCount:0 MaxPinLen:15 MinPinLen:6 TotalPublicMemory:18446744073709551615 FreePublicMemory:18446744073709551615 TotalPrivateMemory:18446744073709551615 FreePrivateMemory:18446744073709551615 HardwareVersion:{Major:24 Minor:13} FirmwareVersion:{Major:3 Minor:3} UTCTime:\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}"time="2020-03-04T13:32:29Z"level=info msg="HSM SlotInfo for slot 0: {SlotDescription:Nitrokey Nitrokey HSM (DENK02000740000 ) 00 00 ManufacturerID:Nitrokey Flags:7 HardwareVersion:{Major:0 Minor:0} FirmwareVersion:{Major:0 Minor:0}}"time="2020-03-04T13:32:29Z"level=info msg="found objects with label \"bank-vaults\" in HSM"time="2020-03-04T13:32:29Z"level=info msg="this HSM device doesn't support encryption, extracting public key and doing encrytion on the computer"time="2020-03-04T13:32:29Z"level=info msg="no storage backend specified for HSM, using on device storage"time="2020-03-04T13:32:29Z"level=info msg="joining leader vault..."time="2020-03-04T13:32:29Z"level=info msg="vault metrics exporter enabled: :9091/metrics"[GIN-debug][WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: exportGIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)[GIN-debug] Listening and serving HTTP on :9091
time="2020-03-04T13:32:30Z"level=info msg="initializing vault..."time="2020-03-04T13:32:30Z"level=info msg="initializing vault"time="2020-03-04T13:32:31Z"level=info msg="unseal key stored in key store"key=vault-unseal-0
time="2020-03-04T13:32:31Z"level=info msg="unseal key stored in key store"key=vault-unseal-1
time="2020-03-04T13:32:32Z"level=info msg="unseal key stored in key store"key=vault-unseal-2
time="2020-03-04T13:32:32Z"level=info msg="unseal key stored in key store"key=vault-unseal-3
time="2020-03-04T13:32:33Z"level=info msg="unseal key stored in key store"key=vault-unseal-4
time="2020-03-04T13:32:33Z"level=info msg="root token stored in key store"key=vault-root
time="2020-03-04T13:32:33Z"level=info msg="vault is sealed, unsealing"time="2020-03-04T13:32:39Z"level=info msg="successfully unsealed vault"
Find the unseal keys and the root token on the HSM:
pkcs11-tool --list-objects
Expected output:
Using slot 0 with a present token (0x0)Public Key Object; RSA 2048 bits
label: bank-vaults
ID: a9548075b20243627e971873826ead172e932359
Usage: encrypt, verify, wrap
Access: none
Data object 2168561792 label: 'vault-test' application: 'vault-test' app_id: <empty>
flags: modifiable
Data object 2168561168 label: 'vault-unseal-0' application: 'vault-unseal-0' app_id: <empty>
flags: modifiable
Data object 2168561264 label: 'vault-unseal-1' application: 'vault-unseal-1' app_id: <empty>
flags: modifiable
Data object 2168561360 label: 'vault-unseal-2' application: 'vault-unseal-2' app_id: <empty>
flags: modifiable
Data object 2168562304 label: 'vault-unseal-3' application: 'vault-unseal-3' app_id: <empty>
flags: modifiable
Data object 2168562400 label: 'vault-unseal-4' application: 'vault-unseal-4' app_id: <empty>
flags: modifiable
Data object 2168562496 label: 'vault-root' application: 'vault-root' app_id: <empty>
flags: modifiable
If you don’t need the encryption keys or the unseal keys on the HSM anymore, you can delete them with the following commands:
PIN=banzai
# Delete the unseal keys and the root tokenfor label in "vault-test""vault-root""vault-unseal-0""vault-unseal-1""vault-unseal-2""vault-unseal-3""vault-unseal-4"do pkcs11-tool --delete-object --type data --label ${label} --pin ${PIN}done# Delete the encryption keypkcs11-tool --delete-object --type privkey --label bank-vaults --pin ${PIN}
11.2 - SoftHSM support for testing
You can use SoftHSMv2 to implement and test software interacting with PKCS11 implementations. You can install it on macOS by running the following commands:
# Initializing SoftHSM to be able to create a working example (only for dev),# sharing the HSM device is emulated with a pre-created keypair in the image.brew install softhsm
softhsm2-util --init-token --free --label bank-vaults --so-pin banzai --pin banzai
pkcs11-tool --module /usr/local/lib/softhsm/libsofthsm2.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults
To interact with SoftHSM when using the vault-operator, include the following unsealConfig snippet in the Vault CR:
# This example relies on the SoftHSM device initialized in the Docker image.unsealConfig:hsm:# The HSM SO module path (softhsm is built into the bank-vaults image)modulePath:/usr/lib/softhsm/libsofthsm2.so tokenLabel:bank-vaultspin:banzaikeyLabel:bank-vaults
To run the whole SoftHSM based example in Kubernetes, run the following commands:
If the configuration is updated Vault will be reconfigured
It supports configuring Vault secret engines, plugins, auth methods, and policies
The bank-vaults CLI command needs certain cloud permissions to function properly (init, unseal, configuration).
13 - The Go library
The Bank-Vaults repository contains several Go packages for interacting with Vault, these packages are organized into the sdk Go module, which can be pulled in with go get github.com/bank-vaults/bank-vaults/pkg/sdk and is versioned by the pkg/sdk/vX.Y.Z Git tags:
A helper for creating database source strings (MySQL/PostgreSQL) with database credentials dynamically based on configured Vault roles (instead of username:password).
At Banzai Cloud we prefer Prometheus for monitoring and use it also for Vault. If you configure, Vault can expose metrics through statsd. Both the Helm chart and the Vault Operator installs the Prometheus StatsD exporter and annotates the pods correctly with Prometheus annotations so Prometheus can discover and scrape them. All you have to do is to put the telemetry stanza into your Vault configuration:
These labels are common to all Vault Configurer Deployment Created resources
Vault Configurer Deployment
Vault Configurer Pod
Vault Configurer Service
16 - Watching External Secrets
In some cases, you might have to restart the Vault Statefulset when secrets that are not managed by the operator control are changed. For example:
Cert-Manager managing a public Certificate for vault using Let’s Encrypt.
Cloud IAM Credentials created with an external tool (like terraform) to allow vault to interact with the cloud services.
The operator can watch a set of secrets in the namespace of the Vault resource using either a list of label selectors or an annotations selector. When the content of any of those secrets changes, the operator updates the statefulset and triggers a rolling restart.
Configure label selectors
Set the secrets to watch using the watchedSecretsAnnotations and watchedSecretsLabels fields in your Vault custom resource.
Note: For cert-manager 0.11 or newer, use the watchedSecretsAnnotations field.
In the following example, the Vault StatefulSet is restarted when:
A secret with label certmanager.k8s.io/certificate-name: vault-letsencrypt-cert changes its contents (cert-manager 0.10 and earlier).
A secret with label test.com/scope: gcp AND test.com/credentials: vault changes its contents.
A secret with annotation cert-manager.io/certificate-name: vault-letsencrypt-cert changes its contents (cert-manager 0.11 and newer).
The following section lists some questions, problems, and tips that came up on the Bank-Vaults Slack channel.
Login to the Vault web UI
To login to the Vault web UI, you can use the root token, or any configured authentication backend.
Can changing the vault CR delete the Vault instance and data?
Bank-Vaults never ever deletes the Vault instance from the cluster. However, if you delete the Vault CR, then the Kubernetes garbage controller deletes the vault pods. You are recommended to keep backups.
Set default for vault.security.banzaicloud.io/vault-addr
Bank-Vaults is a Vault swiss-army knife: a K8s operator, Go client with automatic token renewal, automatic configuration, multiple unseal options and more. A CLI tool to init, unseal and configure Vault (auth methods, secret engines). Direct secret injection into Pods.
Community support
If you encounter problems while using Bank-Vaults that the documentation does not address, open an issue or talk to us on the Emerging Tech Community Slack channel #bank-vaults.
19 - Contributing
If you find this project useful here’s how you can help:
Now you have a running vault server which is uninitialized and unsealed you can init and unseal it with bank-vaults cli-tool and unseal keys will be stored to a local file:
Developing the operator requires a working Kubernetes cluster, minikube and Docker for Mac Kubernetes will suffice.
The operator consists of two parts, the bank-vaults sidecar running inside a container and the operator itself.
You can fire up the operator on your machine, so you can debug it locally (yes you don’t have to build a container from it), if your kube context points to the development cluster:
make operator-up
This installs all the necessary RBAC rules and other CRDs that you need to create a Vault instance. If you change the code of the operator you have to CTRL + C this make command and rerun it again.
Now it is time create a Vault instance for yourself, which you can work on:
kubectl apply -f operator/deploy/cr.yaml
If you change the bank-vaults sidecar code you have to build a new Docker image from it:
DOCKER_LATEST=1 make docker
There are at least four ways to distribute this image in your Kubernetes cluster, by default IfNotPresent image pull policy is used:
If you are using Docker for Mac, you don’t have to anything, the Kubernetes cluster and your host shares the same Docker daemon.
If you are using Minikube with --vm-driver=none (you are probably using Linux) the same applies as for Docker for Mac
If you are using Minikube with some real vm-driver you have to run eval $(minikube docker-env) before building the Docker image with the make command so you build it with the minikube Docker daemon and the image will be stored there
Build and re-tag the image and push it to the Docker registry of your choice, dont forget to change the bankVaultsImage attribute in the the Vault Custom Resource YAML file (cr.yaml in this case).
Restart the containers using the bank-vaults image: Vault instances and the configurer.
WebHook
This will deploy the webhook via the Helm chart, scale it to 0, start it locally and proxy it into the cluster (somehow similar to operator-up but a bit more complex).
You will need Helm and kuruninstalled to run this:
