Documentation

• 1: Getting started
• 2: Concepts
• 2.1: Initialize Vault and store the root token and unseal keys
• 2.1.1: Decrypt the root token
• 2.1.2: Migrate unseal keys between cloud providers
• 2.2: Cloud permissions
• 2.3: External configuration for Vault
• 2.3.1: Fully or partially purging unmanaged configuration in Vault
• 2.3.2: Audit devices
• 2.3.3: Authentication
• 2.3.4: Plugins
• 2.3.5: Policies
• 2.3.6: Secrets engines
• 2.3.7: Startup secrets
• 3: Vault operator
• 3.1: Running Vault with external end to end encryption
• 3.2: Using templates for injecting dynamic configuration
• 3.3: Environment variables
• 3.4: Upgrade strategies
• 3.5: Operator Configuration for Functioning Webhook Secrets Mutation
• 3.6: TLS
• 3.7: Backing up Vault
• 3.8: Running the Bank-Vaults secret webhook alongside Istio
• 3.8.1: Scenario 1 - Vault runs outside, the application inside the mesh
• 3.8.2: Scenario 2 - Running Vault inside the mesh
• 3.8.3: Scenario 3 - Both Vault and the app are running inside the mesh
• 3.9: HSM Support
• 3.9.1: NitroKey HSM support (OpenSC)
• 3.9.2: SoftHSM support for testing
• 3.10: Monitoring
• 3.11: Annotations and labels
• 3.12: Watching External Secrets
• 3.13: API Reference
• 4: Secret injection webhook
• 4.1: Deploy the webhook
• 4.2: Configuration examples and scenarios
• 4.3: Annotations
• 4.4: Using Vault Agent Templating in the mutating webhook
• 4.5: Using consul-template in the mutating webhook
• 4.6: Transit Encryption
• 4.7: Monitoring the Webhook with Grafana and Prometheus
• 4.8: Injecting consul-template into the Prometheus operator for Vault metrics
• 4.9: Comparison of Banzai Cloud and HashiCorp mutating webhook for Vault
• 4.10: Running the webhook and Vault on different clusters
• 5: bank-vaults CLI
• 6: The Go library
• 7: Tips and tricks
• 7.1: Deploy vault into a custom namespace
• 8: Support
• 9: Contributing guide
• 10: Maintainer guide
• 11: Development
• 12: Security procedures
• 13: Code of Conduct

1 - Getting started

Bank-Vaults is a swiss-army knife with multiple manifestations, so the first steps depend on what you want to achieve. Check one of the following guides to get an overview:

• Dynamic credentials with Vault using Kubernetes Service Accounts
• Vault Operator
• Vault unseal flow with KMS
• Inject secrets directly into pods from Vault

Deploy with Helm

We have some fully fledged, production-ready Helm charts for deploying:

• Vault using bank-vaults,
• the Vault Operator, and also
• the Vault Secrets Webhook.

With the help of these charts you can run a HA Vault instance with automatic initialization, unsealing, and external configuration which would otherwise be a tedious manual operation. Also secrets from Vault can be injected into your Pods directly as environment variables (without using Kubernetes Secrets). These charts can be used easily for development purposes as well.

Note: Starting with Bank-Vaults version 1.6.0, only Helm 3 is supported.

Deploy a local Vault operator

This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets. If you want to customize the Helm chart, see the list of vault-operator Helm chart values.

1. Install the Bank-Vaults operator:

   helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
   

   Expected output:

   Release "vault-operator" does not exist. Installing it now.
   Pulled: ghcr.io/bank-vaults/helm-charts/vault-operator:1.20.0
   Digest: sha256:46045be1c3b215f0c734908bb1d4022dc91eae48d2285382bb71d63f72c737d1
   NAME: vault-operator
   LAST DEPLOYED: Thu Jul 27 11:22:55 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

2. Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
       

   Expected output:

   serviceaccount/vault created
       role.rbac.authorization.k8s.io/vault created
       role.rbac.authorization.k8s.io/leader-election-role created
       rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
       rolebinding.rbac.authorization.k8s.io/vault created
       clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
       

   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml
       

   Expected output:

   vault.vault.banzaicloud.com/vault created
       

   Note: If needed, you can install the latest CustomResource from the main branch, but that’s usually under development and might not be stable.

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/crd | kubectl apply -f -
       

3. Wait a few seconds, then check the operator and the vault pods:

   kubectl get pods
   

   Expected output:

   NAME                                                        READY     STATUS    RESTARTS   AGE
   vault-0                                                     3/3       Running   0          10s
   vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
   vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s
   

4. Configure your Vault client to access the Vault instance running in the vault-0 pod.

   1. Port-forward into the pod:

      kubectl port-forward vault-0 8200 &
      

   2. Set the address of the Vault instance.

      export VAULT_ADDR=https://127.0.0.1:8200
      

   3. Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):

      kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
      export VAULT_CACERT=$PWD/vault-ca.crt
      

      Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true

   4. If you already have the Vault CLI installed, check that you can access the Vault:

      vault status
      

      Expected output:

      Key             Value
      ---             -----
      Seal Type       shamir
      Initialized     true
      Sealed          false
      Total Shares    5
      Threshold       3
      Version         1.5.4
      Cluster Name    vault-cluster-27ecd0e6
      Cluster ID      ed5492f3-7ef3-c600-aef3-bd77897fd1e7
      HA Enabled      false
      

   5. To authenticate to Vault, you can access its root token by running:

      export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
      

      Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.

   6. Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).

For other configuration examples of the Vault CustomResource, see the YAML files in the deploy/examples and test/deploy directories of the vault-operator repository. After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:

kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl delete -f -
kubectl delete -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml

Deploy the mutating webhook

You can deploy the Vault Secrets Webhook using Helm. Note that:

• The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
• The deployed RBAC objects contain the necessary permissions fo running the webhook.

Prerequisites

• The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
• The chart requires Helm 3.
• To interact with Vault (for example, for testing), the Vault command line client must be installed on your computer.
• You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.

Deploy the webhook

1. Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:

   kubectl create namespace vault-infra
   kubectl label namespace vault-infra name=vault-infra
   

2. Deploy the vault-secrets-webhook chart. If you want to customize the Helm chart, see the list of vault-secrets-webhook Helm chart values.

   helm upgrade --install --wait vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook --namespace vault-infra
   

   Expected output:

   Release "vault-secrets-webhook" does not exist. Installing it now.
   NAME: vault-secrets-webhook
   LAST DEPLOYED: Fri Jul 14 15:42:36 2023
   NAMESPACE: vault-infra
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

   For further details, see the webhook’s Helm chart repository.

3. Check that the pods are running:

   kubectl get pods --namespace vault-infra
   

   Expected output:

   NAME                                     READY   STATUS    RESTARTS   AGE
   vault-secrets-webhook-58b97c8d6d-qfx8c   1/1     Running   0          22s
   vault-secrets-webhook-58b97c8d6d-rthgd   1/1     Running   0          22s
   

4. If you already have the Vault CLI installed, write a secret into Vault:

   vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
   

   Expected output:

   Key              Value
   ---              -----
   created_time     2020-11-04T11:39:01.863988395Z
   deletion_time    n/a
   destroyed        false
   version          1
   

5. Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:

       kubectl create -f - <<EOF
       apiVersion: apps/v1
       kind: Deployment
       metadata:
         name: vault-test
       spec:
         replicas: 1
         selector:
           matchLabels:
             app.kubernetes.io/name: vault
         template:
           metadata:
             labels:
               app.kubernetes.io/name: vault
             annotations:
               vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200
               vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"
               vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate
               vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optional, the name of the Secret where the Vault CA cert is, if not defined it is not mounted
               vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication
               vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"
           spec:
             serviceAccountName: default
             containers:
             - name: alpine
               image: alpine
               command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
               env:
               - name: AWS_SECRET_ACCESS_KEY
                 value: vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
       EOF
       

   Expected output:

   deployment.apps/vault-test created
   

6. Check the mutated deployment.

   kubectl describe deployment vault-test
   

   The output should look similar to the following:

   Name:                   vault-test
   Namespace:              default
   CreationTimestamp:      Wed, 04 Nov 2020 12:44:18 +0100
   Labels:                 <none>
   Annotations:            deployment.kubernetes.io/revision: 1
   Selector:               app.kubernetes.io/name=vault
   Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
   StrategyType:           RollingUpdate
   MinReadySeconds:        0
   RollingUpdateStrategy:  25% max unavailable, 25% max surge
   Pod Template:
     Labels:           app.kubernetes.io/name=vault
     Annotations:      vault.security.banzaicloud.io/vault-addr: https://vault:8200
                       vault.security.banzaicloud.io/vault-agent: false
                       vault.security.banzaicloud.io/vault-path: kubernetes
                       vault.security.banzaicloud.io/vault-role: default
                       vault.security.banzaicloud.io/vault-skip-verify: false
                       vault.security.banzaicloud.io/vault-tls-secret: vault-tls
     Service Account:  default
     Containers:
      alpine:
       Image:      alpine
       Port:       <none>
       Host Port:  <none>
       Command:
         sh
         -c
         echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000
       Environment:
         AWS_SECRET_ACCESS_KEY:  vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
       Mounts:                   <none>
     Volumes:                    <none>
   Conditions:
     Type           Status  Reason
     ----           ------  ------
     Available      True    MinimumReplicasAvailable
     Progressing    True    NewReplicaSetAvailable
   OldReplicaSets:  <none>
   NewReplicaSet:   vault-test-55c569f9 (1/1 replicas created)
   Events:
     Type    Reason             Age   From                   Message
     ----    ------             ----  ----                   -------
     Normal  ScalingReplicaSet  29s   deployment-controller  Scaled up replica set vault-test-55c569f9 to 1
   

   As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.

Install the CLI tool

You can download the bank-vaults CLI from the Bank-Vaults releases page. Select the binary for your platform from the Assets section for the version you want to use.

Alternatively, fetch the source code and compile it using go get:

go get github.com/bank-vaults/bank-vaults/cmd/bank-vaults
go get github.com/bank-vaults/bank-vaults/cmd/vault-env

Docker images

If you want to build upon our Docker images, you can find them on Docker Hub:

docker pull ghcr.io/bank-vaults/vault-operator:latest
docker pull ghcr.io/bank-vaults/vault-env:latest

2 - Concepts

The following sections give you an overview of the main concepts of Bank-Vaults. Most of these apply equally to the bank-vaults CLI and to the Vault operator, because under the hood the operator often uses the CLI tool with the appropriate parameters.

2.1 - Initialize Vault and store the root token and unseal keys

Vault starts in an uninitialized state, which means it has to be initialized with an initial set of parameters. The response to the init request is the root token and unseal keys. After that, Vault becomes initialized, but remains in a sealed state.

Bank-Vaults stores the root token and the unseal keys in one of the following:

• AWS KMS keyring (backed by S3)
• Azure Key Vault
• Google Cloud KMS keyring (backed by GCS)
• Alibaba Cloud KMS (backed by OSS)

For development and testing purposes, the following solutions are also supported. Do not use these in production environments.

• Kubernetes Secrets (should be used only for development purposes)
• Dev Mode (useful for vault server -dev dev mode Vault servers)
• Files (backed by files, should be used only for development purposes)

Keys stored by Bank-Vaults

Bank-Vaults stores the following keys:

• vault-root, which is Vault’s root token.
• vault-unseal-N unseal keys, where N is a number, starting at 0 up to the maximum defined minus 1. For example, 5 unseal keys will be vault-unseal-0 ... vault-unseal-4.

HashiCorp recommends revoking the root tokens after the initial set up of Vault has been completed.

Note: The vault-root token is not needed to unseal Vault, and can be removed from the storage if it was put there via the --init call to bank-vaults.

If you want to decrypt the root token for some reason, see Decrypt the root token.

Unseal Vault

Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt data, allowing access to Vault. (From the official Vault documentation)

After initialization, Vault remains in a sealed state. In sealed state no secrets can reach or leave Vault until a person, possibly more people than one, unseals it with the required number of unseal keys.

Vault data and the unseal keys live together: if you delete a Vault instance installed by the operator, or if you delete the Helm chart, all your data and the unseal keys to that initialized state should remain untouched. For details, see the official documentation.

Note: If you change the unseal configuration after initializing Vault, you may have to move the unseal keys from the old location to the new one, or reinitialize vault.

The Bank-Vaults Init and Unseal process

Bank-Vaults runs in an endless loop and does the following:

1. Bank-Vaults checks if Vault is initialized. If yes, it continues to step 2, otherwise Bank-Vaults:
   1. Calls Vault init, which returns the root token and the configured number of unseal keys.
   2. Encrypts the received token and keys with the configured KMS key.
   3. Stores the encrypted token and keys in the cloud provider’s object storage.
   4. Flushes the root token and keys from its memory with explicit garbage control as soon as possible.
2. Bank-Vaults checks if Vault is sealed. If it isn’t, it continues to step 3, otherwise Bank-Vaults:
   1. Reads the encrypted unseal keys from the cloud provider’s object storage.
   2. Decrypts the unseal keys with the configured KMS key.
   3. Unseals Vault with the decrypted unseal keys.
   4. Flushes the keys from its memory with explicit garbage control as soon as possible.
3. If the external configuration file was changed and an OS signal is received, then Bank-Vaults:
   1. Parses the configuration file.
   2. Reads the encrypted root token from the cloud provider’s object storage.
   3. Decrypts the root token with the configured KMS key.
   4. Applies the parsed configuration on the Vault API.
   5. Flushes the root token from its memory with explicit garbage control as soon as possible.
4. Repeats from the second step after the configured time period.

2.1.1 - Decrypt the root token

If you want to decrypt the root token for some reason, see the section corresponding to the storage provider you used to store the token.

AWS

To use the KMS-encrypted root token with Vault CLI:

Required CLI tools:

• aws

Steps:

1. Download and decrypt the root token (and the unseal keys, but that is not mandatory) into a file on your local file system:

   BUCKET=bank-vaults-0
   REGION=eu-central-1
   
   for key in "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
   do
       aws s3 cp s3://${BUCKET}/${key} .
   
       aws kms decrypt \
           --region ${REGION} \
           --ciphertext-blob fileb://${key} \
           --encryption-context Tool=bank-vaults \
           --output text \
           --query Plaintext | base64 -d > ${key}.txt
   
       rm ${key}
   done
   

2. Save it as an environment variable:

   export VAULT_TOKEN="$(cat vault-root.txt)"
   

Google Cloud

To use the KMS-encrypted root token with vault CLI:

Required CLI tools:

• gcloud
• gsutil

GOOGLE_PROJECT="my-project"
GOOGLE_REGION="us-central1"
BUCKET="bank-vaults-bucket"
KEYRING="beta"
KEY="beta"

export VAULT_TOKEN=$(gsutil cat gs://${BUCKET}/vault-root | gcloud kms decrypt \
                     --project ${GOOGLE_PROJECT} \
                     --location ${GOOGLE_REGION} \
                     --keyring ${KEYRING} \
                     --key ${KEY} \
                     --ciphertext-file - \
                     --plaintext-file -)

Kubernetes

There is a Kubernetes Secret backed unseal storage in Bank-Vaults, you should be aware of that Kubernetes Secrets are base64 encoded only if you are not using a EncryptionConfiguration in your Kubernetes cluster.

VAULT_NAME="vault"

export VAULT_TOKEN=$(kubectl get secrets ${VAULT_NAME}-unseal-keys -o jsonpath={.data.vault-root} | base64 -d)

2.1.2 - Migrate unseal keys between cloud providers

Note: If you change the unseal configuration after initializing Vault, you may have to move the unseal keys from the old location to the new one, or reinitialize vault.

If you need to move your Vault instance from one provider or an external managed Vault, you have to:

1. Retrieve and decrypt the unseal keys (and optionally the root token) in the Bank-Vaults format. For details, see Decrypt the root token.
2. Migrate the Vault storage data to the new provider. Use the official migration command provided by Vault.

All examples assume that you have created files holding the root-token and the 5 unseal keys in plaintext:

• vault-root.txt
• vault-unseal-0.txt
• vault-unseal-1.txt
• vault-unseal-2.txt
• vault-unseal-3.txt
• vault-unseal-4.txt

AWS

Move the above mentioned files to an AWS bucket and encrypt them with KMS before:

REGION=eu-central-1
KMS_KEY_ID=02a2ba49-42ce-487f-b006-34c64f4b760e
BUCKET=bank-vaults-1

for key in "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
do
    aws kms encrypt \
        --region ${REGION} --key-id ${KMS_KEY_ID} \
        --plaintext fileb://${key}.txt \
        --encryption-context Tool=bank-vaults \
        --output text \
        --query CiphertextBlob | base64 -d > ${key}

    aws s3 cp ./${key} s3://${BUCKET}/

    rm ${key} ${key}.txt
done

2.2 - Cloud permissions

The operator and the bank-vaults CLI command needs certain cloud permissions to function properly (init, unseal, configuration).

Google Cloud

The Service Account in which the Pod is running has to have the following IAM Roles:

• Cloud KMS Admin
• Cloud KMS CryptoKey Encrypter/Decrypter
• Storage Admin

A CLI example how to run bank-vaults based Vault configuration on Google Cloud:

bank-vaults configure --google-cloud-kms-key-ring vault --google-cloud-kms-crypto-key bank-vaults --google-cloud-kms-location global --google-cloud-storage-bucket vault-ha --google-cloud-kms-project continual-flow-276578

Azure

The Access Policy in which the Pod is running has to have the following IAM Roles:

• Key Vault All Key permissions
• Key Vault All Secret permissions

AWS

Enable IAM OIDC provider for an EKS cluster

To allow Vault pods to assume IAM roles in order to access AWS services the IAM OIDC provider needs to be enabled on the cluster.

BANZAI_CURRENT_CLUSTER_NAME="mycluster"

# Enable OIDC provider for the cluster with eksctl
# Follow the docs here to do it manually https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
eksctl utils associate-iam-oidc-provider \
    --cluster ${BANZAI_CURRENT_CLUSTER_NAME} \
    --approve

# Create a KMS key and S3 bucket and enter details here
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
REGION="eu-west-1"
KMS_KEY_ID="9f054126-2a98-470c-9f10-9b3b0cad94a1"
KMS_KEY_ARN="arn:aws:kms:${REGION}:${AWS_ACCOUNT_ID}:key/${KMS_KEY_ID}"
BUCKET="bank-vaults"
OIDC_PROVIDER=$(aws eks describe-cluster --name ${BANZAI_CURRENT_CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
SERVICE_ACCOUNT_NAME="vault"
SERVICE_ACCOUNT_NAMESPACE="vault"

cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

cat > vault-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt"
            ],
            "Resource": [
                "${KMS_KEY_ARN}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::${BUCKET}"
        }
    ]
}
EOF

# AWS IAM role and Kubernetes service account setup
aws iam create-role --role-name vault --assume-role-policy-document file://trust.json
aws iam create-policy --policy-name vault --policy-document file://vault-policy.json
aws iam attach-role-policy --role-name vault --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/vault

# If you are having a ServiceAccount already, only the annotation is needed
kubectl create serviceaccount $SERVICE_ACCOUNT_NAME --namespace $SERVICE_ACCOUNT_NAMESPACE
kubectl annotate serviceaccount $SERVICE_ACCOUNT_NAME  --namespace $SERVICE_ACCOUNT_NAMESPACE eks.amazonaws.com/role-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:role/vault"

# Cleanup
rm vault-policy.json trust.json

Getting the root token

After Vault is successfully deployed, you can query the root-token for admin access.

# Fetch Vault root token, check bucket for actual name based on unsealConfig.aws.s3Prefix
aws s3 cp s3://$s3_bucket_name/vault-root /tmp/vault-root

export VAULT_TOKEN="$(aws kms decrypt \
  --ciphertext-blob fileb:///tmp/vault-root \
  --encryption-context Tool=bank-vaults \
  --query Plaintext --output text | base64 --decode)"

The Instance profile in which the Pod is running has to have the following IAM Policies:

• KMS: kms:Encrypt, kms:Decrypt
• S3: s3:GetObject, s3:PutObject on object level and s3:ListBucket on bucket level

An example command how to init and unseal Vault on AWS:

bank-vaults unseal --init --mode aws-kms-s3 --aws-kms-key-id 9f054126-2a98-470c-9f10-9b3b0cad94a1 --aws-s3-region eu-west-1 --aws-kms-region eu-west-1 --aws-s3-bucket bank-vaults

When using existing unseal keys, you need to make sure to kms encrypt these with the proper EncryptionContext. If this is not done, the invocation of bank-vaults will trigger an InvalidCiphertextException from AWS KMS. An example how to encrypt the keys (specify --profile and --region accordingly):

aws kms encrypt --key-id "alias/kms-key-alias" --encryption-context "Tool=bank-vaults"  --plaintext fileb://vault-unseal-0.txt --output text --query CiphertextBlob | base64 -D > vault-unseal-0

From this point on copy the encrypted files to the appropriate S3 bucket. As an additional security measure make sure to turn on encryption of the S3 bucket before uploading the files.

Alibaba Cloud

A CLI example how to run bank-vaults based Vault unsealing on Alibaba Cloud:

bank-vaults unseal --mode alibaba-kms-oss --alibaba-access-key-id ${ALIBABA_ACCESS_KEY_ID} --alibaba-access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} --alibaba-kms-region eu-central-1 --alibaba-kms-key-id ${ALIBABA_KMS_KEY_UUID} --alibaba-oss-endpoint oss-eu-central-1.aliyuncs.com --alibaba-oss-bucket bank-vaults

Kubernetes

The Service Account in which the bank-vaults Pod is running has to have the following Roles rules:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs:     ["get", "create", "update"]

2.3 - External configuration for Vault

In addition to the standard Vault configuration, the operator and CLI can continuously configure Vault using an external YAML/JSON configuration. That way you can configure Vault declaratively using your usual automation tools and workflow.

The following sections describe the configuration sections you can use.

2.3.1 - Fully or partially purging unmanaged configuration in Vault

Bank-Vaults gives you a full control over Vault in a declarative style by removing any unmanaged configuration.

By enabling purgeUnmanagedConfig you keep Vault configuration up-to-date. So if you added a policy using Bank-Vaults then removed it from the configuration, Bank-Vaults will remove it from Vault too. In other words, if you enabled purgeUnmanagedConfig then any changes not in Bank-Vaults configuration will be removed (including manual changes).

WARNING:

This feature is destructive, so be careful when you enable it especially for the first time because it can delete all data in your Vault. Always test it a non-production environment first.

This feature is disabled by default and it needs to be enabled explicitly in your configuration.

Mechanism

Bank-Vaults handles unmanaged configuration by simply comparing what in Bank-Vaults configuration (the desired state) and what’s already in Vault (the actual state), then it removes any differences that are not in Bank-Vaults configuration.

Fully purge unmanaged configuration

You can remove all unmanaged configuration by enabling the purge option as following:

purgeUnmanagedConfig:
  enabled: true

Partially purge unmanaged configuration

You can also enable the purge feature for some of the config by excluding any config that you don’t want to purge its unmanaged config.

It could be done by explicitly exclude the Vault configuration that you don’t want to mange:

purgeUnmanagedConfig:
  enabled: true
  exclude:
    secrets: true

This will remove any unmanaged or manual changes in Vault but it will leave secrets untouched. So if you enabled a new secret engine manually (and it’s not in Bank-Vaults configuration), Bank-Vaults will not remove it.

2.3.2 - Audit devices

You can configure Audit Devices in Vault (File, Syslog, Socket).

audit:
  - type: file
    description: "File based audit logging device"
    options:
      file_path: /tmp/vault.log

2.3.3 - Authentication

You can configure Auth Methods in Vault.

Currently the following auth methods are supported:

• AppRole
• AWS
• Azure
• GCP
• GitHub
• JWT
• Kubernetes
• LDAP

AppRole auth method

Allow machines/apps to authenticate with Vault-defined roles. For details, see the official Vault documentation.

auth:
  - type: approle
    roles:
    - name: default
      policies: allow_secrets
      secret_id_ttl: 10m
      token_num_uses: 10
      token_ttl: 20m
      token_max_ttl: 30m
      secret_id_num_uses: 40

AWS auth method

Creating roles in Vault which can be used for AWS IAM based authentication.

auth:
  - type: aws
    # Make the auth provider visible in the web ui
    # See https://developer.hashicorp.com/vault/api-docs/system/auth#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      access_key: VKIAJBRHKH6EVTTNXDHA
      secret_key: vCtSM8ZUEQ3mOFVlYPBQkf2sO6F/W7a5TVzrl3Oj
      iam_server_id_header_value: vault-dev.example.com # consider setting this to the Vault server's DNS name
    crossaccountrole:
    # Add cross account number and role to assume in the cross account
    # https://developer.hashicorp.com/vault/api-docs/auth/aws#create-sts-role
    - sts_account: 12345671234
      sts_role: arn:aws:iam::12345671234:role/crossaccountrole
    roles:
    # Add roles for AWS instances or principals
    # See https://developer.hashicorp.com/vault/api-docs/auth/aws#create-role
    - name: dev-role-iam
      bound_iam_principal_arn: arn:aws:iam::123456789012:role/dev-vault
      policies: allow_secrets
      period: 1h
    - name: cross-account-role
      bound_iam_principal_arn: arn:aws:iam::12345671234:role/crossaccountrole
      policies: allow_secrets
      period: 1h

Azure auth method

The Azure auth method allows authentication against Vault using Azure Active Directory credentials for more information.

auth:
  - type: azure
    config:
      tenant_id: 00000000-0000-0000-0000-000000000000
      resource: https://vault-dev.example.com
      client_id: 00000000-0000-0000-0000-000000000000
      client_secret: 00000000-0000-0000-0000-000000000000
    roles:
    # Add roles for azure identities
    # See https://developer.hashicorp.com/vault/api-docs/auth/azure#create-role
      - name: dev-mi
        policies: allow_secrets
        bound_subscription_ids:
          - "00000000-0000-0000-0000-000000000000"
        bound_service_principal_ids:
          - "00000000-0000-0000-0000-000000000000"

GCP auth method

Create roles in Vault which can be used for GCP IAM based authentication.

auth:
  - type: gcp
    # Make the auth provider visible in the web ui
    # See https://developer.hashicorp.com/vault/api-docs/system/auth#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      # Credentials context is service account's key. Can download when you create a key for service account.
      # No need to manually create it. Just paste the json context as multiline yaml.
      credentials: -|
        {
          "type": "service_account",
          "project_id": "PROJECT_ID",
          "private_key_id": "KEY_ID",
          "private_key": "-----BEGIN PRIVATE KEY-----.....-----END PRIVATE KEY-----\n",
          "client_email": "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com",
          "client_id": "CLIENT_ID",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token",
          "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
          "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT%40PROJECT_ID.iam.gserviceaccount.com"
        }        
    roles:
    # Add roles for gcp service account
    # See https://developer.hashicorp.com/vault/api-docs/auth/gcp#create-role
    - name: user-role
      type: iam
      project_id: PROJECT_ID
      policies: "readonly_secrets"
      bound_service_accounts: "USER_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
    - name: admin-role
      type: iam
      project_id: PROJECT_ID
      policies: "allow_secrets"
      bound_service_accounts: "ADMIN_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"

GitHub auth method

Create team mappings in Vault which can be used later on for the GitHub authentication.

auth:
  - type: github
    # Make the auth provider visible in the web ui
    # See https://developer.hashicorp.com/vault/api-docs/system/auth#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      organization: banzaicloud
    map:
      # Map the banzaicloud GitHub team on to the dev policy in Vault
      teams:
        dev: dev
      # Map my username (bonifaido) to the root policy in Vault
      users:
        bonifaido: allow_secrets

JWT auth method

Create roles in Vault which can be used for JWT-based authentication.

auth:
  - type: jwt
    path: jwt
    config:
      oidc_discovery_url: https://myco.auth0.com/
    roles:
    - name: role1
      bound_audiences:
        - https://vault.plugin.auth.jwt.test
      user_claim: https://vault/user
      groups_claim: https://vault/groups
      policies: allow_secrets
      ttl: 1h

Kubernetes auth method

Use the Kubernetes auth method to authenticate with Vault using a Kubernetes Service Account Token.

auth:
  - type: kubernetes
    # If you want to configure with specific kubernetes service account instead of default service account
    # https://developer.hashicorp.com/vault/docs/auth/kubernetes
    # config:
    #   token_reviewer_jwt: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9....
    #   kubernetes_ca_cert: |
    #     -----BEGIN CERTIFICATE-----
    #     ...
    #     -----END CERTIFICATE-----
    #   kubernetes_host: https://192.168.64.42:8443
    # Allows creating roles in Vault which can be used later on for the Kubernetes based
    # authentication.
    #  See https://developer.hashicorp.com/vault/docs/auth/kubernetes#creating-a-role for
    # more information.
    roles:
      # Allow every pod in the default namespace to use the secret kv store
      - name: default
        bound_service_account_names: default
        bound_service_account_namespaces: default
        policies: allow_secrets
        ttl: 1h

LDAP auth method

Create group mappings in Vault which can be used for LDAP based authentication.

• To start an LDAP test server, run: docker run -it –rm -p 389:389 -e LDAP_TLS=false –name ldap osixia/openldap
• To start an LDAP admin server, run: docker run -it –rm -p 6443:443 –link ldap:ldap -e PHPLDAPADMIN_LDAP_HOSTS=ldap -e PHPLDAPADMIN_LDAP_CLIENT_TLS=false osixia/phpldapadmin

auth:
  - type: ldap
    description: LDAP directory auth.
    # add mount options
    # See https://developer.hashicorp.com/vault/api-docs/system/auth#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      url: ldap://localhost
      binddn: "cn=admin,dc=example,dc=org"
      bindpass: "admin"
      userattr: uid
      userdn: "ou=users,dc=example,dc=org"
      groupdn: "ou=groups,dc=example,dc=org"
    groups:
      # Map the banzaicloud dev team on GitHub to the dev policy in Vault
      developers:
        policies: allow_secrets
    # Map myself to the allow_secrets policy in Vault
    users:
      bonifaido:
        groups: developers
        policies: allow_secrets

2.3.4 - Plugins

To register a new plugin in Vault’s plugin catalog, set the plugin_directory option in the Vault server configuration to the directory where the plugin binary is located. Also, for some plugins readOnlyRootFilesystem Pod Security Policy should be disabled to allow RPC communication between plugin and Vault server via Unix socket. For details, see the Hashicorp Go plugin documentation.

plugins:
  - plugin_name: ethereum-plugin
    command: ethereum-vault-plugin --ca-cert=/vault/tls/client/ca.crt --client-cert=/vault/tls/server/server.crt --client-key=/vault/tls/server/server.key
    sha256: 62fb461a8743f2a0af31d998074b58bb1a589ec1d28da3a2a5e8e5820d2c6e0a
    type: secret

2.3.5 - Policies

You can create policies in Vault, and later use these policies in roles for the Kubernetes-based authentication. For details, see Policies in the official Vault documentation.

policies:
  - name: allow_secrets
    rules: path "secret/*" {
             capabilities = ["create", "read", "update", "delete", "list"]
           }
  - name: readonly_secrets
    rules: path "secret/*" {
             capabilities = ["read", "list"]
           }

2.3.6 - Secrets engines

You can configure Secrets Engines in Vault. The Key-Value, Database, and SSH values are tested, but the configuration is free form, so probably others work as well.

AWS

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies.

secrets:
  - type: aws
    path: aws
    description: AWS Secrets Engine
    configuration:
        config:
          - name: root
            access_key: "${env `AWS_ACCESS_KEY_ID`}"
            secret_key: "${env `AWS_SECRET_ACCESS_KEY`}"
            region: us-east-1
        roles:
          - credential_type: iam_user
            policy_arns: arn-of-policy
            name: my-aws-role

Consul

The Consul secrets engine generates Consul ACL tokens dynamically based on policies created in Consul.

secrets:
  - path: consul
    type: consul
    description: Consul secrets
    configuration:
      config:
        - name: "access"
          address: "consul-server:8500"
          token: "${env `CONSUL_GLOBAL_MANAGEMENT_TOKEN`}" # Example how to read environment variables
      roles:
        - name: "<application_name>-read-only-role"
          consul_policies: "<application_name>-read-only-policy"
        - name: "<application_name>-read-write-role"
          consul_policies: "<application_name>-read-write-policy"

Database

This plugin stores database credentials dynamically based on configured roles for the MySQL/MariaDB database.

secrets:
  - type: database
    description: MySQL Database secret engine.
    configuration:
      config:
        - name: my-mysql
          plugin_name: "mysql-database-plugin"
          connection_url: "{{username}}:{{password}}@tcp(127.0.0.1:3306)/"
          allowed_roles: [pipeline]
          username: "${env `ROOT_USERNAME`}" # Example how to read environment variables
          password: "${env `ROOT_PASSWORD`}"
      roles:
        - name: pipeline
          db_name: my-mysql
          creation_statements: "GRANT ALL ON *.* TO '{{name}}'@'%' IDENTIFIED BY '{{password}}';"
          default_ttl: "10m"
          max_ttl: "24h"

Identity Groups

Allows you to configure identity groups.

Note:

Only external groups are supported at the moment through the use of group-aliases. For supported authentication backends (for example JWT, which automatically matches those aliases to groups returned by the backend) the configuration files for the groups and group-aliases need to be parsed after the authentication backend has been mounted. Ideally they should be in the same file to avoid of errors.

groups:
  - name: admin
    policies:
      - admin
    metadata:
      admin: "true"
      priviliged: "true"
    type: external

group-aliases:
  - name: admin
    mountpath: jwt
    group: admin

Key-Values

This plugin stores arbitrary secrets within the configured physical storage for Vault.

secrets:
  - path: secret
    type: kv
    description: General secrets.
    options:
      version: 2
    configuration:
      config:
        - max_versions: 100

Non-default plugin path

Mounts a non-default plugin’s path.

  - path: ethereum-gateway
    type: plugin
    plugin_name: ethereum-plugin
    description: Immutability's Ethereum Wallet

PKI

The PKI secrets engine generates X.509 certificates.

secrets:
  - type: pki
    description: Vault PKI Backend
    config:
      default_lease_ttl: 168h
      max_lease_ttl: 720h
    configuration:
      config:
      - name: urls
        issuing_certificates: https://vault.default:8200/v1/pki/ca
        crl_distribution_points: https://vault.default:8200/v1/pki/crl
      root/generate:
      - name: internal
        common_name: vault.default
      roles:
      - name: default
        allowed_domains: localhost,pod,svc,default
        allow_subdomains: true
        generate_lease: true
        ttl: 30m

RabbitMQ

The RabbitMQ secrets engine generates user credentials dynamically based on configured permissions and virtual hosts.

To start a RabbitMQ test server, run: docker run -it –rm -p 15672:15672 rabbitmq:3.7-management-alpine

secrets:
  - type: rabbitmq
    description: local-rabbit
    configuration:
      config:
        - name: connection
          connection_uri: "http://localhost:15672"
          username: guest
          password: guest
      roles:
        - name: prod_role
          vhosts: '{"/web":{"write": "production_.*", "read": "production_.*"}}'

SSH

Create a named Vault role for signing SSH client keys.

secrets:
  - type: ssh
    path: ssh-client-signer
    description: SSH Client Key Signing.
    configuration:
      config:
        - name: ca
          generate_signing_key: "true"
      roles:
        - name: my-role
          allow_user_certificates: "true"
          allowed_users: "*"
          key_type: "ca"
          default_user: "ubuntu"
          ttl: "24h"
          default_extensions:
            permit-pty: ""
            permit-port-forwarding: ""
            permit-agent-forwarding: ""

2.3.7 - Startup secrets

Allows writing some secrets to Vault (useful for development purposes). For details, see the Key-Value secrets engine.

startupSecrets:
  - type: kv
    path: secret/data/accounts/aws
    data:
      data:
        AWS_ACCESS_KEY_ID: secretId
        AWS_SECRET_ACCESS_KEY: s3cr3t

3 - Vault operator

The Vault operator builds on Bank-Vaults features such as:

• external, API based configuration (secret engines, auth methods, policies) to automatically re/configure a Vault cluster
• automatic unsealing (AWS, GCE, Azure, Alibaba, Kubernetes Secrets (for dev purposes), Oracle)
• TLS support

The operator flow is the following:

The source code can be found in the vault-operator repository.

The operator requires the following cloud permissions.

Deploy a local Vault operator

This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets. If you want to customize the Helm chart, see the list of vault-operator Helm chart values.

1. Install the Bank-Vaults operator:

   helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
   

   Expected output:

   Release "vault-operator" does not exist. Installing it now.
   Pulled: ghcr.io/bank-vaults/helm-charts/vault-operator:1.20.0
   Digest: sha256:46045be1c3b215f0c734908bb1d4022dc91eae48d2285382bb71d63f72c737d1
   NAME: vault-operator
   LAST DEPLOYED: Thu Jul 27 11:22:55 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

2. Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
       

   Expected output:

   serviceaccount/vault created
       role.rbac.authorization.k8s.io/vault created
       role.rbac.authorization.k8s.io/leader-election-role created
       rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
       rolebinding.rbac.authorization.k8s.io/vault created
       clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
       

   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml
       

   Expected output:

   vault.vault.banzaicloud.com/vault created
       

   Note: If needed, you can install the latest CustomResource from the main branch, but that’s usually under development and might not be stable.

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/crd | kubectl apply -f -
       

3. Wait a few seconds, then check the operator and the vault pods:

   kubectl get pods
   

   Expected output:

   NAME                                                        READY     STATUS    RESTARTS   AGE
   vault-0                                                     3/3       Running   0          10s
   vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
   vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s
   

4. Configure your Vault client to access the Vault instance running in the vault-0 pod.

   1. Port-forward into the pod:

      kubectl port-forward vault-0 8200 &
      

   2. Set the address of the Vault instance.

      export VAULT_ADDR=https://127.0.0.1:8200
      

   3. Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):

      kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
      export VAULT_CACERT=$PWD/vault-ca.crt
      

      Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true

   4. If you already have the Vault CLI installed, check that you can access the Vault:

      vault status
      

      Expected output:

      Key             Value
      ---             -----
      Seal Type       shamir
      Initialized     true
      Sealed          false
      Total Shares    5
      Threshold       3
      Version         1.5.4
      Cluster Name    vault-cluster-27ecd0e6
      Cluster ID      ed5492f3-7ef3-c600-aef3-bd77897fd1e7
      HA Enabled      false
      

   5. To authenticate to Vault, you can access its root token by running:

      export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
      

      Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.

   6. Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).

For other configuration examples of the Vault CustomResource, see the YAML files in the deploy/examples and test/deploy directories of the vault-operator repository. After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:

kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl delete -f -
kubectl delete -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml

HA setup with Raft

In a production environment you want to run Vault as a cluster. The following CR creates a 3-node Vault instance that uses the Raft storage backend:

1. Install the Bank-Vaults operator:

   helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
   

2. Create a Vault instance using the cr-raft.yaml custom resource. This will create a Kubernetes CustomResource called vault that uses the Raft backend:

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
       

   Expected output:

   serviceaccount/vault created
       role.rbac.authorization.k8s.io/vault created
       role.rbac.authorization.k8s.io/leader-election-role created
       rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
       rolebinding.rbac.authorization.k8s.io/vault created
       clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
       

   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml
       

   Expected output:

   vault.vault.banzaicloud.com/vault created
       

   Note: If needed, you can install the latest CustomResource from the main branch, but that’s usually under development and might not be stable.

   kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/crd | kubectl apply -f -
       

Pod anti-affinity

If you want to setup pod anti-affinity, you can set podAntiAffinity vault with a topologyKey value. For example, you can use failure-domain.beta.kubernetes.io/zone to force K8S deploy vault on multi AZ.

Delete a resource created by the operator

If you manually delete a resource that the Bank-Vaults operator has created (for example, the Ingress resource), the operator automatically recreates it every 30 seconds. If it doesn’t, then something went wrong, or the operator is not running. In this case, check the logs of the operator.

3.1 - Running Vault with external end to end encryption

This document assumes you have a working Kubernetes cluster which has a:

• Working install of Vault.
• That you have a working knowledge of Kubernetes.
• A working install of helm
• A working knowledge of Kubernetes ingress
• A valid external (www.example.com) SSL certificate, verified by your provider as a Kubernetes secret.

Background

The bank-vaults operator takes care of creating and maintaining internal cluster communications but if you wish to use your vault install outside of your Kubernetes cluster what is the best way to maintain a secure state. Creating a standard Ingress object will reverse proxy these requests to your vault instance but this is a hand off between the external SSL connection and the internal one. This might not be acceptable under some circumstances, for example, if you have to adhere to strict security standards.

Workflow

Here we will create a separate TCP listener for vault using a custom SSL certificate on an external domain of your choosing. We will then install a unique ingress-nginx controller allowing SSL pass through. SSL Pass through comes with a performance hit, so you would not use this on a production website or ingress-controller that has a lot of traffic.

Install

ingress-nginx

values.yaml

controller:
  electionID: vault-ingress-controller-leader
  ingressClass: nginx-vault
  extraArgs:
    enable-ssl-passthrough:
  publishService:
    enabled: true
  scope:
    enabled: true
  replicaCount: 2
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: release
            operator: In
            values: ["vault-ingress"]
        topologyKey: kubernetes.io/hostname

Install nginx-ingress via helm

helm install nginx-stable/nginx-ingress --name my-release -f values.yaml

Configuration

SSL Secret example:

apiVersion: v1
data:
  tls.crt: LS0tLS1......=
  tls.key: LS0tLS.......==
kind: Secret
metadata:
  labels:
    ssl: "true"
    tls: "true"
  name: wildcard.example.com
type: Opaque

CR Vault Config:

---
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
  namespace: secrets
spec:
  size: 2
  image: hashicorp/vault:1.14.1
  bankVaultsImage: ghcr.io/bank-vaults/bank-vaults:latest

  # A YAML representation of a final vault config file.
  # See https://developer.hashicorp.com/vault/docs/configuration for more information.
  config:
    listener:
      - tcp:
          address: "0.0.0.0:8200"
          tls_cert_file: /vault/tls/server.crt
          tls_key_file: /vault/tls/server.key
      - tcp:
          address: "0.0.0.0:8300"
          tls_cert_file: /etc/ingress-tls/tls.crt
          tls_key_file: /etc/ingress-tls/tls.key
    api_addr: https://vault:8200
    cluster_addr: https://vault:8201
    ui: true

CR Service:

  # Specify the Service's type where the Vault Service is exposed
  serviceType: ClusterIP
  servicePorts:
    api-port: 8200
    cluster-port: 8201
    ext-api-port: 8300
    ext-clu-port: 8301

Mount the secret into your vault pod

  volumes:
    - name: wildcard-ssl
      secret:
        defaultMode: 420
        secretName: wildcard.example.com

  volumeMounts:
    - name: wildcard-ssl
      mountPath: /etc/ingress-tls

CR Ingress:

  # Request an Ingress controller with the default configuration
  ingress:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      kubernetes.io/ingress.class: "nginx-vault"
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/whitelist-source-range: "127.0.0.1"

    spec:
      rules:
        - host: vault.example.com
          http:
            paths:
              - path: /
                backend:
                  serviceName: vault
                  servicePort: 8300

3.2 - Using templates for injecting dynamic configuration

Background

When configuring a Vault object via the externalConfig property, sometimes it’s convenient (or necessary) to inject settings that are only known at runtime, for example:

• secrets that you don’t want to store in source control
• dynamic resources managed elsewhere
• computations based on multiple values (string or arithmetic operations).

For these cases, the operator supports parameterized templating. The vault-configurer component evaluates the templates and injects the rendered configuration into Vault.

This templating is based on Go templates, extended by Sprig, with some custom functions available specifically for bank-vaults (for example, to decrypt strings using the AWS Key Management Service or the Cloud Key Management Service of the Google Cloud Platform).

Using templates

To avoid confusion and potential parsing errors (and interference with other templating tools like Helm), the templates don’t use the default delimiters that Go templates use ({{ and }}). Instead, use the following characters:

• ${ for the left delimiter
• } for the right one.
• To quote parameters being passed to functions, surround them with backticks (`) instead.

For example, to call the env function, you can use this in your manifest:

password: "${ env `MY_ENVIRONMENT_VARIABLE` }"

In this case, vault-configurer evaluates the value of MY_ENVIRONMENT_VARIABLE at runtime (assuming it was properly injected), and sets the result as the value of the password field.

Note that you can also use Sprig functions and custom Kubernetes-related functions in your templates.

For a detailed example, see the Using templates for injecting dynamic configuration in Vault blog post.

Sprig functions

In addition to the default functions in Go templates, you can also use Sprig functions in your configuration.

Custom functions

To provide functionality that’s more Kubernetes-friendly and cloud-native, bank-vaults provides a few additional functions not available in Sprig or Go. The functions and their parameters (in the order they should go in the function) are documented below.

awskms

Takes a base64-encoded, KMS-encrypted string and returns the decrypted string. Additionally, the function takes an optional second parameter for any encryption context that might be required for decrypting. If any encryption context is required, the function will take any number of additional parameters, each of which should be a key-value pair (separated by a = character), corresponding to the full context.

Note: This function assumes that the vault-configurer pod has the appropriate AWS IAM credentials and permissions to decrypt the given string. You can inject the AWS IAM credentials by using Kubernetes secrets as environment variables, an EC2 instance role, kube2iam, or EKS IAM roles, and so on.

For example:

password: '${ awskms (env `ENCRYPTED_DB_CREDS`) }'

You can also nest functions in the template, for example:

password: '${ awskms (blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1`) }'

gcpkms

Takes a base64-encoded string, encrypted with a Google Cloud Platform (GCP) symmetric key and returns the decrypted string.

Note: This function assumes that the vault-configurer pod has the appropriate GCP IAM credentials and permissions to decrypt the given string. You can inject the GCP IAM credentials by using Kubernetes secrets as environment variables, or they can be acquired via a service account authentication, and so on.

blob

Reads the content of a blob from disk (file) or from cloud blob storage services (object storage) at the given URL and returns it. This assumes that the path exists, and is readable by vault-configurer.

Valid values for the URL parameters are listed below, for more fine grained options check the documentation of the underlying library:

• file:///path/to/dir/file
• s3://my-bucket/object?region=us-west-1
• gs://my-bucket/object
• azblob://my-container/blob

Note: This function assumes that the vault-configurer pod has the appropriate rights to access the given cloud service. For details, see the awskms and gcpkms functions.

For example:

password: '${ blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1` }'

You can also nest functions in the template, for example:

password: '${ awskms (blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1`) }'

file

Reads the content of a file from disk at the given path and returns it. This assumes that the file exists, it’s mounted, and readable by vault-configurer.

accessor

Looks up the accessor id of the given auth path and returns it. This function is only useful in policies that use templated policies, to generalize the <mount accessor> field.

For example:

policies:
  - name: allow_secrets
    rules: path "secret/data/{{identity.entity.aliases.${ accessor `kubernetes/` }.metadata.service_account_namespace}}/*" {
             capabilities = ["read"]
           }

3.3 - Environment variables

You can add environment variables to the different containers of the Bank-Vaults pod using the following configuration options:

• envsConfig: Adds environment variables to all Bank-Vaults pods.
• sidecarEnvsConfig: Adds environment variables to Vault sidecar containers.
• vaultEnvsConfig: Adds environment variables to the Vault container.

For example:

envsConfig:
  - name: ROOT_USERNAME
    valueFrom:
      secretKeyRef:
        name: mysql-login
        key: user
  - name: ROOT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mysql-login
        key: password

See the database secret engine section for usage. Further information:

• List of Kubernetes environment variables
• Using secrets as environment variables

3.4 - Upgrade strategies

Upgrade Vault

To upgrade the Vault, complete the following steps.

1. Check the release notes of Vault for any special upgrade instructions. Usually there are no instructions, but it’s better to be safe than sorry.
2. Adjust the spec.image field in the Vault custom resource. If you are using the Vault Helm chart, adjust the image.tag field in the values.yaml.
3. The Vault Helm chart updates the StatefulSet. It does not take the HA leader into account in HA scenarios, but this has never caused any issues so far.

Upgrade Vault operator

v1.20.0 upgrade guide

The release of the Vault Operator v1.20.0 marks the beginning of a new chapter in the development of the Bank-Vaults ecosystem, as this is the first release across the project after it has been dissected and migrated from the original banzaicloud/bank-vaults repository under its own bank-vaults organization. We paid attention to not introduce breaking changes during the process, however, the following changes are now in effect:

• All Helm charts will now be distributed via OCI registry on GitHub.
• All releases will be tagged with v prefix, starting from v1.20.0.

Upgrade to the new Vault Operator by changing the helm repository to the new OCI registry, and specifying the new version numbers:

helm upgrade vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator \
	--set image.tag=v1.20.0 \
	--set bankVaults.image.tag=v1.20.0 \
	--wait

Make sure to also change the bank-vaults image in the Vault CR’s spec.bankVaultsImage field to ghcr.io/bank-vaults/bank-vaults:1.20.x.

3.5 - Operator Configuration for Functioning Webhook Secrets Mutation

You can find several examples of the vault operator CR manifest in the vault-operator repository. The following examples use only this vanilla CR to demonstrate some main points about how to properly configure the operator for secrets mutations to function.

This document does not attempt to explain every possible scenario with respect to the CRs in the aforementioned directory, but instead attempts to explain at a high level the important aspects of the CR, so that you can determine how best to configure your operator.

Main points

Some important aspects of the operator and its configuration with respect to secrets mutation are:

• The vault operator instantiates:
  • the vault configurer pod(s),
  • the vault pod(s),
  • the vault-secrets-webhook pod(s).
• The vault configurer:
  • unseals the vault,
  • configures vault with policies, roles, and so on.
• vault-secrets-webhook does nothing more than:
  • monitors cluster for resources with specific annotations for secrets injection, and
  • integrates with vault API to answer secrets requests from those resources for requested secrets.
  • For pods using environment secrets, it injects a binary vault-env into pods and updates ENTRYPOINT to run vault-env CMD instead of CMD. vault-env intercepts requests for env secrets requests during runtime of pod and upon such requests makes vault API call for requested secret injecting secret into environment variable so CMD works with proper secrets.
• Vault
  • the secrets workhorse
  • surfaces a RESTful API for secrets management

CR configuration properties

This section goes over some important properties of the CR and their purpose.

Vault’s service account

This is the serviceaccount where Vault will be running. The Configurer runs in the same namespace and should have the same service account. The operator assigns this serviceaccount to Vault.

  # Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
  serviceAccount: vault

caNamespaces

In order for vault communication to be encrypted, valid TLS certificates need to be used. The following property automatically creates TLS certificate secrets for the namespaces specified here. Notice that this is a list, so you can specify multiple namespaces per line, or use the splat or wildcard asterisk to specify all namespaces:

  # Support for distributing the generated CA certificate Secret to other namespaces.
  # Define a list of namespaces or use ["*"] for all namespaces.
  caNamespaces:
    - "*"

Vault Config

The following is simply a YAML representation (as the comment says) for the Vault configuration you want to run. This is the configuration that vault configurer uses to configure your running Vault:

  # A YAML representation of a final vault config file.
  config:
    api_addr: https://vault:8200
    cluster_addr: https://${.Env.POD_NAME}:8201
    listener:
      tcp:
        address: 0.0.0.0:8200
        # Commenting the following line and deleting tls_cert_file and tls_key_file disables TLS
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    storage:
      file:
        path: "${ .Env.VAULT_STORAGE_FILE }"
    ui: true
  credentialsConfig:
    env: ""
    path: ""
    secretName: ""
  etcdSize: 0
  etcdVersion: ""
  externalConfig:
    policies:
      - name: allow_secrets
        rules: path "secret/*" {
          capabilities = ["create", "read", "update", "delete", "list"]
          }
    auth:
      - type: kubernetes
        roles:
          # Allow every pod in the default namespace to use the secret kv store
          - name: default
            bound_service_account_names:
              - external-secrets
              - vault
              - dex
            bound_service_account_namespaces:
              - external-secrets
              - vault
              - dex
              - auth-system
              - loki
              - grafana
            policies:
              - allow_secrets
            ttl: 1h

          # Allow mutation of secrets using secrets-mutation annotation to use the secret kv store
          - name: secretsmutation
            bound_service_account_names:
              - vault-secrets-webhook
            bound_service_account_namespaces:
              - vault-secrets-webhook
            policies:
              - allow_secrets
            ttl: 1h

externalConfig

The externalConfig portion of this CR example correlates to Kubernetes configuration as specified by .auth[].type.

This YAML representation of configuration is flexible enough to work with any auth methods available to Vault as documented in the Vault documentation. For now, we’ll stick with this kubernetes configuration.

externalConfig.purgeUnmanagedConfig

Delete any configuration that in Vault but not in externalConfig. For more details please check Purge unmanaged configuration

externalConfig.policies

Correlates 1:1 to the creation of the specified policy in conjunction with Vault policies.

externalConfig.auth[].type

- type: kubernetes - specifies to configure Vault to use Kubernetes authentication

Other types are yet to be documented with respect to the operator configuration.

externalConfig.auth[].roles[]

Correlates to Creating Kubernetes roles. Some important nuances here are:

• Vault does not respect inline secrets serviceaccount annotations, so the namespace of any serviceaccount annotations for secrets are irrelevant to getting inline secrets mutations functioning.
• Instead, the serviceaccount of the vault-secrets-webhook pod(s) should be used to configure the bound_service_account_names and bound_service_account_namespaces for inline secrets to mutate.
• Pod serviceaccounts, however, are respected so bound_service_account_namespaces and bound_service_account_names for environment mutations must identify such of the running pods.

Note: There are two roles specified in the YAML example above: one for pods, and one for inline secrets mutations. While this was not strictly required, it makes for cleaner implementation.

3.6 - TLS

Bank-Vaults tries to automate as much as possible for handling TLS certificates.

• The vault-operator automates the creation and renewal of TLS certificates for Vault.
• The vault Helm Chart automates only the creation of TLS certificates for Vault via Sprig.

Both the operator and the chart generate a Kubernetes Secret holding the TLS certificates, this is named ${VAULT_CR_NAME}-tls. For most examples in the vault-operator repository, the name of the secret is vault-tls.

The Secret data keys are:

• ca.crt
• ca.key
• server.crt
• server.key

Note: The operator doesn’t overwrite this Secret if it already exists, so you can provide this certificate in any other way, for example using cert-manager or by simply placing it there manually.

Operator custom TLS settings

The following attributes influence the TLS settings of the operator. The ca.crt key is mandatory in existingTlsSecretName, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

CANamespaces

The list of namespaces where the generated CA certificate for Vault should be distributed. Use ["*"] for all namespaces.

Default value: []

ExistingTLSSecretName

The name of the secret that contains a TLS server certificate, key, and the corresponding CA certificate. The secret must be in the kubernetes.io/tls type secret keys + ca.crt key format. If the attribute is set, the operator uses the certificate already set in the secret, otherwise it generates a new one.

The ca.crt key is mandatory, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

Default value: ""

TLSAdditionalHosts

A list hostnames or IP addresses to add to the SAN on the automatically generated TLS certificate.

Default value: []

TLSExpiryThreshold

The expiration threshold of the Vault TLS certificate in Go Duration format.

Default value: 168h

Helm chart custom TLS settings

Starting with version 1.20, the Vault Helm chart allows you to set custom TLS settings. The following attributes influence the TLS settings of the Helm chart. The ca.crt key is mandatory in secretName, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

SecretName

The name of the secret that contains a TLS server certificate, key, and the corresponding CA certificate. The secret must be in the kubernetes.io/tls type secret keys + ca.crt key format. If the attribute is set, the operator uses the certificate already set in the secret, otherwise it generates a new one.

The ca.crt key is mandatory, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

Default value: ""

CANamespaces

The list of namespaces where the generated CA certificate for Vault should be distributed.

Default value: []

Using the generated custom TLS certificate with vault-operator

To use an existing secret which contains the TLS certificate, define existingTlsSecretName in the Vault custom resource.

Generate custom certificates with CFSSL

If you don’t want to use the certificates generated by Helm or the Bank-Vaults operator, the easiest way to create a custom certificate for Bank-Vaults is using CFSSL.

The TLS directory in the documentation holds a set of custom CFSSL configurations which are prepared for the Helm release name vault in the default namespace. Of course, you can put any other certificates into the Secret below, this is just an example.

1. Install CFSSL.

2. Create a CA:

   cfssl genkey -initca csr.json | cfssljson -bare ca
   

3. Create a server certificate:

   cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config.json -profile=server server.json | cfssljson -bare server
   

4. Put these certificates (and the server key) into a Kubernetes Secret:

   kubectl create secret generic vault-tls --from-file=ca.crt=ca.pem --from-file=server.crt=server.pem --from-file=server.key=server-key.pem
   

5. Install the Vault instance:

   • With the chart which uses this certificate:

   helm upgrade --install vault ../charts/vault --set tls.secretName=vault-tls
   

   • With the operator, create a Vault custom resource, and apply it:

   kubectl apply -f vault-cr.yaml
   

Generate custom certificates with cert-manager

You can use the following cert-manager custom resource to generate a certificate for Bank-Vaults.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-cert
spec:
  commonName: vault
  usages:
    - server auth
  dnsNames:
    - vault
    - vault.default
    - vault.default.svc
    - vault.default.svc.cluster.local
  ipAddresses:
    - 127.0.0.1
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF

3.7 - Backing up Vault

You can configure the vault-operator to create backups of the Vault cluster with Velero.

Prerequisites

• The Velero CLI must be installed on your computer.
• To create Persistent Volume (PV) snapshots, you need access to an object storage. The following example uses an Amazon S3 bucket called bank-vaults-velero in the Stockholm region.

Install Velero

To configure the vault-operator to create backups of the Vault cluster, complete the following steps.

1. Install Velero on the target cluster with Helm.

   1. Add the Velero Helm repository:

      helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
      

   2. Create a namespace for Velero:

      kubectl create namespace velero
      

   3. Install Velero with Restic so you can create PV snapshots as well:

      BUCKET=bank-vaults-velero
      REGION=eu-north-1
      KMS_KEY_ID=alias/bank-vaults-velero
      SECRET_FILE=~/.aws/credentials
      
      helm upgrade --install velero --namespace velero \
                --set "configuration.backupStorageLocation[0].name"=aws \
                --set "configuration.backupStorageLocation[0].provider"=aws \
                --set "configuration.backupStorageLocation[0].bucket"=YOUR_BUCKET_NAME \
                --set "configuration.backupStorageLocation[0].config.region"=${REGION} \
                --set "configuration.backupStorageLocation[0].config.kmsKeyId"=${KMS_KEY_ID} \
                --set "configuration.volumeSnapshotLocation[0].name"=aws \
                --set "configuration.volumeSnapshotLocation[0].provider"=aws \
                --set "configuration.volumeSnapshotLocation[0].config.region"=${REGION} \
                --set "initContainers[0].name"=velero-plugin-for-aws \
                --set "initContainers[0].image"=velero/velero-plugin-for-aws:v1.7.0 \
                --set "initContainers[0].volumeMounts[0].mountPath"=/target \
                --set "initContainers[0].volumeMounts[0].name"=plugins \
                vmware-tanzu/velero
      

2. Install the vault-operator to the cluster:

   helm upgrade --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
   
   kubectl apply -f operator/deploy/rbac.yaml
   kubectl apply -f operator/deploy/cr-raft.yaml
   

   Note: The Vault CR in cr-raft.yaml has a special flag called veleroEnabled. This is useful for file-based Vault storage backends (file, raft), see the Velero documentation:

     # Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:
     # https://velero.io/docs/v1.2.0/hooks/
     veleroEnabled: true
   

3. Create a backup with the Velero CLI or with the predefined Velero Backup CR:

   velero backup create --selector vault_cr=vault vault-1
   
   # OR
   
   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/bank-vaults.dev/main/content/docs/operator/backup/backup.yaml
   

   Note: For a daily scheduled backup, see schedule.yaml.

4. Check that the Velero backup got created successfully:

   velero backup describe --details vault-1
   

   Expected output:

   Name:         vault-1
   Namespace:    velero
   Labels:       velero.io/backup=vault-1
                 velero.io/pv=pvc-6eb4d9c1-25cd-4a28-8868-90fa9d51503a
                 velero.io/storage-location=default
   Annotations:  <none>
   
   Phase:  Completed
   
   Namespaces:
     Included:  *
     Excluded:  <none>
   
   Resources:
     Included:        *
     Excluded:        <none>
     Cluster-scoped:  auto
   
   Label selector:  vault_cr=vault
   
   Storage Location:  default
   
   Snapshot PVs:  auto
   
   TTL:  720h0m0s
   
   Hooks:  <none>
   
   Backup Format Version:  1
   
   Started:    2020-01-29 14:17:41 +0100 CET
   Completed:  2020-01-29 14:17:45 +0100 CET
   
   Expiration:  2020-02-28 14:17:41 +0100 CET
   

Test the backup

1. To emulate a catastrophe, remove Vault entirely from the cluster:

   kubectl delete vault -l vault_cr=vault
   kubectl delete pvc -l vault_cr=vault
   

2. Now restore Vault from the backup.

   1. Scale down the vault-operator, so it won’t reconcile during the restore process:

      kubectl scale deployment vault-operator --replicas 0
      

   2. Restore all Vault-related resources from the backup:

      velero restore create --from-backup vault-1
      

   3. Check that the restore has finished properly:

      velero restore get
      NAME                    BACKUP   STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
      vault1-20200129142409   vault1   Completed   0          0        2020-01-29 14:24:09 +0100 CET   <none>
      

   4. Check that the Vault cluster got actually restored:

      kubectl get pods
      NAME                                READY   STATUS    RESTARTS   AGE
      vault-0                             4/4     Running   0          1m42s
      vault-1                             4/4     Running   0          1m42s
      vault-2                             4/4     Running   0          1m42s
      vault-configurer-5499ff64cb-g75vr   1/1     Running   0          1m42s
      

   5. Scale the operator back after the restore process:

      kubectl scale deployment vault-operator --replicas 1
      

3. Delete the backup if you don’t wish to keep it anymore:

   velero backup delete vault-1
   

3.8 - Running the Bank-Vaults secret webhook alongside Istio

Both the vault-operator and the vault-secrets-webhook can work on Istio-enabled clusters.

We support the following three scenarios:

• Scenario 1: Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled
• Scenario 2: The namespace where Vault is running has Istio sidecar injection enabled
• Scenario 3: Both namespaces have Istio sidecar injection enabled

Prerequisites

1. Install the Istio operator.

2. Make sure you have mTLS enabled in the Istio mesh through the operator with the following command:

   Enable mTLS if it is not set to STRICT:

   kubectl patch istio -n istio-system mesh --type=json -p='[{"op": "replace", "path": "/spec/meshPolicy/mtlsMode", "value":STRICT}]'
   

3. Check that mesh is configured with mTLS turned on which applies to all applications in the cluster in Istio-enabled namespaces. You can change this if you would like to use another policy.

   kubectl get meshpolicy default -o yaml
   

   Expected output:

   apiVersion: authentication.istio.io/v1alpha1
   kind: MeshPolicy
   metadata:
     name: default
     labels:
       app: security
   spec:
     peers:
     - mtls: {}
   

Now your cluster is properly running on Istio with mTLS enabled globally.

Install the Bank-Vaults components

1. You are recommended to create a separate namespace for Bank-Vaults called vault-system. You can enable Istio sidecar injection here as well, but Kubernetes won’t be able to call back the webhook properly since mTLS is enabled (and Kubernetes is outside of the Istio mesh). To overcome this, apply a PERMISSIVE Istio authentication policy to the vault-secrets-webhook Service itself, so Kubernetes can call it back without Istio mutual TLS authentication.

   kubectl create namespace vault-system
   kubectl label namespace vault-system name=vault-system istio-injection=enabled
   

   kubectl apply -f - <<EOF
   apiVersion: authentication.istio.io/v1alpha1
   kind: Policy
   metadata:
     name: vault-secrets-webhook
     namespace: vault-system
     labels:
       app: security
   spec:
     targets:
     - name: vault-secrets-webhook
     peers:
     - mtls:
         mode: PERMISSIVE
   EOF
   

2. Now you can install the operator and the webhook to the prepared namespace:

   helm upgrade --install --wait vault-secrets-webhook oci://ghcr.io/bank-vaults/helm-charts/vault-secrets-webhook --namespace vault-system --create-namespace
   helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator --namespace vault-system
   

Soon the webhook and the operator become up and running. Check that the istio-proxy got injected into all Pods in vault-system.

Proceed to the description of your scenario:

• Scenario 1: Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled
• Scenario 2: The namespace where Vault is running has Istio sidecar injection enabled
• Scenario 3: Both namespaces have Istio sidecar injection enabled

3.8.1 - Scenario 1 - Vault runs outside, the application inside the mesh

In this scenario, Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled.

First, complete the Prerequisites, then install Vault outside the mesh, and finally install an application within the mesh.

Install Vault outside the mesh

1. Provision a Vault instance with the Bank-Vaults operator in a separate namespace:

   kubectl create namespace vault
   

2. Apply the RBAC and CR files to the cluster to create a Vault instance in the vault namespace with the operator:

   kubectl apply -f rbac.yaml -f cr-istio.yaml
   

   kubectl get pods -n vault
   

   Expected output:

   NAME                               READY   STATUS    RESTARTS   AGE
   vault-0                            3/3     Running   0          22h
   vault-configurer-6458cc4bf-6tpkz   1/1     Running   0          22h
   

   If you are writing your own Vault CR make sure that istioEnabled: true is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly.

3. The vault-secrets-webhook can’t inject Vault secrets into initContainers in an Istio-enabled namespace when the STRICT authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase when initContainers are running the Pod doesn’t have a sidecar yet. If you wish to inject into initContainers as well, you need to apply a PERMISSIVE authentication policy in the vault namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).

   kubectl apply -f - <<EOF
   apiVersion: authentication.istio.io/v1alpha1
   kind: Policy
   metadata:
     name: default
     namespace: vault
     labels:
       app: security
   spec:
     peers:
     - mtls:
         mode: PERMISSIVE
   EOF
   

Install the application inside a mesh

In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:

1. Create a namespace first for the application and enable Istio sidecar injection:

   kubectl create namespace app
   kubectl label namespace app istio-injection=enabled
   

2. Install the application manifest to the cluster:

   kubectl apply -f app.yaml
   

3. Check that the application is up and running. It should have two containers, the app itself and the istio-proxy:

   kubectl get pods -n app
   

   Expected output:

   NAME                  READY   STATUS    RESTARTS   AGE
   app-5df5686c4-sl6dz   2/2     Running   0          119s
   

   kubectl logs -f -n app deployment/app app
   

   Expected output:

   time="2020-02-18T14:26:01Z" level=info msg="Received new Vault token"
   time="2020-02-18T14:26:01Z" level=info msg="Initial Vault token arrived"
   s3cr3t
   going to sleep...
   

3.8.2 - Scenario 2 - Running Vault inside the mesh

To run Vault inside the mesh, complete the following steps.

Note: These instructions assume that you have Scenario 1 up and running, and modifying it to run Vault inside the mesh.

1. Turn off Istio in the app namespace by removing the istio-injection label:

   kubectl label namespace app istio-injection-
   kubectl label namespace vault istio-injection=enabled
   

2. Delete the Vault pods in the vault namespace, so they will get recreated with the istio-proxy sidecar:

   kubectl delete pods --all -n vault
   

3. Check that they both come back with an extra container (4/4 and 2/2 now):

   kubectl get pods -n vault
   

   Expected output:

   NAME                                READY   STATUS    RESTARTS   AGE
   vault-0                             4/4     Running   0          1m
   vault-configurer-6d9b98c856-l4flc   2/2     Running   0          1m
   

4. Delete the application pods in the app namespace, so they will get recreated without the istio-proxy sidecar:

   kubectl delete pods --all -n app
   

The app pod got recreated with only the app container (1/1) and Vault access still works:

kubectl get pods -n app

Expected output:

NAME                  READY   STATUS    RESTARTS   AGE
app-5df5686c4-4n6r7   1/1     Running   0          71s

kubectl logs -f -n app deployment/app

Expected output:

time="2020-02-18T14:41:20Z" level=info msg="Received new Vault token"
time="2020-02-18T14:41:20Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...

3.8.3 - Scenario 3 - Both Vault and the app are running inside the mesh

In this scenario, both Vault and the app are running inside the mesh.

1. Complete the Prerequisites.

2. Enable sidecar auto-injection for both namespaces:

   kubectl label namespace app   istio-injection=enabled
   kubectl label namespace vault istio-injection=enabled
   

3. Delete all pods so they are getting injected with the proxy:

   kubectl delete pods --all -n app
   kubectl delete pods --all -n vault
   

4. Check the logs in the app container. It should sill show success:

   kubectl logs -f -n app deployment/app
   

   Expected output:

   time="2020-02-18T15:04:03Z" level=info msg="Initial Vault token arrived"
   time="2020-02-18T15:04:03Z" level=info msg="Renewed Vault Token"
   s3cr3t
   going to sleep...
   

3.9 - HSM Support

Bank-Vaults offers a lot of alternatives for encrypting and storing the unseal-keys and the root-token for Vault. One of the encryption technics is the HSM - Hardware Security Module. HSM offers an industry-standard way to encrypt your data in on-premise environments.

You can use a Hardware Security Module (HSM) to generate and store the private keys used by Bank-Vaults. Some articles still point out the speed of HSM devices as their main selling point, but an average PC can do more cryptographic operations. Actually, the main benefit is from the security point of view. An HSM protects your private keys and handles cryptographic operations, which allows the encryption of protected information without exposing the private keys (they are not extractable). Bank-Vaults currently supports the PKCS11 software standard to communicate with an HSM. Fulfilling compliance requirements (for example, PCI DSS) is also a great benefit of HSMs, so from now on you can achieve that with Bank-Vaults.

Implementation in Bank-Vaults

To support HSM devices for encrypting unseal-keys and root-tokens, Bank-Vaults:

• implements an encryption/decryption Service named hsm in the bank-vaults CLI,
• the bank-vaults Docker image now includes the SoftHSM (for testing) and the OpenSC tooling,
• the operator is aware of HSM and its nature.

The HSM offers an encryption mechanism, but the unseal-keys and root-token have to be stored somewhere after they got encrypted. Currently there are two possible solutions for that:

• Some HSM devices can store a limited quantity of arbitrary data (like Nitrokey HSM), and Bank-Vaults can store the unseal-keys and root-token here.
• If the HSM does not support that, Bank-Vaults uses the HSM to encrypt the unseal-keys and root-token, then stores them in Kubernetes Secrets. We believe that it is safe to store these keys in Kubernetes Secrets in encrypted format.

Bank-Vaults offers the ability to use the pre-created the cryptographic encryption keys on the HSM device, or generate a key pair on the fly if there isn’t any with the specified label in the specified slot.

Since Bank-Vaults is written in Go, it uses the github.com/miekg/pkcs11 wrapper to pull in the PKCS11 library, to be more precise the p11 high-level wrapper .

Supported HSM solutions

Bank-Vaults currently supports the following HSM solutions:

• SoftHSM, recommended for testing
• NitroKey HSM.
• AWS CloudHSM supports the PKCS11 API as well, so it probably works, though it needs a custom Docker image.

3.9.1 - NitroKey HSM support (OpenSC)

Nitrokey HSM is a USB HSM device based on the OpenSC project. We are using NitroKey to develop real hardware-based HSM support for Bank-Vaults. This device is not a cryptographic accelerator, only key generation and the private key operations (sign and decrypt) are supported. Public key operations should be done by extracting the public key and working on the computer, and this is how it is implemented in Bank-Vaults. It is not possible to extract private keys from NitroKey HSM, the device is tamper-resistant.

This device supports only RSA based encryption/decryption, and thus this is implemented in Bank-Vaults currently. It supports ECC keys as well, but only for sign/verification operations.

To start using a NitroKey HSM, complete the following steps.

1. Install OpenSC and initialize the NitroKey HSM stick:

   brew install opensc
   sc-hsm-tool --initialize --label bank-vaults --pin banzai --so-pin banzaicloud
   pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults
   

2. Check that you got a keypair object in slot 0:

   pkcs11-tool --list-objects
   
   
   Using slot 0 with a present token (0x0)
   Public Key Object; RSA 2048 bits
     label:      bank-vaults
     ID:         a9548075b20243627e971873826ead172e932359
     Usage:      encrypt, verify, wrap
     Access:     none
   

   pkcs15-tool --list-keys
   
   Using reader with a card: Nitrokey Nitrokey HSM
   Private RSA Key [bank-vaults]
     Object Flags   : [0x03], private, modifiable
     Usage          : [0x0E], decrypt, sign, signRecover
     Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
     ModLength      : 2048
     Key ref        : 1 (0x01)
     Native         : yes
     Auth ID        : 01
     ID             : a9548075b20243627e971873826ead172e932359
     MD:guid        : a6b2832c-1dc5-f4ef-bb0f-7b3504f67015
   

3. If you are testing the HSM on macOS, setup minikube. Otherwise, continue with the next step.

4. Configure the operator to use NitroKey HSM for unsealing.

   You must adjust the unsealConfig section in the vault-operator configuration, so the operator can communicate with OpenSC HSM devices correctly. Adjust your configuration based on the following snippet:

     # This example relies on an OpenSC HSM (NitroKey HSM) device initialized and plugged in to the Kubernetes Node.
     unsealConfig:
       hsm:
         # OpenSC daemon is needed in this case to communicate with the device
         daemon: true
         # The HSM SO module path (opensc is built into the bank-vaults image)
         modulePath: /usr/lib/opensc-pkcs11.so
         # For OpenSC slotId is the preferred way instead of tokenLabel
         # (OpenSC appends/prepends some extra stuff to labels)
         slotId: 0
         pin: banzai # This can be specified in the BANK_VAULTS_HSM_PIN environment variable as well, from a Secret
         keyLabel: bank-vaults
   

5. Configure your Kubernetes node that has the HSM attached so Bank-Vaults can access it.

Setup on Minikube for testing (optional)

On macOS where you run Docker in VMs you need to do some extra steps before mounting your HSM device to Kubernetes. Complete the following steps to mount NitroKey into the minikube Kubernetes cluster:

1. Make sure that the Oracle VM VirtualBox Extension Pack for USB 2.0 support is installed.

   VBoxManage list extpacks
   

2. Remove the HSM device from your computer if it is already plugged in.

3. Specify VirtualBox as the VM backend for Minikube.

   minikube config set vm-driver virtualbox
   

4. Create a minikube cluster with the virtualbox driver and stop it, so you can modify the VM.

   minikube start
   minikube stop
   

5. Enable USB 2.0 support for the minikube VM.

   VBoxManage modifyvm minikube --usbehci on
   

6. Find the vendorid and productid for your Nitrokey HSM device.

   VBoxManage list usbhost
   
   VENDORID=0x20a0
   PRODUCTID=0x4230
   

7. Create a filter for it.

   VBoxManage usbfilter add 1 --target minikube --name "Nitrokey HSM" --vendorid ${VENDORID} --productid ${PRODUCTID}
   

8. Restart the minikube VM.

   minikube start
   

9. Plug in the USB device.

10. Check that minikube captured your NitorKey HSM.

    minikube ssh lsusb | grep ${VENDORID:2}:${PRODUCTID:2}
    

Now your minikube Kubernetes cluster has access to the HSM device through USB.

Kubernetes node setup

Some HSM vendors offer network daemons to enhance the reach of their HSM equipment to different servers. Unfortunately, there is no networking standard defined for PKCS11 access and thus currently Bank-Vaults has to be scheduled to the same node where the HSM device is attached directly (if not using a Cloud HSM).

Since the HSM is a hardware device connected to a physical node, Bank-Vaults has to find its way to that node. To make this work, create an HSM extended resource on the Kubernetes nodes for which the HSM device is plugged in. Extended resources must be advertised in integer amounts, for example, a Node can advertise four HSM devices, but not 4.5.

1. You need to patch the node to specify that it has an HSM device as a resource. Because of the integer constraint and because all Bank-Vaults related Pods has to land on a Node where an HSM resource is available we need to advertise two units for 1 device, one will be allocated by each Vault Pod and one by the Configurer. If you would like to run Vault in HA mode - multiple Vault instances in different nodes - you will need multiple HSM devices plugged into those nodes, having the same key and slot setup.

   kubectl proxy &
   
   NODE=minikube
   
   curl --header "Content-Type: application/json-patch+json" \
       --request PATCH \
       --data '[{"op": "add", "path": "/status/capacity/nitrokey.com~1hsm", "value": "2"}]' \
       http://localhost:8001/api/v1/nodes/${NODE}/status
   

   From now on, you can request the nitrokey.com/hsm resource in the PodSpec

2. Include the nitrokey.com/hsm resource in your PodSpec:

     # If using the NitroKey HSM example, that resource has to be part of the resource scheduling request.
     resources:
       hsmDaemon:
         requests:
           cpu: 100m
           memory: 64Mi
           nitrokey.com/hsm: 1
         limits:
           cpu: 200m
           memory: 128Mi
           nitrokey.com/hsm: 1
   

3. Apply the modified setup from scratch:

   kubectl delete vault vault
   kubectl delete pvc vault-file-vault-0
   kubectl delete secret vault-unseal-keys
   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.20.0/deploy/examples/cr-hsm-nitrokey.yaml
   

4. Check the logs that unsealing uses the NitroKey HSM device. Run the following command:

   kubectl logs -f vault-0 bank-vaults
   

   The output should be something like:

   time="2020-03-04T13:32:29Z" level=info msg="HSM Information {CryptokiVersion:{Major:2 Minor:20} ManufacturerID:OpenSC Project Flags:0 LibraryDescription:OpenSC smartcard framework LibraryVersion:{Major:0 Minor:20}}"
   time="2020-03-04T13:32:29Z" level=info msg="HSM Searching for slot in HSM slots [{ctx:0xc0000c0318 id:0}]"
   time="2020-03-04T13:32:29Z" level=info msg="found HSM slot 0 in HSM by slot ID"
   time="2020-03-04T13:32:29Z" level=info msg="HSM TokenInfo {Label:bank-vaults (UserPIN)\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ManufacturerID:www.CardContact.de Model:PKCS#15 emulated SerialNumber:DENK0200074 Flags:1037 MaxSessionCount:0 SessionCount:0 MaxRwSessionCount:0 RwSessionCount:0 MaxPinLen:15 MinPinLen:6 TotalPublicMemory:18446744073709551615 FreePublicMemory:18446744073709551615 TotalPrivateMemory:18446744073709551615 FreePrivateMemory:18446744073709551615 HardwareVersion:{Major:24 Minor:13} FirmwareVersion:{Major:3 Minor:3} UTCTime:\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}"
   time="2020-03-04T13:32:29Z" level=info msg="HSM SlotInfo for slot 0: {SlotDescription:Nitrokey Nitrokey HSM (DENK02000740000         ) 00 00 ManufacturerID:Nitrokey Flags:7 HardwareVersion:{Major:0 Minor:0} FirmwareVersion:{Major:0 Minor:0}}"
   time="2020-03-04T13:32:29Z" level=info msg="found objects with label \"bank-vaults\" in HSM"
   time="2020-03-04T13:32:29Z" level=info msg="this HSM device doesn't support encryption, extracting public key and doing encrytion on the computer"
   time="2020-03-04T13:32:29Z" level=info msg="no storage backend specified for HSM, using on device storage"
   time="2020-03-04T13:32:29Z" level=info msg="joining leader vault..."
   time="2020-03-04T13:32:29Z" level=info msg="vault metrics exporter enabled: :9091/metrics"
   [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
   - using env:	export GIN_MODE=release
   - using code:	gin.SetMode(gin.ReleaseMode)
   
   [GIN-debug] GET    /metrics                  --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)
   [GIN-debug] Listening and serving HTTP on :9091
   time="2020-03-04T13:32:30Z" level=info msg="initializing vault..."
   time="2020-03-04T13:32:30Z" level=info msg="initializing vault"
   time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-0
   time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-1
   time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-2
   time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-3
   time="2020-03-04T13:32:33Z" level=info msg="unseal key stored in key store" key=vault-unseal-4
   time="2020-03-04T13:32:33Z" level=info msg="root token stored in key store" key=vault-root
   time="2020-03-04T13:32:33Z" level=info msg="vault is sealed, unsealing"
   time="2020-03-04T13:32:39Z" level=info msg="successfully unsealed vault"
   

5. Find the unseal keys and the root token on the HSM:

   pkcs11-tool --list-objects
   

   Expected output:

   Using slot 0 with a present token (0x0)
   Public Key Object; RSA 2048 bits
     label:      bank-vaults
     ID:         a9548075b20243627e971873826ead172e932359
     Usage:      encrypt, verify, wrap
     Access:     none
   Data object 2168561792
     label:          'vault-test'
     application:    'vault-test'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168561168
     label:          'vault-unseal-0'
     application:    'vault-unseal-0'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168561264
     label:          'vault-unseal-1'
     application:    'vault-unseal-1'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168561360
     label:          'vault-unseal-2'
     application:    'vault-unseal-2'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168562304
     label:          'vault-unseal-3'
     application:    'vault-unseal-3'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168562400
     label:          'vault-unseal-4'
     application:    'vault-unseal-4'
     app_id:         <empty>
     flags:           modifiable
   Data object 2168562496
     label:          'vault-root'
     application:    'vault-root'
     app_id:         <empty>
     flags:           modifiable
   

6. If you don’t need the encryption keys or the unseal keys on the HSM anymore, you can delete them with the following commands:

   PIN=banzai
   
   # Delete the unseal keys and the root token
   for label in "vault-test" "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
   do
     pkcs11-tool --delete-object --type data --label ${label} --pin ${PIN}
   done
   
   # Delete the encryption key
   pkcs11-tool --delete-object --type privkey --label bank-vaults --pin ${PIN}
   

3.9.2 - SoftHSM support for testing

You can use SoftHSMv2 to implement and test software interacting with PKCS11 implementations. You can install it on macOS by running the following commands:

# Initializing SoftHSM to be able to create a working example (only for dev),
# sharing the HSM device is emulated with a pre-created keypair in the image.
brew install softhsm
softhsm2-util --init-token --free --label bank-vaults --so-pin banzai --pin banzai
pkcs11-tool --module /usr/local/lib/softhsm/libsofthsm2.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

To interact with SoftHSM when using the vault-operator, include the following unsealConfig snippet in the Vault CR:

  # This example relies on the SoftHSM device initialized in the Docker image.
  unsealConfig:
    hsm:
      # The HSM SO module path (softhsm is built into the bank-vaults image)
      modulePath: /usr/lib/softhsm/libsofthsm2.so 
      tokenLabel: bank-vaults
      pin: banzai
      keyLabel: bank-vaults