This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Overview

Bank-Vaults is a Vault swiss-army knife: a K8s operator, Go client with automatic token renewal, automatic configuration, multiple unseal options and more. A CLI tool to init, unseal and configure Vault (auth methods, secret engines). Direct secret injection into Pods.

Bank-Vaults provides the following tools for Hashicorp Vault to make its use easier and more automated:

Bank-Vaults overview

The package also includes Helm charts for installing the various components, and a collection of scripts to support advanced features (for example, dynamic SSH).

First step

1 - Getting started

Bank-Vaults is a swiss-army knife with multiple manifestations, so the first steps depend on what you want to achieve. Check one of the following guides to get an overview:

Deploy with Helm

We have some fully fledged, production-ready Helm charts for deploying:

With the help of these charts you can run a HA Vault instance with automatic initialization, unsealing, and external configuration which would otherwise be a tedious manual operation. Also secrets from Vault can be injected into your Pods directly as environment variables (without using Kubernetes Secrets). These charts can be used easily for development purposes as well.

Note: Starting with Bank-Vaults version 1.6.0, only Helm 3 is supported. If you have installed the chart with Helm 2 and now you are trying to upgrade with Helm3, see the Bank-Vaults 1.6.0 release notes for detailed instructions.

Deploy a local Vault operator

This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets.

  1. Install the Bank-Vaults operator:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --install vault-operator banzaicloud-stable/vault-operator
    
  2. Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:

    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml
    
  3. Wait a few seconds, then check the operator and the vault pods:

    kubectl get pods
    

    Expected output:

    NAME                                                        READY     STATUS    RESTARTS   AGE
    vault-0                                                     3/3       Running   0          10s
    vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
    vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s
    
  4. Configure your Vault client to access the Vault instance running in the vault-0 pod.

    1. Port-forward into the pod:

      kubectl port-forward vault-0 8200 &
      
    2. Set the address of the Vault instance.

      export VAULT_ADDR=https://127.0.0.1:8200
      
    3. Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):

      kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
      export VAULT_CACERT=$PWD/vault-ca.crt
      

      Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true

    4. Check that you can access the vault:

      vault status
      

      Expected output:

      Key             Value
      ---             -----
      Seal Type       shamir
      Initialized     true
      Sealed          false
      Total Shares    5
      Threshold       3
      Version         1.5.4
      Cluster Name    vault-cluster-27ecd0e6
      Cluster ID      ed5492f3-7ef3-c600-aef3-bd77897fd1e7
      HA Enabled      false
      
    5. To authenticate to Vault, you can access its root token by running:

      export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
      

      Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.

    6. Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).

For other configuration examples of the Vault CustomResource, see the YAML files in the operator/deploy directory of the project (we use these for testing). After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:

kubectl delete -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
kubectl delete -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml

Deploy the mutating webhook

You can deploy the Vault Secrets Webhook using Helm. Note that:

  • The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
  • The deployed RBAC objects contain the necessary permissions fo running the webhook.

Prerequisites

  • The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
  • The chart requires Helm 3.
  • To interact with Vault (for example, for testing), the vault command line client must be installed on your computer.
  • You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.

Deploy the webhook

  1. Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:

    kubectl create namespace vault-infra
    kubectl label namespace vault-infra name=vault-infra
    
  2. Deploy the vault-secrets-webhook chart:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook
    

    For further details, see the webhook’s Helm chart repository.

  3. Check that the pods are running:

    kubectl get pods --namespace vault-infra
    NAME                                     READY   STATUS    RESTARTS   AGE
    vault-secrets-webhook-58b97c8d6d-qfx8c   1/1     Running   0          22s
    vault-secrets-webhook-58b97c8d6d-rthgd   1/1     Running   0          22s
    
  4. Write a secret into Vault (the Vault CLI must be installed on your computer):

    vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
    

    Expected output:

    Key              Value
    ---              -----
    created_time     2020-11-04T11:39:01.863988395Z
    deletion_time    n/a
    destroyed        false
    version          1
    
  5. Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:

  6. Check the mutated deployment.

    kubectl describe deployment vault-test
    

    The output should look similar to the following:

    Name:                   vault-test
    Namespace:              default
    CreationTimestamp:      Wed, 04 Nov 2020 12:44:18 +0100
    Labels:                 <none>
    Annotations:            deployment.kubernetes.io/revision: 1
    Selector:               app.kubernetes.io/name=vault
    Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
    StrategyType:           RollingUpdate
    MinReadySeconds:        0
    RollingUpdateStrategy:  25% max unavailable, 25% max surge
    Pod Template:
      Labels:           app.kubernetes.io/name=vault
      Annotations:      vault.security.banzaicloud.io/vault-addr: https://vault:8200
                        vault.security.banzaicloud.io/vault-agent: false
                        vault.security.banzaicloud.io/vault-path: kubernetes
                        vault.security.banzaicloud.io/vault-role: default
                        vault.security.banzaicloud.io/vault-skip-verify: false
                        vault.security.banzaicloud.io/vault-tls-secret: vault-tls
      Service Account:  default
      Containers:
       alpine:
        Image:      alpine
        Port:       <none>
        Host Port:  <none>
        Command:
          sh
          -c
          echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000
        Environment:
          AWS_SECRET_ACCESS_KEY:  vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
        Mounts:                   <none>
      Volumes:                    <none>
    Conditions:
      Type           Status  Reason
      ----           ------  ------
      Available      True    MinimumReplicasAvailable
      Progressing    True    NewReplicaSetAvailable
    OldReplicaSets:  <none>
    NewReplicaSet:   vault-test-55c569f9 (1/1 replicas created)
    Events:
      Type    Reason             Age   From                   Message
      ----    ------             ----  ----                   -------
      Normal  ScalingReplicaSet  29s   deployment-controller  Scaled up replica set vault-test-55c569f9 to 1
    

    As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.

Install the CLI tool

On macOs, you can directly install the CLI from Homebrew:

brew install banzaicloud/tap/bank-vaults

Alternatively, fetch the source code and compile it using go get:

go get github.com/bank-vaults/bank-vaults/cmd/bank-vaults
go get github.com/bank-vaults/bank-vaults/cmd/vault-env

Docker images

If you want to build upon our Docker images, you can find them on Docker Hub:

docker pull banzaicloud/bank-vaults
docker pull banzaicloud/vault-operator
docker pull banzaicloud/vault-env

1.1 - Deploy vault into a custom namespace

To deploy vault into a custom namespace (not into default), you have to deploy the vault CustomResource to the custom namespace. Also, you have to use the custom namespace in the following fields:

In the RBAC resources:

In the Vault CR:

2 - External configuration for Vault

In addition to the standard Vault configuration, the operator and CLI can continuously configure Vault using an external YAML/JSON configuration. That way you can configure Vault declaratively using your usual automation tools and workflow.

The following sections describe the configuration sections you can use.

2.1 - Fully or partially purging unmanaged configuration in Vault

Bank-Vaults gives you a full control over Vault in a declarative style by removing any unmanaged configuration.

By enabling purgeUnmanagedConfig you keep Vault configuration up-to-date. So if you added a policy using Bank-Vaults then removed it from the configuration, Bank-Vaults will remove it from Vault too. In other words, if you enabled purgeUnmanagedConfig then any changes not in Bank-Vaults configuration will be removed (including manual changes).

Note:

This feature is destructive, so be carful when you enable it especially for the first time because it could delete all data in your Vault. We recommend you to test it a non-production environment first.

This feature is disabled by default and it needs to be enabled explicitly in your configuration.

Mechanism

Bank-Vaults handles unmanaged configuration by simply comparing what in Bank-Vaults configuration (the desired state) and what’s already in Vault (the actual state), then it removes any differences that are not in Bank-Vaults configuration.

Fully purge unmanaged configuration

You can remove all unmanaged configuration by enabling the purge option as following:

purgeUnmanagedConfig:
  enabled: true

Partially purge unmanaged configuration

You can also enable the purge feature for some of the config by excluding any config that you don’t want to purge its unmanaged config.

It could be done by explicitly exclude the Vault configuration that you don’t want to mange:

purgeUnmanagedConfig:
  enabled: true
  exclude:
    secrets: true

This will remove any unmanaged or manual changes in Vault but it will leave secrets untouched. So if you enabled a new secret engine manually (and it’s not in Bank-Vaults configuration), Bank-Vaults will not remove it.

2.2 - Audit devices

You can configure Audit Devices in Vault (File, Syslog, Socket).

audit:
  - type: file
    description: "File based audit logging device"
    options:
      file_path: /tmp/vault.log

2.3 - Authentication

You can configure Auth Methods in Vault.

Currently the following auth methods are supported:

AppRole auth method

Allow machines/apps to authenticate with Vault-defined roles. For details, see the official Vault documentation.

auth:
  - type: approle
    roles:
    - name: default
      policies: allow_secrets
      secret_id_ttl: 10m
      token_num_uses: 10
      token_ttl: 20m
      token_max_ttl: 30m
      secret_id_num_uses: 40

AWS auth method

Creating roles in Vault which can be used for AWS IAM based authentication.

auth:
  - type: aws
    # Make the auth provider visible in the web ui
    # See https://www.vaultproject.io/api/system/auth.html#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      access_key: VKIAJBRHKH6EVTTNXDHA
      secret_key: vCtSM8ZUEQ3mOFVlYPBQkf2sO6F/W7a5TVzrl3Oj
      iam_server_id_header_value: vault-dev.example.com # consider setting this to the Vault server's DNS name
    crossaccountrole:
    # Add cross account number and role to assume in the cross account
    # https://www.vaultproject.io/api/auth/aws/index.html#create-sts-role
    - sts_account: 12345671234
      sts_role: arn:aws:iam::12345671234:role/crossaccountrole
    roles:
    # Add roles for AWS instances or principals
    # See https://www.vaultproject.io/api/auth/aws/index.html#create-role
    - name: dev-role-iam
      bound_iam_principal_arn: arn:aws:iam::123456789012:role/dev-vault
      policies: allow_secrets
      period: 1h
    - name: cross-account-role
      bound_iam_principal_arn: arn:aws:iam::12345671234:role/crossaccountrole
      policies: allow_secrets
      period: 1h

Azure auth method

The Azure auth method allows authentication against Vault using Azure Active Directory credentials for more information.

auth:
  - type: azure
    config:
      tenant_id: 00000000-0000-0000-0000-000000000000
      resource: https://vault-dev.example.com
      client_id: 00000000-0000-0000-0000-000000000000
      client_secret: 00000000-0000-0000-0000-000000000000
    roles:
    # Add roles for azure identities
    # See https://www.vaultproject.io/api/auth/azure/index.html#create-role
      - name: dev-mi
        policies: allow_secrets
        bound_subscription_ids: 
          - "00000000-0000-0000-0000-000000000000"
        bound_service_principal_ids: 
          - "00000000-0000-0000-0000-000000000000"

GCP auth method

Create roles in Vault which can be used for GCP IAM based authentication.

auth:
  - type: gcp
    # Make the auth provider visible in the web ui
    # See https://www.vaultproject.io/api/system/auth.html#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      # Credentials context is service account's key. Can download when you create a key for service account. 
      # No need to manually create it. Just paste the json context as multiline yaml.
      credentials: -|
        {
          "type": "service_account",
          "project_id": "PROJECT_ID",
          "private_key_id": "KEY_ID",
          "private_key": "-----BEGIN PRIVATE KEY-----.....-----END PRIVATE KEY-----\n",
          "client_email": "SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com",
          "client_id": "CLIENT_ID",
          "auth_uri": "https://accounts.google.com/o/oauth2/auth",
          "token_uri": "https://oauth2.googleapis.com/token",
          "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
          "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/SERVICE_ACCOUNT%40PROJECT_ID.iam.gserviceaccount.com"
        }        
    roles:
    # Add roles for gcp service account
    # See https://www.vaultproject.io/api/auth/gcp/index.html#create-role
    - name: user-role
      type: iam
      project_id: PROJECT_ID
      policies: "readonly_secrets"
      bound_service_accounts: "USER_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"
    - name: admin-role
      type: iam
      project_id: PROJECT_ID
      policies: "allow_secrets"
      bound_service_accounts: "ADMIN_SERVICE_ACCOUNT@PROJECT_ID.iam.gserviceaccount.com"

GitHub auth method

Create team mappings in Vault which can be used later on for the GitHub authentication.

auth:
  - type: github
    # Make the auth provider visible in the web ui
    # See https://www.vaultproject.io/api/system/auth.html#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      organization: banzaicloud
    map:
      # Map the banzaicloud GitHub team on to the dev policy in Vault
      teams:
        dev: dev
      # Map my username (bonifaido) to the root policy in Vault
      users:
        bonifaido: allow_secrets

JWT auth method

Create roles in Vault which can be used for JWT-based authentication.

auth:
  - type: jwt
    path: jwt
    config:
      oidc_discovery_url: https://myco.auth0.com/
    roles:
    - name: role1
      bound_audiences:
        - https://vault.plugin.auth.jwt.test
      user_claim: https://vault/user
      groups_claim: https://vault/groups
      policies: allow_secrets
      ttl: 1h

Kubernetes auth method

Use the Kubernetes auth method to authenticate with Vault using a Kubernetes Service Account Token.

auth:
  - type: kubernetes
    # If you want to configure with specific kubernetes service account instead of default service account
    # https://www.vaultproject.io/docs/auth/kubernetes.html
    # config:
    #   token_reviewer_jwt: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9....
    #   kubernetes_ca_cert: |
    #     -----BEGIN CERTIFICATE-----
    #     ...
    #     -----END CERTIFICATE-----
    #   kubernetes_host: https://192.168.64.42:8443
    # Allows creating roles in Vault which can be used later on for the Kubernetes based
    # authentication.
    #  See https://www.vaultproject.io/docs/auth/kubernetes.html#creating-a-role for
    # more information.
    roles:
      # Allow every pod in the default namespace to use the secret kv store
      - name: default
        bound_service_account_names: default
        bound_service_account_namespaces: default
        policies: allow_secrets
        ttl: 1h

LDAP auth method

Create group mappings in Vault which can be used for LDAP based authentication.

  • To start an LDAP test server, run: docker run -it –rm -p 389:389 -e LDAP_TLS=false –name ldap osixia/openldap
  • To start an LDAP admin server, run: docker run -it –rm -p 6443:443 –link ldap:ldap -e PHPLDAPADMIN_LDAP_HOSTS=ldap -e PHPLDAPADMIN_LDAP_CLIENT_TLS=false osixia/phpldapadmin
auth:
  - type: ldap
    description: LDAP directory auth.
    # add mount options
    # See https://www.vaultproject.io/api/system/auth.html#config for more
    # information.
    options:
      listing_visibility: "unauth"
    config:
      url: ldap://localhost
      binddn: "cn=admin,dc=example,dc=org"
      bindpass: "admin"
      userattr: uid
      userdn: "ou=users,dc=example,dc=org"
      groupdn: "ou=groups,dc=example,dc=org"
    groups:
      # Map the banzaicloud dev team on GitHub to the dev policy in Vault
      developers:
        policies: allow_secrets
    # Map myself to the allow_secrets policy in Vault
    users:
      bonifaido:
        groups: developers
        policies: allow_secrets

2.4 - Plugins

To register a new plugin in Vault’s plugin catalog, set the plugin_directory option in the Vault server configuration to the directory where the plugin binary is located. Also, for some plugins readOnlyRootFilesystem Pod Security Policy should be disabled to allow RPC communication between plugin and Vault server via Unix socket. For details, see the Hashicorp Go plugin documentation.

plugins:
  - plugin_name: ethereum-plugin
    command: ethereum-vault-plugin --ca-cert=/vault/tls/client/ca.crt --client-cert=/vault/tls/server/server.crt --client-key=/vault/tls/server/server.key
    sha256: 62fb461a8743f2a0af31d998074b58bb1a589ec1d28da3a2a5e8e5820d2c6e0a
    type: secret

2.5 - Policies

You can create policies in Vault, and later use these policies in roles for the Kubernetes-based authentication. For details, see Policies in the official Vault documentation.

policies:
  - name: allow_secrets
    rules: path "secret/*" {
             capabilities = ["create", "read", "update", "delete", "list"]
           }
  - name: readonly_secrets
    rules: path "secret/*" {
             capabilities = ["read", "list"]
           }

2.6 - Secrets engines

You can configure Secrets Engines in Vault. The Key-Value, Database, and SSH values are tested, but the configuration is free form, so probably others work as well.

AWS

The AWS secrets engine generates AWS access credentials dynamically based on IAM policies.

secrets:
  - type: aws
    path: aws
    description: AWS Secret Backend
    configuration:
        config: 
          - name: root
            access_key: "${env `AWS_ACCESS_KEY_ID`}"
            secret_key: "${env `AWS_SECRET_ACCESS_KEY`}"
            region: us-east-1
        roles:
          - credential_type: iam_user
            policy_arns: arn-of-policy
            name: my-aws-role

Database

This plugin stores database credentials dynamically based on configured roles for the MySQL/MariaDB database.

secrets:
  - type: database
    description: MySQL Database secret engine.
    configuration:
      config:
        - name: my-mysql
          plugin_name: "mysql-database-plugin"
          connection_url: "{{username}}:{{password}}@tcp(127.0.0.1:3306)/"
          allowed_roles: [pipeline]
          username: "${env `ROOT_USERNAME`}" # Example how to read environment variables
          password: "${env `ROOT_PASSWORD`}"
      roles:
        - name: pipeline
          db_name: my-mysql
          creation_statements: "GRANT ALL ON *.* TO '{{name}}'@'%' IDENTIFIED BY '{{password}}';"
          default_ttl: "10m"
          max_ttl: "24h"

Identity Groups

Allows you to configure identity groups.

Note:

Only external groups are supported at the moment through the use of group-aliases. For supported authentication backends (for example JWT, which automatically matches those aliases to groups returned by the backend) the configuration files for the groups and group-aliases need to be parsed after the authentication backend has been mounted. Ideally they should be in the same file to avoid of errors.

groups:
  - name: admin
    policies:
      - admin
    metadata:
      admin: "true"
      priviliged: "true"
    type: external

group-aliases:
  - name: admin
    mountpath: jwt
    group: admin

Key-Values

This plugin stores arbitrary secrets within the configured physical storage for Vault.

secrets:
  - path: secret
    type: kv
    description: General secrets.
    options:
      version: 2
    configuration:
      config:
        - max_versions: 100

Non-default plugin path

Mounts a non-default plugin’s path.

  - path: ethereum-gateway
    type: plugin
    plugin_name: ethereum-plugin
    description: Immutability's Ethereum Wallet

PKI

The PKI secrets engine generates X.509 certificates.

secrets:
  - type: pki
    description: Vault PKI Backend
    config:
      default_lease_ttl: 168h
      max_lease_ttl: 720h
    configuration:
      config:
      - name: urls
        issuing_certificates: https://vault.default:8200/v1/pki/ca
        crl_distribution_points: https://vault.default:8200/v1/pki/crl
      root/generate:
      - name: internal
        common_name: vault.default
      roles:
      - name: default
        allowed_domains: localhost,pod,svc,default
        allow_subdomains: true
        generate_lease: true
        ttl: 30m

RabbitMQ

The RabbitMQ secrets engine generates user credentials dynamically based on configured permissions and virtual hosts.

To start a RabbitMQ test server, run: docker run -it –rm -p 15672:15672 rabbitmq:3.7-management-alpine

secrets:
  - type: rabbitmq
    description: local-rabbit
    configuration:
      config:
        - name: connection
          connection_uri: "http://localhost:15672"
          username: guest
          password: guest
      roles:
        - name: prod_role
          vhosts: '{"/web":{"write": "production_.*", "read": "production_.*"}}'

SSH

Create a named Vault role for signing SSH client keys.

secrets:
  - type: ssh
    path: ssh-client-signer
    description: SSH Client Key Signing.
    configuration:
      config:
        - name: ca
          generate_signing_key: "true"
      roles:
        - name: my-role
          allow_user_certificates: "true"
          allowed_users: "*"
          key_type: "ca"
          default_user: "ubuntu"
          ttl: "24h"
          default_extensions:
            permit-pty: ""
            permit-port-forwarding: ""
            permit-agent-forwarding: ""

2.7 - Startup secrets

Allows writing some secrets to Vault (useful for development purposes). For details, see the Key-Value secrets engine.

startupSecrets:
  - type: kv
    path: secret/data/accounts/aws
    data:
      data:
        AWS_ACCESS_KEY_ID: secretId
        AWS_SECRET_ACCESS_KEY: s3cr3t

3 - Environment variables

Add environment variables. See the database secret engine section for usage. Further information:

envsConfig:
  - name: ROOT_USERNAME
    valueFrom:
      secretKeyRef:
        name: mysql-login
        key: user
  - name: ROOT_PASSWORD
    valueFrom:
      secretKeyRef:
        name: mysql-login
        key: password

4 - Operator

The Vault operator builds on Bank-Vaults features such as:

  • external, API based configuration (secret engines, auth methods, policies) to automatically re/configure a Vault cluster
  • automatic unsealing (AWS, GCE, Azure, Alibaba, Kubernetes Secrets (for dev purposes), Oracle)
  • TLS support

The operator flow is the following:

operator

The source code can be found in the operator directory.

The operator requires the following cloud permissions.

Deploy a local Vault operator

This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets.

  1. Install the Bank-Vaults operator:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --install vault-operator banzaicloud-stable/vault-operator
    
  2. Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:

    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml
    
  3. Wait a few seconds, then check the operator and the vault pods:

    kubectl get pods
    

    Expected output:

    NAME                                                        READY     STATUS    RESTARTS   AGE
    vault-0                                                     3/3       Running   0          10s
    vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
    vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s
    
  4. Configure your Vault client to access the Vault instance running in the vault-0 pod.

    1. Port-forward into the pod:

      kubectl port-forward vault-0 8200 &
      
    2. Set the address of the Vault instance.

      export VAULT_ADDR=https://127.0.0.1:8200
      
    3. Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):

      kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
      export VAULT_CACERT=$PWD/vault-ca.crt
      

      Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true

    4. Check that you can access the vault:

      vault status
      

      Expected output:

      Key             Value
      ---             -----
      Seal Type       shamir
      Initialized     true
      Sealed          false
      Total Shares    5
      Threshold       3
      Version         1.5.4
      Cluster Name    vault-cluster-27ecd0e6
      Cluster ID      ed5492f3-7ef3-c600-aef3-bd77897fd1e7
      HA Enabled      false
      
    5. To authenticate to Vault, you can access its root token by running:

      export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
      

      Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.

    6. Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).

For other configuration examples of the Vault CustomResource, see the YAML files in the operator/deploy directory of the project (we use these for testing). After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:

kubectl delete -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
kubectl delete -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr.yaml

HA setup with Raft

In a production environment you want to run Vault as a cluster. The following CR creates a 3-node Vault instance that uses the Raft storage backend:

  1. Install the Bank-Vaults operator:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --install vault-operator banzaicloud-stable/vault-operator
    
  2. Create a Vault instance using the cr-raft.yaml custom resource. This will create a Kubernetes CustomResource called vault that uses the Raft backend:

    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/rbac.yaml
    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-raft.yaml
    

CAUTION:

Backing up the storage backend to prevent data loss, is not handled by the Vault operator. We recommend using Velero for backups.

Pod anti-affinity

If you want to setup pod anti-affinity, you can set podAntiAffinity vault with a topologyKey value. For example, you can use failure-domain.beta.kubernetes.io/zone to force K8S deploy vault on multi AZ.

Delete a resource created by the operator

If you manually delete a resource that the Bank-Vaults operator has created (for example, the Ingress resource), the operator automatically recreates it every 30 seconds. If it doesn’t, then something went wrong, or the operator is not running. In this case, check the logs of the operator.

4.1 - Running Vault with external end to end encryption

This document assumes you have a working Kubernetes cluster which has a:

  • Working install of Vault.
  • That you have a working knowledge of Kubernetes.
  • A working install of helm
  • A working knowledge of Kubernetes ingress
  • A valid external (www.example.com) SSL certificate, verified by your provider as a Kubernetes secret.

Background

The bank-vaults operator takes care of creating and maintaining internal cluster communications but if you wish to use your vault install outside of your Kubernetes cluster what is the best way to maintain a secure state. Creating a standard Ingress object will reverse proxy these requests to your vault instance but this is a hand off between the external SSL connection and the internal one. This might not be acceptable under some circumstances, for example, if you have to adhere to strict security standards.

Workflow

Here we will create a separate TCP listener for vault using a custom SSL certificate on an external domain of your choosing. We will then install a unique ingress-nginx controller allowing SSL pass through. SSL Pass through comes with a performance hit, so you would not use this on a production website or ingress-controller that has a lot of traffic.

Install

ingress-nginx

values.yaml

controller:
  electionID: vault-ingress-controller-leader
  ingressClass: nginx-vault
  extraArgs:
    enable-ssl-passthrough:
  publishService:
    enabled: true
  scope:
    enabled: true
  replicaCount: 2
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: release
            operator: In
            values: ["vault-ingress"]
        topologyKey: kubernetes.io/hostname

Install nginx-ingress via helm

helm install nginx-stable/nginx-ingress --name my-release -f values.yaml

Configuration

SSL Secret example:

apiVersion: v1
data:
  tls.crt: LS0tLS1......=
  tls.key: LS0tLS.......==
kind: Secret
metadata:
  labels:
    ssl: "true"
    tls: "true"
  name: wildcard.example.com
type: Opaque

CR Vault Config:

---
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
  namespace: secrets
spec:
  size: 2
  image: vault:1.1.2
  bankVaultsImage: banzaicloud/bank-vaults:0.4.16

  # A YAML representation of a final vault config file.
  # See https://www.vaultproject.io/docs/configuration/ for more information.
  config:
    listener:
      - tcp:
          address: "0.0.0.0:8200"
          tls_cert_file: /vault/tls/server.crt
          tls_key_file: /vault/tls/server.key
      - tcp:
          address: "0.0.0.0:8300"
          tls_cert_file: /etc/ingress-tls/tls.crt
          tls_key_file: /etc/ingress-tls/tls.key
    api_addr: https://vault:8200
    cluster_addr: https://vault:8201
    ui: true

CR Service:

  # Specify the Service's type where the Vault Service is exposed
  serviceType: ClusterIP
  servicePorts:
    api-port: 8200
    cluster-port: 8201
    ext-api-port: 8300
    ext-clu-port: 8301

Mount the secret into your vault pod

  volumes:
    - name: wildcard-ssl
      secret:
        defaultMode: 420
        secretName: wildcard.example.com

  volumeMounts:
    - name: wildcard-ssl
      mountPath: /etc/ingress-tls

CR Ingress:

  # Request an Ingress controller with the default configuration
  ingress:
    annotations:
      kubernetes.io/ingress.allow-http: "false"
      kubernetes.io/ingress.class: "nginx-vault"
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      nginx.ingress.kubernetes.io/whitelist-source-range: "127.0.0.1"

    spec:
      rules:
        - host: vault.example.com
          http:
            paths:
              - path: /
                backend:
                  serviceName: vault
                  servicePort: 8300

4.2 - Using templates for injecting dynamic configuration

Background

When configuring a Vault object via the externalConfig property, sometimes it’s convenient (or necessary) to inject settings that are only known at runtime, for example:

  • secrets that you don’t want to store in source control
  • dynamic resources managed elsewhere
  • computations based on multiple values (string or arithmetic operations).

For these cases, the operator supports parameterized templating. The vault-configurer component evaluates the templates and injects the rendered configuration into Vault.

This templating is based on Go templates, extended by Sprig, with some custom functions available specifically for bank-vaults (for example, to decrypt strings using the AWS Key Management Service or the Cloud Key Management Service of the Google Cloud Platform).

Using templates

To avoid confusion and potential parsing errors (and interference with other templating tools like Helm), the templates don’t use the default delimiters that Go templates use ({{ and }}). Instead, use the following characters:

  • ${ for the left delimiter
  • } for the right one.
  • To quote parameters being passed to functions, surround them with backticks (`) instead.

For example, to call the env function, you can use this in your manifest:

password: "${ env `MY_ENVIRONMENT_VARIABLE` }"

In this case, vault-configurer evaluates the value of MY_ENVIRONMENT_VARIABLE at runtime (assuming it was properly injected), and sets the result as the value of the password field.

Note that you can also use Sprig functions and custom Kubernetes-related functions in your templates.

For a detailed example, see the Using templates for injecting dynamic configuration in Vault blog post.

Sprig functions

In addition to the default functions in Go templates, you can also use Sprig functions in your configuration.

CAUTION:

Use only functions that return a string, otherwise the generated configuration is invalid.

Custom functions

To provide functionality that’s more Kubernetes-friendly and cloud-native, bank-vaults provides a few additional functions not available in Sprig or Go. The functions and their parameters (in the order they should go in the function) are documented below.

awskms

Takes a base64-encoded, KMS-encrypted string and returns the decrypted string. Additionally, the function takes an optional second parameter for any encryption context that might be required for decrypting. If any encryption context is required, the function will take any number of additional parameters, each of which should be a key-value pair (separated by a = character), corresponding to the full context.

Note: This function assumes that the vault-configurer pod has the appropriate AWS IAM credentials and permissions to decrypt the given string. You can inject the AWS IAM credentials by using Kubernetes secrets as environment variables, an EC2 instance role, kube2iam, or EKS IAM roles, and so on.

ParameterTypeRequired
encodedStringBase64-encoded stringYes
encryptionContextVariadic list of stringsNo

For example:

password: '${ awskms (env `ENCRYPTED_DB_CREDS`) }'

You can also nest functions in the template, for example:

password: '${ awskms (blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1`) }'

gcpkms

Takes a base64-encoded string, encrypted with a Google Cloud Platform (GCP) symmetric key and returns the decrypted string.

Note: This function assumes that the vault-configurer pod has the appropriate GCP IAM credentials and permissions to decrypt the given string. You can inject the GCP IAM credentials by using Kubernetes secrets as environment variables, or they can be acquired via a service account authentication, and so on.

ParameterTypeRequired
encodedStringBase64-encoded stringYes
projectIdStringYes
locationStringYes
keyRingStringYes
keyStringYes

blob

Reads the content of a blob from disk (file) or from cloud blob storage services (object storage) at the given URL and returns it. This assumes that the path exists, and is readable by vault-configurer.

Valid values for the URL parameters are listed below, for more fine grained options check the documentation of the underlying library:

  • file:///path/to/dir/file
  • s3://my-bucket/object?region=us-west-1
  • gs://my-bucket/object
  • azblob://my-container/blob

Note: This function assumes that the vault-configurer pod has the appropriate rights to access the given cloud service. For details, see the awskms and gcpkms functions.

ParameterTypeRequired
urlStringYes

For example:

password: '${ blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1` }'

You can also nest functions in the template, for example:

password: '${ awskms (blob `s3://bank-vaults/encrypted/db-creds?region=eu-west-1`) }'

file

Reads the content of a file from disk at the given path and returns it. This assumes that the file exists, it’s mounted, and readable by vault-configurer.

ParameterTypeRequired
pathStringYes

accessor

Looks up the accessor id of the given auth path and returns it. This function is only useful in policies that use templated policies, to generalize the <mount accessor> field.

ParameterTypeRequired
pathStringYes

For example:

policies:
  - name: allow_secrets
    rules: path "secret/data/{{identity.entity.aliases.${ accessor `kubernetes/` }.metadata.service_account_namespace}}/*" {
             capabilities = ["read"]
           }

4.3 - Upgrade Vault with the Bank-Vaults operator

To upgrade Vault using the Bank-Vaults operator, complete the following steps.

  1. Check the release notes of Vault for any special upgrade instructions. Usually there are no instructions, but it’s better to be safe than sorry.
  2. Adjust the spec.image in field in the Vault custom resource.
  3. The Bank-Vaults operator updates the statefulset. (It does not take the HA leader into account in HA scenarios, but this has never caused any issues so far.)

4.4 - Operator Configuration for Functioning Webhook Secrets Mutation

You can find several examples of the vault operator CR manifest in the project repository. The following examples use only this vanilla CR to demonstrate some main points about how to properly configure the operator for secrets mutations to function.

This document does not attempt to explain every possible scenario with respect to the CRs in the aforementioned directory, but instead attempts to explain at a high level the important aspects of the CR, so that you can determine how best to configure your operator.

Main points

Some important aspects of the operator and its configuration with respect to secrets mutation are:

  • The vault operator instantiates:
    • the vault configurer pod(s),
    • the vault pod(s),
    • the vault-secrets-webhook pod(s).
  • The vault configurer:
    • unseals the vault,
    • configures vault with policies, roles, and so on.
  • vault-secrets-webhook does nothing more than:
    • monitors cluster for resources with specific annotations for secrets injection, and
    • integrates with vault API to answer secrets requests from those resources for requested secrets.
    • For pods using environment secrets, it injects a binary vault-env into pods and updates ENTRYPOINT to run vault-env CMD instead of CMD. vault-env intercepts requests for env secrets requests during runtime of pod and upon such requests makes vault API call for requested secret injecting secret into environment variable so CMD works with proper secrets.
  • Vault
    • the secrets workhorse
    • surfaces a RESTful API for secrets management

CR configuration properties

This section goes over some important properties of the CR and their purpose.

Vault’s service account

This is the serviceaccount where Vault will be running. The Configurer runs in the same namespace and should have the same service account. The operator assigns this serviceaccount to Vault.

  # Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
  serviceAccount: vault

caNamespaces

In order for vault communication to be encrypted, valid TLS certificates need to be used. The following property automatically creates TLS certificate secrets for the namespaces specified here. Notice that this is a list, so you can specify multiple namespaces per line, or use the splat or wildcard asterisk to specify all namespaces:

  # Support for distributing the generated CA certificate Secret to other namespaces.
  # Define a list of namespaces or use ["*"] for all namespaces.
  caNamespaces:
    - "*"

Vault Config

The following is simply a YAML representation (as the comment says) for the Vault configuration you want to run. This is the configuration that vault configurer uses to configure your running Vault:

  # A YAML representation of a final vault config file.
  config:
    api_addr: https://vault:8200
    cluster_addr: https://${.Env.POD_NAME}:8201
    listener:
      tcp:
        address: 0.0.0.0:8200
        # Commenting the following line and deleting tls_cert_file and tls_key_file disables TLS
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    storage:
      file:
        path: "${ .Env.VAULT_STORAGE_FILE }"
    ui: true
  credentialsConfig:
    env: ""
    path: ""
    secretName: ""
  etcdSize: 0
  etcdVersion: ""
  externalConfig:
    policies:
      - name: allow_secrets
        rules: path "secret/*" {
          capabilities = ["create", "read", "update", "delete", "list"]
          }
    auth:
      - type: kubernetes
        roles:
          # Allow every pod in the default namespace to use the secret kv store
          - name: default
            bound_service_account_names:
              - external-secrets
              - vault
              - dex
            bound_service_account_namespaces:
              - external-secrets
              - vault
              - dex
              - auth-system
              - loki
              - grafana
            policies:
              - allow_secrets
            ttl: 1h

          # Allow mutation of secrets using secrets-mutation annotation to use the secret kv store
          - name: secretsmutation
            bound_service_account_names:
              - vault-secrets-webhook
            bound_service_account_namespaces:
              - vault-secrets-webhook
            policies:
              - allow_secrets
            ttl: 1h

externalConfig

The externalConfig portion of this CR example correlates to Kubernetes configuration as specified by .auth[].type.

This YAML representation of configuration is flexible enough to work with any auth methods available to Vault as documented in the Vault documentation. For now, we’ll stick with this kubernetes configuration.

externalConfig.purgeUnmanagedConfig

Delete any configuration that in Vault but not in externalConfig. For more details please check Purge unmanaged configuration

externalConfig.policies

Correlates 1:1 to the creation of the specified policy in conjunction with Vault policies.

externalConfig.auth[].type

- type: kubernetes - specifies to configure Vault to use Kubernetes authentication

Other types are yet to be documented with respect to the operator configuration.

externalConfig.auth[].roles[]

Correlates to Creating Kubernetes roles. Some important nuances here are:

  • Vault does not respect inline secrets serviceaccount annotations, so the namespace of any serviceaccount annotations for secrets are irrelevant to getting inline secrets mutations functioning.
  • Instead, the serviceaccount of the vault-secrets-webhook pod(s) should be used to configure the bound_service_account_names and bound_service_account_namespaces for inline secrets to mutate.
  • Pod serviceaccounts, however, are respected so bound_service_account_namespaces and bound_service_account_names for environment mutations must identify such of the running pods.

Note: There are two roles specified in the YAML example above: one for pods, and one for inline secrets mutations. While this was not strictly required, it makes for cleaner implementation.

5 - Mutating Webhook

How the webhook works - overview

Kubernetes secrets are the standard way in which applications consume secrets and credentials on Kubernetes. Any secret that is securely stored in Vault and then unsealed for consumption eventually ends up as a Kubernetes secret. However, despite their name, Kubernetes secrets are not exactly secure, since they are only base64 encoded.

The mutating webhook of Bank-Vaults is a solution that bypasses the Kubernetes secrets mechanism and injects the secrets retrieved from Vault directly into the Pods. Specifically, the mutating admission webhook injects (in a very non-intrusive way) an executable into containers of Deployments and StatefulSets. This executable can request secrets from Vault through special environment variable definitions.

Kubernetes API requests

An important and unique aspect of the webhook is that it is a daemonless solution (although if you need it, you can deploy the webhook in daemon mode as well).

Why is this more secure than using Kubernetes secrets or any other custom sidecar container?

Our solution is particularly lightweight and uses only existing Kubernetes constructs like annotations and environment variables. No confidential data ever persists on the disk or in etcd - not even temporarily. All secrets are stored in memory, and are only visible to the process that requested them. Additionally, there is no persistent connection with Vault, and any Vault token used to read environment variables is flushed from memory before the application starts, in order to minimize attack surface.

If you want to make this solution even more robust, you can disable kubectl exec-ing in running containers. If you do so, no one will be able to hijack injected environment variables from a process.

The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.

        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-key-secret
              key: AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              configMapKeyRef:
                name: aws-key-configmap
                key: AWS_SECRET_ACCESS_KEY

The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:

        envFrom:
          - secretRef:
              name: aws-key-secret
# or
          - configMapRef:
              name: aws-key-configmap

Secret and ConfigMap examples

Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them. The secret value should contain a base64 encoded template string referencing the vault path you want to insert. Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.

apiVersion: v1
kind: Secret
metadata:
  name: aws-key-secret
data:
  AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ==
type: Opaque
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-key-configmap
data:
  AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY

For further examples and use cases, see Configuration examples and scenarios.

5.1 - Deploy the webhook

Deploy the mutating webhook

You can deploy the Vault Secrets Webhook using Helm. Note that:

  • The Helm chart of the vault-secrets-webhook contains the templates of the required permissions as well.
  • The deployed RBAC objects contain the necessary permissions fo running the webhook.

Prerequisites

  • The user you use for deploying the chart to the Kubernetes cluster must have cluster-admin privileges.
  • The chart requires Helm 3.
  • To interact with Vault (for example, for testing), the vault command line client must be installed on your computer.
  • You have deployed Vault with the operator and configured your Vault client to access it, as described in Deploy a local Vault operator.

Deploy the webhook

  1. Create a namespace for the webhook and add a label to the namespace, for example, vault-infra:

    kubectl create namespace vault-infra
    kubectl label namespace vault-infra name=vault-infra
    
  2. Deploy the vault-secrets-webhook chart:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook
    

    For further details, see the webhook’s Helm chart repository.

  3. Check that the pods are running:

    kubectl get pods --namespace vault-infra
    NAME                                     READY   STATUS    RESTARTS   AGE
    vault-secrets-webhook-58b97c8d6d-qfx8c   1/1     Running   0          22s
    vault-secrets-webhook-58b97c8d6d-rthgd   1/1     Running   0          22s
    
  4. Write a secret into Vault (the Vault CLI must be installed on your computer):

    vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t
    

    Expected output:

    Key              Value
    ---              -----
    created_time     2020-11-04T11:39:01.863988395Z
    deletion_time    n/a
    destroyed        false
    version          1
    
  5. Apply the following deployment to your cluster. The webhook will mutate this deployment because it has an environment variable having a value which is a reference to a path in Vault:

  6. Check the mutated deployment.

    kubectl describe deployment vault-test
    

    The output should look similar to the following:

    Name:                   vault-test
    Namespace:              default
    CreationTimestamp:      Wed, 04 Nov 2020 12:44:18 +0100
    Labels:                 <none>
    Annotations:            deployment.kubernetes.io/revision: 1
    Selector:               app.kubernetes.io/name=vault
    Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
    StrategyType:           RollingUpdate
    MinReadySeconds:        0
    RollingUpdateStrategy:  25% max unavailable, 25% max surge
    Pod Template:
      Labels:           app.kubernetes.io/name=vault
      Annotations:      vault.security.banzaicloud.io/vault-addr: https://vault:8200
                        vault.security.banzaicloud.io/vault-agent: false
                        vault.security.banzaicloud.io/vault-path: kubernetes
                        vault.security.banzaicloud.io/vault-role: default
                        vault.security.banzaicloud.io/vault-skip-verify: false
                        vault.security.banzaicloud.io/vault-tls-secret: vault-tls
      Service Account:  default
      Containers:
       alpine:
        Image:      alpine
        Port:       <none>
        Host Port:  <none>
        Command:
          sh
          -c
          echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000
        Environment:
          AWS_SECRET_ACCESS_KEY:  vault:secret/data/demosecret/aws#AWS_SECRET_ACCESS_KEY
        Mounts:                   <none>
      Volumes:                    <none>
    Conditions:
      Type           Status  Reason
      ----           ------  ------
      Available      True    MinimumReplicasAvailable
      Progressing    True    NewReplicaSetAvailable
    OldReplicaSets:  <none>
    NewReplicaSet:   vault-test-55c569f9 (1/1 replicas created)
    Events:
      Type    Reason             Age   From                   Message
      ----    ------             ----  ----                   -------
      Normal  ScalingReplicaSet  29s   deployment-controller  Scaled up replica set vault-test-55c569f9 to 1
    

    As you can see, the original environment variables in the definition are unchanged, and the sensitive value of the AWS_SECRET_ACCESS_KEY variable is only visible within the alpine container.

Deploy the webhook from a private registry

If you are getting the x509: certificate signed by unknown authority app=vault-secrets-webhook error when the webhook is trying to download the manifest from a private image registry, you can:

  • Build a docker image where the CA store of the OS layer of the image contains the CA certificate of the registry.
  • Alternatively, you can disable certificate verification for the registry by using the REGISTRY_SKIP_VERIFY=“true” environment variable in the deployment of the webhook.

Deploy in daemon mode

vault-env by default replaces itself with the original process of the Pod after reading the secrets from Vault, but with the vault.security.banzaicloud.io/vault-env-daemon: "true" annotation this behavior can be changed. So vault-env can change to daemon mode, so vault-env starts the original process as a child process and remains in memory, and renews the lease of the requested Vault token and of the dynamic secrets (if requested any) until their final expiration time.

You can find a full example using MySQL dynamic secrets in the Bank-Vaults project repository:

# Deploy MySQL first as the Vault storage backend and our application will request dynamic secrets for this database as well:
helm upgrade --install mysql stable/mysql --set mysqlRootPassword=your-root-password --set mysqlDatabase=vault --set mysqlUser=vault --set mysqlPassword=secret --set 'initializationFiles.app-db\.sql=CREATE DATABASE IF NOT EXISTS app;'

# Deploy the vault-operator and the vault-secerts-webhook
kubectl create namespace vault-infra
kubectl label namespace vault-infra name=vault-infra
helm upgrade --namespace vault-infra --install vault-operator banzaicloud-stable/vault-operator
helm upgrade --namespace vault-infra --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook

# Create a Vault instance with MySQL storage and a configured dynamic database secrets backend
kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-mysql-ha.yaml

# Deploy the example application requesting dynamic database credentials from the above Vault instance
kubectl apply -f deploy/test-dynamic-env-vars.yaml
kubectl logs -f deployment/hello-secrets

5.2 - Configuration examples and scenarios

The following examples show you how to configure the mutating webhook to best suit your environment.

The webhook checks if a container has environment variables defined in the following formats, and reads the values for those variables directly from Vault during startup time.

        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-key-secret
              key: AWS_SECRET_ACCESS_KEY
# or
        - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              configMapKeyRef:
                name: aws-key-configmap
                key: AWS_SECRET_ACCESS_KEY

The webhook checks if a container has envFrom and parses the defined ConfigMaps and Secrets:

        envFrom:
          - secretRef:
              name: aws-key-secret
# or
          - configMapRef:
              name: aws-key-configmap

Secret and ConfigMap examples

Secrets require their payload to be base64 encoded, the API rejects manifests with plaintext in them. The secret value should contain a base64 encoded template string referencing the vault path you want to insert. Run echo -n "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY" | base64 to get the correct string.

apiVersion: v1
kind: Secret
metadata:
  name: aws-key-secret
data:
  AWS_SECRET_ACCESS_KEY: dmF1bHQ6c2VjcmV0L2RhdGEvYWNjb3VudHMvYXdzI0FXU19TRUNSRVRfQUNDRVNTX0tFWQ==
type: Opaque
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-key-configmap
data:
  AWS_SECRET_ACCESSKEY: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY

Prerequisites for inline injection to work

Vault needs to be properly configured for mutation to function; namely externalConfig.auth and externConfig.roles (from the perspective of the vault operator CR) need to be properly configured. If you’re not using the vault operator then you must make sure that your Vault configuration for Kubernetes auth methods are properly configured. This configuration is outside the scope of this document. If you use the operator for managing Vault in your cluster, see the Vault operator documentation.

Inject secret into resources

The webhook can inject into any kind of resources, even into CRDs, for example:

apiVersion: mysql.example.github.com/v1
kind: MySQLCluster
metadata:
  name: "my-cluster"
spec:
  caBundle: "vault:pki/cert/43138323834372136778363829719919055910246657114#ca"

Inline mutation

The webhook also supports inline mutation when your secret needs to be replaced somewhere inside a string.

apiVersion: v1
kind: Secret
metadata:
  name: aws-key-secret
data:
  config.yaml: >
foo: bar
secret: ${vault:secret/data/mysecret#supersecret}
type: Opaque

This works also for ConfigMap resources when configMapMutation: true is set in the webhook’s Helm chart.

You can specify the version of the injected Vault secret as well in the special reference, the format is: vault:PATH#KEY_OR_TEMPLATE#VERSION

Example:

        env:
        - name: AWS_SECRET_ACCESS_KEY
          value: vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY#2

Define multiple inline-secrets in resources

You can also inject multiple secrets under the same key in a Secret/ConfigMap/Object. This means that you can use multiple Vault paths in a value, for example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sample-configmap
  annotations:
    vault.security.banzaicloud.io/vault-addr: "https://vault.default:8200"
    vault.security.banzaicloud.io/vault-role: "default"
    vault.security.banzaicloud.io/vault-tls-secret: vault-tls
    vault.security.banzaicloud.io/vault-path: "kubernetes"
data:
  aws-access-key-id: "vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID"
  aws-access-template: "vault:secret/data/accounts/aws#AWS key in base64: ${.AWS_ACCESS_KEY_ID | b64enc}"
  aws-access-inline: "AWS_ACCESS_KEY_ID: ${vault:secret/data/accounts/aws#AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY: ${vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY}"

This example also shows how a CA certificate (created by the operator) can be used with the vault.security.banzaicloud.io/vault-tls-secret: vault-tls annotation to validate the TLS connection in case of a non-Pod resource.

Request a Vault token

There is a special vault:login reference format to request a working Vault token into an environment variable to be later consumed by your application:

        env:
        - name: VAULT_TOKEN
          value: vault:login

Read a value from Vault

Values starting with "vault:" issue a read (HTTP GET) request towards the Vault API, this can be also used to request a dynamic database username/password pair for MySQL:

NOTE: This feature takes advantage of secret caching since we need to access the my-role endpoint twice, but in the background, it is written only once in Vault:

    env:
    - name: MYSQL_USERNAME
      value: "vault:database/creds/my-role#username"
    - name: MYSQL_PASSWORD
      value: "vault:database/creds/my-role#password"
    - name: REDIS_URI
      value: "redis://${vault:database/creds/my-role#username}:${vault:database/creds/my-role#password}@127.0.0.1:6739"

Write a value into Vault

Values starting with ">>vault:" issue a write (HTTP POST/PUT) request towards the Vault API, some secret engine APIs should be written instead of reading from like the Password Generator for HashiCorp Vault:

    env:
    - name: MY_SECRET_PASSWORD
      value: ">>vault:gen/password#value"

Or with Transit Secret Engine which is a fairly complex example since we are using templates when rendering the response and send data in the write request as well, the format is: vault:PATH#KEY_OR_TEMPLATE#DATA

Example:

    env:
    - name: MY_SECRET_PASSWORD
      value: ">>vault:transit/decrypt/mykey#${.plaintext | b64dec}#{"ciphertext":"vault:v1:/DupSiSbX/ATkGmKAmhqD0tvukByrx6gmps7dVI="}"

Templating in values

Templating is also supported on the secret sourced from Vault (in the key part, after the first #), in the very same fashion as in the Vault configuration and external configuration with all the Sprig functions (this is supported only for Pods right now):

    env:
    - name: DOCKER_USERNAME
      value: "vault:secret/data/accounts/dockerhub#My username on DockerHub is: ${title .DOCKER_USERNAME}"

In this case, an init-container will be injected into the given Pod. This container copies the vault-env binary into an in-memory volume and mounts that Volume to every container which has an environment variable definition like that. It also changes the command of the container to run vault-env instead of your application directly. When vault-env starts up, it connects to Vault to checks the environment variables. (By default, vault-env uses the Kubernetes Auth method, but you can also configure other authentication methods for the webhook.) The variables that have a reference to a value stored in Vault (vault:secret/....) are replaced with that value read from the Secret backend. After this, vault-env immediately executes (with syscall.Exec()) your process with the given arguments, replacing itself with that process (in non-daemon mode).

With this solution none of your Secrets stored in Vault will ever land in Kubernetes Secrets, thus in etcd.

vault-env was designed to work in Kubernetes in the first place, but nothing stops you to use it outside Kubernetes as well. It can be configured with the standard Vault client’s environment variables (because there is a standard Go Vault client underneath).

Currently, the Kubernetes Service Account-based Vault authentication mechanism is used by vault-env, so it requests a Vault token based on the Service Account of the container it is injected into.

Kubernetes 1.12 introduced a feature called APIServer dry-run which became beta as of 1.13. This feature requires some changes in webhooks with side effects. Vault mutating admission webhook is dry-run aware.

Mutate data from Vault and replace it in Kubernetes Secret

You can mutate Secrets (and ConfigMaps) as well if you set annotations and define proper Vault path in the data section:

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
  annotations:
    vault.security.banzaicloud.io/vault-addr: "https://vault.default.svc.cluster.local:8200"
    vault.security.banzaicloud.io/vault-role: "default"
    vault.security.banzaicloud.io/vault-skip-verify: "true"
    vault.security.banzaicloud.io/vault-path: "kubernetes"
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJodHRwczovL2RvY2tlci5pbyI6eyJ1c2VybmFtZSI6InZhdWx0OnNlY3JldC9kYXRhL2RvY2tlcnJlcG8vI0RPQ0tFUl9SRVBPX1VTRVIiLCJwYXNzd29yZCI6InZhdWx0OnNlY3JldC9kYXRhL2RvY2tlcnJlcG8vI0RPQ0tFUl9SRVBPX1BBU1NXT1JEIiwiYXV0aCI6ImRtRjFiSFE2YzJWamNtVjBMMlJoZEdFdlpHOWphMlZ5Y21Wd2J5OGpSRTlEUzBWU1gxSkZVRTlmVlZORlVqcDJZWFZzZERwelpXTnlaWFF2WkdGMFlTOWtiMk5yWlhKeVpYQnZMeU5FVDBOTFJWSmZVa1ZRVDE5UVFWTlRWMDlTUkE9PSJ9fX0=

In the example above the secret type is kubernetes.io/dockerconfigjson and the webhook can get credentials from Vault. The base64 encoded data contain vault path in case of username and password for docker repository and you can create it with commands:

kubectl create secret docker-registry dockerhub --docker-username="vault:secret/data/dockerrepo#DOCKER_REPO_USER" --docker-password="vault:secret/data/dockerrepo#DOCKER_REPO_PASSWORD"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-addr="https://vault.default.svc.cluster.local:8200"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-role="default"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-skip-verify="true"
kubectl annotate secret dockerhub vault.security.banzaicloud.io/vault-path="kubernetes"

Use charts without explicit container.command and container.args

The Webhook can determine the container’s ENTRYPOINT and CMD with the help of image metadata queried from the image registry. This data is cached until the webhook Pod is restarted. If the registry is publicly accessible (without authentication) you don’t need to do anything, but if the registry requires authentication the credentials have to be available in the Pod’s imagePullSecrets section.

Some examples (apply cr.yaml from the operator samples first):

helm upgrade --install mysql stable/mysql \
  --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD \
  --set mysqlPassword=vault:secret/data/mysql#MYSQL_PASSWORD \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-addr"=https://vault:8200 \
  --set "podAnnotations.vault\.security\.banzaicloud\.io/vault-tls-secret"=vault-tls

Registry access

You can also specify a default secret being used by the webhook for cases where a pod has no imagePullSecrets specified. To make this work you have to set the environment variables DEFAULT_IMAGE_PULL_SECRET and DEFAULT_IMAGE_PULL_SECRET_NAMESPACE when deploying the vault-secrets-webhook. Have a look at the values.yaml of the vault-secrets-webhook helm chart to see how this is done.

Note:

  • If your EC2 nodes have the ECR instance role, the webhook can request an ECR access token through that role automatically, instead of an explicit imagePullSecret
  • If your workload is running on GCP nodes, the webhook automatically authenticates to GCR.

Future improvements:

  • on Azure/Alibaba get a credential dynamically with the specific SDK (for AWS ECR and GCP GCR this is already done)

Using a private image repository

# Docker Hub

kubectl create secret docker-registry dockerhub --docker-username=${DOCKER_USERNAME} --docker-password=$DOCKER_PASSWORD

helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=dockerhub" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="private-repo/mysql"

# GCR

kubectl create secret docker-registry gcr \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat ~/json-key-file.json)"

helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=gcr" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="gcr.io/your-repo/mysql"

# ECR

TOKEN=`aws ecr --region=eu-west-1 get-authorization-token --output text --query authorizationData[].authorizationToken | base64 --decode | cut -d: -f2`

kubectl create secret docker-registry ecr \
 --docker-server=https://171832738826.dkr.ecr.eu-west-1.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}"

 helm upgrade --install mysql stable/mysql --set mysqlRootPassword=vault:secret/data/mysql#MYSQL_ROOT_PASSWORD --set "imagePullSecrets[0].name=ecr" --set-string "podAnnotations.vault\.security\.banzaicloud\.io/vault-skip-verify=true" --set image="171832738826.dkr.ecr.eu-west-1.amazonaws.com/mysql" --set-string imageTag=5.7

Mount all keys from Vault secret to env

This feature is very similar to Kubernetes’ standard envFrom: construct, but instead of a Kubernetes Secret/ConfigMap, all its keys are mounted from a Vault secret using the webhook and vault-env.

You can set the Vault secret to mount using the vault.security.banzaicloud.io/vault-env-from-path annotation.

Compared to the original environment variable definition in the Pod env construct, the only difference is that you won’t see the actual environment variables in the definition, because they are dynamic, and are based on the contents of the Vault secret’s, just like envFrom:.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-secrets
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-secrets
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-secrets
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
        vault.security.banzaicloud.io/vault-tls-secret: vault-tls
        vault.security.banzaicloud.io/vault-env-from-path: "secret/data/accounts/aws"
    spec:
      initContainers:
      - name: init-ubuntu
        image: ubuntu
        command: ["sh", "-c", "echo AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID && echo initContainers ready"]
      containers:
      - name: alpine
        image: alpine
        command: ["sh", "-c", "echo AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]

Authenticate the webhook to Vault

By default, the webhook uses Kubernetes ServiceAccount-based authentication in Vault. Use the vault.security.banzaicloud.io/vault-auth-method annotation to request different authentication types from the following supported types: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure”.

Note: GCP IAM authentication (gcp-iam) only allows for authentication with the ‘default’ service account of the caller, and a new token is generated at every request. Note: Azure MSI authentication (azure) a new token is generated at every request.

The following deployment - if running on a GCP instance - will automatically receive a signed JWT token from the metadata server of the cloud provider, and use it to authenticate against Vault. The same goes for vault-auth-method: "aws-ec2", when running on an EC2 node with the right instance-role.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-env-gcp-auth
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: vault-env-gcp-auth
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault-env-gcp-auth
      annotations:
        # These annotations enable Vault GCP GCE auth, see:
        # https://www.vaultproject.io/docs/auth/gcp#gce-login
        vault.security.banzaicloud.io/vault-addr: "https://vault:8200"
        vault.security.banzaicloud.io/vault-tls-secret: vault-tls
        vault.security.banzaicloud.io/vault-role: "my-role"
        vault.security.banzaicloud.io/vault-path: "gcp"
        vault.security.banzaicloud.io/vault-auth-method: "gcp-gce"
    spec:
      containers:
        - name: alpine
          image: alpine
          command:
            - "sh"
            - "-c"
            - "echo $MYSQL_PASSWORD && echo going to sleep... && sleep 10000"
          env:
            - name: MYSQL_PASSWORD
              value: vault:secret/data/mysql#MYSQL_PASSWORD

5.3 - Annotations

The mutating webhook adds the following PodSpec, Secret, ConfigMap, and CRD annotations.

AnnotationdefaultExplanation
vault.security.banzaicloud.io/vault-addr"https://vault:8200"Same as VAULT_ADDR
vault.security.banzaicloud.io/vault-image"vault:latest"Vault agent image
vault.security.banzaicloud.io/vault-image-pull-policyIfNotPresentthe Pull policy for the vault agent container
vault.security.banzaicloud.io/vault-role""The Vault role for Vault agent to use, for Pods it is the name of the ServiceAccount if not specified
vault.security.banzaicloud.io/vault-path"kubernetes"The mount path of the auth method
vault.security.banzaicloud.io/vault-skip-verify"false"Same as VAULT_SKIP_VERIFY
vault.security.banzaicloud.io/vault-tls-secret""Name of the Kubernetes Secret holding the CA certificate for Vault
vault.security.banzaicloud.io/vault-ignore-missing-secrets"false"When enabled will only log warnings when Vault secrets are missing
vault.security.banzaicloud.io/vault-env-passthrough""Comma separated list of VAULT_* related environment variables to pass through to vault-env to the main process. E.g. VAULT_ADDR,VAULT_ROLE.
vault.security.banzaicloud.io/vault-env-daemon"false"Run vault-env as a daemon instead of replacing itself with the main process. For details, see /docs/mutating-webhook/deploy/#daemon-mode.
vault.security.banzaicloud.io/vault-env-image"banzaicloud/vault-env:latest"vault-env image
vault.security.banzaicloud.io/vault-env-image-pull-policyIfNotPresentthe Pull policy for the vault-env container
vault.security.banzaicloud.io/enable-json-log"false"Log in JSON format in vault-env
vault.security.banzaicloud.io/mutate""Defines the mutation of the given resource, possible values: "skip" which prevents it.
vault.security.banzaicloud.io/mutate-probes"false"Mutate the ENV passed to a liveness or readiness probe.
vault.security.banzaicloud.io/vault-env-from-path""Comma-delimited list of vault paths to pull in all secrets as environment variables
vault.security.banzaicloud.io/token-auth-mount""{volume:file} to be injected as .vault-token.
vault.security.banzaicloud.io/vault-auth-method"jwt"The Vault authentication method to be used, one of ["kubernetes", "aws-ec2", "aws-iam", "gcp-gce", "gcp-iam", "jwt", "azure", "namespaced"]
vault.security.banzaicloud.io/vault-serviceaccount""The ServiceAccount in the objects namespace to use, useful for non-pod resources
vault.security.banzaicloud.io/vault-namespace""The Vault Namespace secrets will be pulled from. This annotation sets the VAULT_NAMESPACE environment variable. More information on namespaces within Vault can be found here
vault.security.banzaicloud.io/run-as-non-root"false"When enabled will add runAsNonRoot: true to the securityContext of all injected containers
vault.security.banzaicloud.io/run-as-user"0"Set the UID (runAsUser) for all injected containers. The default value of "0" means that no modifications will be made to the securityContext of injected containers.
vault.security.banzaicloud.io/run-as-group"0"Set the GID (runAsGroup) for all injected containers. The default value of "0" means that no modifications will be made to the securityContext of injected containers.
vault.security.banzaicloud.io/readonly-root-fs"false"When enabled will add readOnlyRootFilesystem: true to the securityContext of all injected containers

5.4 - Using Vault Agent Templating in the mutating webhook

With Bank-Vaults you can use Vault Agent to handle secrets that expire, and supply them to applications that read their configurations from a file.

When to use vault-agent

  • You have an application or tool that requires to read its configuration from a file.
  • You wish to have secrets that have a TTL and expire.
  • You have no issues with running your application with a sidecar.

Note: If you need to revoke tokens, or use additional secret backends, see Using consul-template in the mutating webhook.

Workflow

  • Your pod starts up, the webhook will inject one container into the pods lifecycle.
  • The sidecar container is running Vault, using the vault agent that accesses Vault using the configuration specified inside a configmap and writes a configuration file based on a pre configured template (written inside the same configmap) onto a temporary file system which your application can use.

Prerequisites

This document assumes the following.

  • You have a working Kubernetes cluster which has:

  • You have a working knowledge of Kubernetes.

  • You can apply Deployments or PodSpec’s to the cluster.

  • You can change the configuration of the mutating webhook.

Use Vault TTLs

If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Vault Agent to execute a command when it writes a new configuration file using the command attribute. The following is a basic example which uses the Kubernetes authentication method.

Configuration

To configure the webhook, you can either:

Enable vault agent in the webhook

For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-agent-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Vault Agent.

Defaults via environment variables

VariabledefaultExplanation
VAULT_IMAGEvault:latestthe vault image to use for the sidecar container
VAULT_IMAGE_PULL_POLICYIfNotPresentThe pull policy for the vault agent container
VAULT_ADDRhttps://127.0.0.1:8200Kubernetes service Vault endpoint URL
VAULT_TLS_SECRET""supply a secret with the vault TLS CA so TLS can be verified
VAULT_AGENT_SHARE_PROCESS_NAMESPACEKubernetes version <1.12 default off, 1.12 or higher default onShareProcessNamespace override

PodSpec annotations

AnnotationdefaultExplanation
vault.security.banzaicloud.io/vault-addrSame as VAULT_ADDR above
vault.security.banzaicloud.io/vault-tls-secretSame as VAULT_TLS_SECRET above
vault.security.banzaicloud.io/vault-agent-configmap""A configmap name which holds the vault agent configuration
vault.security.banzaicloud.io/vault-agent-oncefalsedo not run vault-agent in daemon mode, useful for kubernetes jobs
vault.security.banzaicloud.io/vault-agent-share-process-namespaceSame as VAULT_AGENT_SHARE_PROCESS_NAMESPACE above
vault.security.banzaicloud.io/vault-agent-cpu“100m”Specify the vault-agent container CPU resource limit
vault.security.banzaicloud.io/vault-agent-memory“128Mi”Specify the vault-agent container memory resource limit
vault.security.banzaicloud.io/vault-configfile-path“/vault/secrets”Mount path of Vault Agent rendered files

5.5 - Using consul-template in the mutating webhook

With Bank-Vaults you can use Consul Template as an addition to vault-env to handle secrets that expire, and supply them to applications that read their configurations from a file.

When to use consul-template

  • You have an application or tool that must read its configuration from a file.
  • You wish to have secrets that have a TTL and expire.
  • You do not wish to be limited on which vault secrets backend you use.
  • You can also expire tokens/revoke tokens (to do this you need to have a ready/live probe that can send a HUP to consul-template when the current details fail).

Workflow

The following shows the general workflow for using Consul Template:

  1. Your pod starts up. The webhook injects an init container (running vault agent) and a sidecar container (running consul-template) into the pods lifecycle.
  2. The vault agent in the init container logs in to Vault and retrieves a Vault token based on the configured VAULT_ROLE and Kubernetes Service Account.
  3. The consul-template running in the sidecar container logs in to Vault using the Vault token and writes a configuration file based on a pre-configured template in a configmap onto a temporary file system which your application can use.

Prerequisites

This document assumes the following.

  • You have a working Kubernetes cluster which has:

  • You have a working knowledge of Kubernetes.

  • You can apply Deployments or PodSpec’s to the cluster.

  • You can change the configuration of the mutating webhook.

Use Vault TTLs

If you wish to use Vault TTLs, you need a way to HUP your application on configuration file change. You can configure the Consul Template to execute a command when it writes a new configuration file using the command attribute. The following is a basic example (adapted from here).

Configuration

To configure the webhook, you can either:

Enable Consul Template in the webhook

For the webhook to detect that it will need to mutate or change a PodSpec, add the vault.security.banzaicloud.io/vault-ct-configmap annotation to the Deployment or PodSpec you want to mutate, otherwise it will be ignored for configuration with Consul Template.

Defaults via environment variables

VariabledefaultExplanation
VAULT_IMAGEvault:latestthe vault image to use for the init container
VAULT_ENV_IMAGEbanzaicloud/vault-env:latestthe vault-env image to use
VAULT_CT_IMAGEhashicorp/consul-template:latestthe consul template image to use
VAULT_ADDRhttps://127.0.0.1:8200Kubernetes service Vault endpoint URL
VAULT_SKIP_VERIFY“false”should vault agent and consul template skip verifying TLS
VAULT_TLS_SECRET""supply a secret with the vault TLS CA so TLS can be verified
VAULT_AGENT“true”enable the vault agent
VAULT_CT_SHARE_PROCESS_NAMESPACEKubernetes version <1.12 default off, 1.12 or higher default onShareProcessNamespace override

PodSpec annotations

AnnotationdefaultExplanation
vault.security.banzaicloud.io/vault-addrSame as VAULT_ADDR above
vault.security.banzaicloud.io/vault-roledefaultThe Vault role for Vault agent to use
vault.security.banzaicloud.io/vault-pathauth/<method type>The mount path of the method
vault.security.banzaicloud.io/vault-skip-verifySame as VAULT_SKIP_VERIFY above
vault.security.banzaicloud.io/vault-tls-secretSame as VAULT_TLS_SECRET above
vault.security.banzaicloud.io/vault-agentSame as VAULT_AGENT above
vault.security.banzaicloud.io/vault-ct-configmap""A configmap name which holds the consul template configuration
vault.security.banzaicloud.io/vault-ct-image""Specify a custom image for consul template
vault.security.banzaicloud.io/vault-ct-oncefalsedo not run consul-template in daemon mode, useful for kubernetes jobs
vault.security.banzaicloud.io/vault-ct-pull-policyIfNotPresentthe Pull policy for the consul template container
vault.security.banzaicloud.io/vault-ct-share-process-namespaceSame as VAULT_CT_SHARE_PROCESS_NAMESPACE above
vault.security.banzaicloud.io/vault-ct-cpu“100m”Specify the consul-template container CPU resource limit
vault.security.banzaicloud.io/vault-ct-memory“128Mi”Specify the consul-template container memory resource limit
vault.security.banzaicloud.io/vault-ignore-missing-secrets“false”When enabled will only log warnings when Vault secrets are missing
vault.security.banzaicloud.io/vault-env-passthrough""Comma seprated list of VAULT_* related environment variables to pass through to main process. E.g.VAULT_ADDR,VAULT_ROLE.
vault.security.banzaicloud.io/vault-ct-secrets-mount-path“/vault/secret”Mount path of Consul template rendered files

5.6 - Transit Encryption

The transit secrets engine handles cryptographic functions on data in-transit, mainly to encrypt data from applications while still storing that encrypted data in some primary data store. Vault doesn’t store the data sent to the secrets engine, it can also be viewed as “cryptography as a service” or “encryption as a service”. For details about transit encryption, see the official documentation.

Note:

Enable Transit secrets engine

To enable and test the Transit secrets engine, complete the following steps.

  1. Enable the Transit secrets engine:

    vault secrets enable transit
    
  2. Create a named encryption key:

    vault write -f transit/keys/my-key
    
  3. Encrypt data with encryption key:

    vault write transit/encrypt/my-key plaintext=$(base64 <<< "my secret data")
    
  4. After completing the previous steps, the webhook will mutate pods that have at least one environment variable with a value which is encrypted by Vault. For example (in the last line of the example):

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: vault-test
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: vault
      template:
        metadata:
          labels:
            app.kubernetes.io/name: vault
          annotations:
            vault.security.banzaicloud.io/vault-addr: "https://vault:8200" # optional, the address of the Vault service, default values is https://vault:8200
            vault.security.banzaicloud.io/vault-role: "default" # optional, the default value is the name of the ServiceAccount the Pod runs in, in case of Secrets and ConfigMaps it is "default"
            vault.security.banzaicloud.io/vault-skip-verify: "false" # optional, skip TLS verification of the Vault server certificate
            vault.security.banzaicloud.io/vault-tls-secret: "vault-tls" # optinal, the name of the Secret where the Vault CA cert is, if not defined it is not mounted
            vault.security.banzaicloud.io/vault-agent: "false" # optional, if true, a Vault Agent will be started to do Vault authentication, by default not needed and vault-env will do Kubernetes Service Account based Vault authentication
            vault.security.banzaicloud.io/vault-path: "kubernetes" # optional, the Kubernetes Auth mount path in Vault the default value is "kubernetes"
            vault.security.banzaicloud.io/transit-key-id: "my-key" # required if encrypted data was found; transit key id that created before
        spec:
          serviceAccountName: default
          containers:
          - name: alpine
            image: alpine
            command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
            env:
            - name: AWS_SECRET_ACCESS_KEY
              # Value based on encrypted key that stored in Vault, so value from this example
              # not the same as you can get after `encrypt`
              value: vault:v1:8SDd3WHDOjf7mq69CyCqYjBXAiQQAVZRkFM13ok481zoCmHnSeDX9vyf7w==
    

5.7 - Monitoring the Webhook with Grafana and Prometheus

To monitor the webhook with Prometheus and Grafana, complete the following steps.

Prerequisites

  • An already deployed and configured mutating webhook. For details, see Mutating Webhook.

Steps

  1. Install the Prometheus Operator Bundle:

    kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
    
  2. Install the webhook with monitoring and Prometheus Operator ServiceMonitor enabled:

    helm upgrade --wait --install vault-secrets-webhook \
        banzaicloud-stable/vault-secrets-webhook \
        --namespace vault-infra \
        --set metrics.enabled=true \
        --set metrics.serviceMonitor.enabled={}
    
  3. Create a Prometheus instance which monitors the components of Bank-Vaults:

    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/hack/prometheus.yaml
    
  4. Create a Grafana instance and expose it:

    kubectl create deployment grafana --image grafana/grafana
    kubectl expose deployment grafana --port 3000 --type LoadBalancer
    
  5. Fetch the external IP address of the Grafana instance, and open it in your browser on port 3000.

    kubectl get service grafana
    
  6. Create a Prometheus Data Source in this Grafana instance which grabs data from http://prometheus-operated:9090/.

  7. Import the Kubewebhook admission webhook dashboard to Grafana (created by Xabier Larrakoetxea).

  8. Select the previously created Data Source to feed this dashboard.

5.8 - Injecting consul-template into the Prometheus operator for Vault metrics

To get vault metrics into Prometheus you need to log in to Vault to get access to a native Vault endpoint that provides the metrics.

Workflow

  1. The webhook injects vault-agent as an init container, based on the Kubernetes Auth role configuration prometheus-operator-prometheus.
  2. The vault-agent grabs a token with the policy of prometheus-operator-prometheus.
  3. consul-template runs as a sidecar, and uses the token from the previous step to retrieve a new token using the Token Auth role prometheus-metrics which has the policy prometheus-metrics applied to it.
  4. Prometheus can now use this second token to read the Vault Prometheus endpoint.

The trick here is that Prometheus runs with the SecurityContext UID of 1000 but the default consul-template image is running under the UID of 100. This is because of a Dockerfile Volume that is configured which dockerd mounts as 100 (/consul-template/data).

Subsequently using this consul-template means it will never start, so we need to ensure we do not use this declared volume and change the UID using a custom Dockerfile and entrypoint.

Prerequisites

This document assumes you have a working Kubernetes cluster which has a:

  • You have a working Kubernetes cluster which has:

  • You have the CoreOS Prometheus Operator installed and working.

  • You have a working knowledge of Kubernetes.

  • You can apply Deployments or PodSpec’s to the cluster.

  • You can change the configuration of the mutating webhook.

Configuration

Custom consul-template image; docker-entrypoint.sh

#!/bin/dumb-init /bin/sh
set -ex

# Note above that we run dumb-init as PID 1 in order to reap zombie processes
# as well as forward signals to all processes in its session. Normally, sh
# wouldn't do either of these functions so we'd leak zombies as well as do
# unclean termination of all our sub-processes.

# CONSUL_DATA_DIR is exposed as a volume for possible persistent storage.
# CT_CONFIG_DIR isn't exposed as a volume but you can compose additional config
# files in there if you use this image as a base, or use CT_LOCAL_CONFIG below.
CT_DATA_DIR=/consul-template/config
CT_CONFIG_DIR=/consul-template/config

# You can also set the CT_LOCAL_CONFIG environment variable to pass some
# Consul Template configuration JSON without having to bind any volumes.
if [ -n "$CT_LOCAL_CONFIG" ]; then
  echo "$CT_LOCAL_CONFIG" > "$CT_CONFIG_DIR/local-config.hcl"
fi

# If the user is trying to run consul-template directly with some arguments, then
# pass them to consul-template.
if [ "${1:0:1}" = '-' ]; then
  set -- /bin/consul-template "$@"
fi

# If we are running Consul, make sure it executes as the proper user.
if [ "$1" = '/bin/consul-template' ]; then

  # Set the configuration directory
  shift
  set -- /bin/consul-template \
    -config="$CT_CONFIG_DIR" \
    "$@"

  # Check the user we are running as
  current_user="$(id -un)"
  if [ "${current_user}" == "root" ]; then
    # Run under the right user
    set -- gosu consul-template "$@"
  fi
fi

exec "$@"

Dockerfile

FROM hashicorp/consul-template:0.19.6-dev-alpine

ADD build/docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh

RUN apk --no-cache add shadow && \
    usermod -u 1000 consul-template && \
    chown -Rc consul-template:consul-template /consul-template/

USER consul-template:consul-template

ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: prometheus
    prometheus: consul-template
  name: prometheus-consul-template
data:
  config.hcl: |
    vault {
      ssl {
        ca_cert = "/vault/tls/ca.crt"
      }
      grace = "5m"
      retry {
        backoff = "1s"
      }
    }
    template {
      destination = "/vault/secrets/vault-token"
      command     = "/bin/sh -c '/usr/bin/curl -s http://127.0.0.1:9090/-/reload'"
      contents = <<-EOH
      {{with secret "/auth/token/create/prometheus-metrics" "policy=prometheus-metrics" }}{{.Auth.ClientToken}}{{ end }}
      EOH
      wait {
        min = "2s"
        max = "60s"
      }
    }    

Vault CR snippets

Set the vault image to use:

---
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
spec:
  size: 2
  image: vault:1.1.2

Our Vault config for telemetry:

  # A YAML representation of a final vault config file.
  # See https://www.vaultproject.io/docs/configuration/ for more information.
  config:
    telemetry:
      prometheus_retention_time: 30s
      disable_hostname: true

Disable statsd:

  # since we are running Vault 1.1.0 with the native Prometheus support, we do not need the statsD exporter
  statsdDisabled: true

Vault externalConfig policies:

  externalConfig:
    policies:
      - name: prometheus-operator-prometheus
        rules: |
          path "auth/token/create/prometheus-metrics" {
            capabilities = ["read", "update"]
          }          
      - name: prometheus-metrics
        rules: path "sys/metrics" {
          capabilities = ["list", "read"]
          }

auth:

    auth:
      - type: token
        roles:
          - name: prometheus-metrics
            allowed_policies:
              - prometheus-metrics
            orphan: true
      - type: kubernetes
        roles:
          - name: prometheus-operator-prometheus
            bound_service_account_names: prometheus-operator-prometheus
            bound_service_account_namespaces: mynamespace
            policies: prometheus-operator-prometheus
            ttl: 4h

Prometheus Operator Snippets

prometheusSpec

  prometheusSpec:
    # https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
    podMetadata:
      annotations:
        vault.security.banzaicloud.io/vault-ct-configmap: "prometheus-consul-template"
        vault.security.banzaicloud.io/vault-role: prometheus-operator-prometheus
        vault.security.banzaicloud.io/vault-ct-image: "mycustomimage:latest"

    secrets:
      - etcd-client-tls
      - vault-tls

Prometheus CRD ServiceMonitor

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: prometheus-operator
  name: prometheus-operator-vault
spec:
  endpoints:
    - bearerTokenFile: /vault/secrets/vault-token
      interval: 30s
      params:
        format: ['prometheus']
      path: /v1/sys/metrics
      port: api-port
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/secrets/vault-tls/ca.crt
        certFile: /etc/prometheus/secrets/vault-tls/server.crt
        keyFile: /etc/prometheus/secrets/vault-tls/server.key
        insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
      vault_cr: vault

5.9 - Comparison of Banzai Cloud and HashiCorp mutating webhook for Vault

Legend

  • ✅: Implemented
  • o: Planned/In-progress
FeatureBanzai Cloud WebhookHashiCorp Webhook
Automated Vault and K8S setup✅ (operator)
vault-agent/consul-template sidecar injection
Direct env var injection
Injecting into K8S Secrets
Injecting into K8S ConfigMaps
Injecting into K8S CRDs
Sidecar-less dynamic secrets
CSI Drivero
Native Kubernetes sidecaro

5.10 - Running the webhook and Vault on different clusters

This section describes how to configure the webhook and Vault when the webhook runs on a different cluster from Vault, or if Vault runs outside Kubernetes.

Let’s suppose you have two different K8S clusters:

  • cluster1 contains vault-operator
  • cluster2 contains vault-secrets-webhook

Basically, you have to grant cluster2 access to the Vault running on cluster1. To achieve this, complete the following steps.

  1. Extract the cluster.certificate-authority-data and the cluster.server fields from your cluster2 kubeconfig file. You will need them in the externalConfig section of the cluster1 configuration. For example:

    kubectl config view -o yaml --minify=true --raw=true
    
  2. Decode the certificate from the cluster.certificate-authority-data field, for example::

    grep 'certificate-authority-data' $HOME/.kube/config | awk '{print $2}' | base64 --decode
    
  3. On cluster2, create a vault ServiceAccount and the vault-auth-delegator ClusterRoleBinding:

    kubectl apply -f https://github.com/bank-vaults/bank-vaults/raw/main/operator/deploy/rbac.yaml
    

    You can use the vault ServiceAccount token as a token_reviewer_jwt in the auth configuration. To retrieve the token, run the following command:

    kubectl get secret $(kubectl get sa vault -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 --decode
    
  4. In the vault.banzaicloud.com custom resource (for example, https://github.com/bank-vaults/bank-vaults/raw/main/operator/deploy/cr.yaml) of cluster1, define an externalConfig section. Fill the values of the kubernetes_ca_cert, kubernetes_host, and token_reviewer_jwt using the data collected in the previous steps.

      externalConfig:
        policies:
          - name: allow_secrets
            rules: path "secret/*" {
              capabilities = ["create", "read", "update", "delete", "list"]
              }
        auth:
          - type: kubernetes
            config:
              token_reviewer_jwt: <token-for-cluster2-service-account>
              kubernetes_ca_cert: |
                -----BEGIN CERTIFICATE-----
                <certificate-from-certificate-authority-data-on-cluster2>
                -----END CERTIFICATE-----            
              kubernetes_host: <cluster.server-field-on-cluster2>
            roles:
              # Allow every pod in the default namespace to use the secret kv store
              - name: default
                bound_service_account_names: ["default", "vault-secrets-webhook"]
                bound_service_account_namespaces: ["default", "vswh"]
                policies: allow_secrets
                ttl: 1h
    
  5. In a production environment, it is highly recommended to specify TLS config for your Vault ingress.

      # Request an Ingress controller with the default configuration
      ingress:
        # Specify Ingress object annotations here, if TLS is enabled (which is by default)
        # the operator will add NGINX, Traefik and HAProxy Ingress compatible annotations
        # to support TLS backends
        annotations:
        # Override the default Ingress specification here
        # This follows the same format as the standard Kubernetes Ingress
        # See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.13/#ingressspec-v1beta1-extensions
        spec:
          tls:
          - hosts:
            - vault-dns-name
            secretName: vault-ingress-tls-secret
    
  6. Deploy the Vault custom resource containing the externalConfig section to cluster1:

    kubectl apply -f your-proper-vault-cr.yaml
    
  7. After Vault started in cluster1, you can use the vault-secrets-webhook in cluster2 with the proper annotations. For example:

    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: hello-secrets
      template:
        metadata:
          labels:
            app.kubernetes.io/name: hello-secrets
          annotations:
            vault.security.banzaicloud.io/vault-addr: "https://vault-dns-name:443"
            vault.security.banzaicloud.io/vault-role: "default"
            vault.security.banzaicloud.io/vault-skip-verify: "true"
            vault.security.banzaicloud.io/vault-path: "kubernetes"
    

Also you can use directly cloud identity to auth the mutating-webhook against the external vault.

  1. Add your cloud auth method in your external vault https://www.vaultproject.io/docs/auth/azure

  2. Configure your vault-secrets-webhook to use the good method. For example:

env:
  VAULT_ADDR: https://external-vault.example.com
  VAULT_AUTH_METHOD: azure
  VAULT_PATH: azure
  VAULT_ROLE: default

For VAULT_AUTH_METHOD env var, these types: “kubernetes”, “aws-ec2”, “gcp-gce”, “gcp-iam”, “jwt”, “azure” are supported.

6 - Unseal keys

Unsealing is the process of constructing the master key necessary to read the decryption key to decrypt data, allowing access to Vault. (From the official Vault documentation)

Vault starts in an uninitialized state, which means it has to be initialized with an initial set of parameters. The response to the init request is the root token and unseal keys. After that, Vault becomes initialized, but remains in a sealed state. A sealed state is a state in which no secrets can reach or leave Vault until a person, possibly more people than one, unseals it with the required number of unseal keys.

Vault data and the unseal keys live together: if you delete a Vault instance installed by the operator, or if you delete the Helm chart, all your data and the unseal keys to that initialized state should remain untouched. Read more about it in the official documentation.

The Bank-Vaults Init and Unseal process

Bank-Vaults runs in an endless loop and does the following:

Vault Unseal Flow

  1. Bank-Vaults checks if Vault is initialized. If yes, it continues to step 2, otherwise Bank-Vaults:
    1. Calls Vault init, which returns the root token and the configured number of unseal keys.
    2. Encrypts the received token and keys with the configured KMS key.
    3. Stores the encrypted token and keys in the cloud provider’s object storage.
    4. Flushes the root token and keys from its memory with explicit GC as soon as possible.
  2. Bank-Vaults checks if Vault is sealed. If it isn’t, it continues to step 3, otherwise Bank-Vaults:
    1. Reads the encrypted unseal keys from the cloud provider’s object storage.
    2. Decrypts the unseal keys with the configured KMS key.
    3. Unseals Vault with the decrypted unseal keys.
    4. Flushes the keys from its memory with explicit GC as soon as possible.
  3. If the external configuration file was changed and an OS signal is received, then Bank-Vaults:
    1. Parses the configuration file.
    2. Reads the encrypted root token from the cloud provider’s object storage.
    3. Decrypts the root token with the configured KMS key.
    4. Applies the parsed configuration on the Vault API
    5. Flushes the root token from its memory with explicit GC as soon as possible.
  4. Repeats from the second step after the configured time period.

Keys stored by Bank-Vaults

Bank-Vaults stores the following keys:

  • vault-root, which is the Vault’s root token
  • vault-unseal-N, where N is a number, starting at 0 up to the maximum defined minus 1, e.g. 5 unseal keys will be vault-unseal-0 up to including vault-unseal-4

HashiCorp recommends to revoke root tokens after the initial set up of Vault has been completed. To unseal Vault, the vault-root token is not needed and can be removed from the storage if it was put there via the --init call to bank-vaults.

Decrypting root token

AWS

To use the KMS-encrypted root token with Vault CLI:

Required CLI tools:

  • aws

Steps:

  1. Download and decrypt the root token (and the unseal keys, but that is not mandatory) into a file on your local file system:

    BUCKET=bank-vaults-0
    REGION=eu-central-1
    
    for key in "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
    do
        aws s3 cp s3://${BUCKET}/${key} .
    
        aws kms decrypt \
            --region ${REGION} \
            --ciphertext-blob fileb://${key} \
            --encryption-context Tool=bank-vaults \
            --output text \
            --query Plaintext | base64 -d > ${key}.txt
    
        rm ${key}
    done
    
  2. Save it as an environment variable:

    export VAULT_TOKEN="$(cat vault-root.txt)"
    

Google Cloud

To use the KMS-encrypted root token with vault CLI:

Required CLI tools:

  • gcloud
  • gsutil
GOOGLE_PROJECT="my-project"
GOOGLE_REGION="us-central1"
BUCKET="bank-vaults-bucket"
KEYRING="beta"
KEY="beta"

export VAULT_TOKEN=$(gsutil cat gs://${BUCKET}/vault-root | gcloud kms decrypt \
                     --project ${GOOGLE_PROJECT} \
                     --location ${GOOGLE_REGION} \
                     --keyring ${KEYRING} \
                     --key ${KEY} \
                     --ciphertext-file - \
                     --plaintext-file -)

Kubernetes

There is a Kubernetes Secret backed unseal storage in Bank-Vaults, you should be aware of that Kubernetes Secrets are base64 encoded only if you are not using a EncryptionConfiguration in your Kubernetes cluster.

VAULT_NAME="vault"

export VAULT_TOKEN=$(kubectl get secrets ${VAULT_NAME}-unseal-keys -o jsonpath={.data.vault-root} | base64 -d)

Migrate unseal keys between cloud providers

If you need to move your Vault instance from one provider or an external managed Vault, you have to store those the unseal keys and a root token in the Bank-Vaults format.

All examples assume that you have created files holding the root-token and the 5 unseal keys in plaintext:

  • vault-root.txt
  • vault-unseal-0.txt
  • vault-unseal-1.txt
  • vault-unseal-2.txt
  • vault-unseal-3.txt
  • vault-unseal-4.txt

For migrating the Vault storage data you will have to use official migration command provided by Vault.

AWS

Moving the above mentioned files to an AWS bucket and encrypt them with KMS before:

REGION=eu-central-1
KMS_KEY_ID=02a2ba49-42ce-487f-b006-34c64f4b760e
BUCKET=bank-vaults-1

for key in "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
do
    aws kms encrypt \
        --region ${REGION} --key-id ${KMS_KEY_ID} \
        --plaintext fileb://${key}.txt \
        --encryption-context Tool=bank-vaults \
        --output text \
        --query CiphertextBlob | base64 -d > ${key}

    aws s3 cp ./${key} s3://${BUCKET}/

    rm ${key} ${key}.txt
done

7 - TLS

Bank-Vaults tries to automate as much as possible for handling TLS certificates.

  • The vault-operator automates the creation and renewal of TLS certificates for Vault.
  • The vault Helm Chart automates only the creation of TLS certificates for Vault via Sprig.

Both the operator and the chart generate a Kubernetes Secret holding the TLS certificates, this is named ${VAULT_CR_NAME}-tls. For most examples in the Bank-Vaults project repository, the name of the secret is vault-tls.

The Secret data keys are:

  • ca.crt
  • server.crt
  • server.key

Note: The operator doesn’t overwrite this Secret if it already exists, so you can provide this certificate in any other way, for example using cert-manager or by simply placing it there manually.

Operator custom TLS settings

The following attributes influence the TLS settings of the operator. The ca.crt key is mandatory in existingTlsSecretName, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

CANamespaces

The list of namespaces where the generated CA certificate for Vault should be distributed. Use ["*"] for all namespaces.

Default value: []

ExistingTLSSecretName

The name of the secret that contains a TLS server certificate, key, and the corresponding CA certificate. The secret must be in the kubernetes.io/tls type secret keys + ca.crt key format. If the attribute is set, the operator uses the certificate already set in the secret, otherwise it generates a new one.

The ca.crt key is mandatory, otherwise the Bank-Vaults components can’t verify the Vault server certificate.

Default value: ""

TLSAdditionalHosts

A list hostnames or IP addresses to add to the SAN on the automatically generated TLS certificate.

Default value: []

TLSExpiryThreshold

The expiration threshold of the Vault TLS certificate in Go Duration format.

Default value: 168h

Using the generated custom TLS certificate with vault-operator

To use an existing secret which contains the TLS certificate, define existingTlsSecretName in the Vault custom resource.

Generate custom certificates with CFSSL

If you don’t want to use the certificates generated by Helm or the Bank-Vaults operator, the easiest way to create a custom certificate for Bank-Vaults is using CFSSL.

The https://github.com/bank-vaults/bank-vaults-docs/tree/master/docs/tls directory holds a set of custom CFSSL configurations which are prepared for the Helm release name vault in the default namespace. Of course, you can put any other certificates into the Secret below, this is just an example.

  1. Install CFSSL.

  2. Create a CA:

    cfssl genkey -initca csr.json | cfssljson -bare ca
    
  3. Create a server certificate:

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config.json -profile=server server.json | cfssljson -bare server
    
  4. Put these certificates (and the server key) into a Kubernetes Secret:

    kubectl create secret generic vault-tls --from-file=ca.crt=ca.pem --from-file=server.crt=server.pem --from-file=server.key=server-key.pem
    
  5. Install the Vault instance:

    • With the chart which uses this certificate:
    helm upgrade --install vault ../charts/vault --set tls.secretName=vault-tls
    
    • With the operator, create a Vault custom resource, and apply it:
    kubectl apply -f vault-cr.yaml
    

Generate custom certificates with cert-manager

You can use the following cert-manager custom resource to generate a certificate for Bank-Vaults.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-cert
spec:
  commonName: vault
  usages:
    - server auth
  dnsNames:
    - vault
    - vault.default
    - vault.default.svc
    - vault.default.svc.cluster.local
  ipAddresses:
    - 127.0.0.1
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF

8 - Cloud permissions

The operator and the bank-vaults CLI command needs certain cloud permissions to function properly (init, unseal, configuration).

Google Cloud

The Service Account in which the Pod is running has to have the following IAM Roles:

  • Cloud KMS Admin
  • Cloud KMS CryptoKey Encrypter/Decrypter
  • Storage Admin

A CLI example how to run bank-vaults based Vault configuration on Google Cloud:

bank-vaults configure --google-cloud-kms-key-ring vault --google-cloud-kms-crypto-key bank-vaults --google-cloud-kms-location global --google-cloud-storage-bucket vault-ha --google-cloud-kms-project continual-flow-276578

Azure

The Access Policy in which the Pod is running has to have the following IAM Roles:

  • Key Vault All Key permissions
  • Key Vault All Secret permissions

AWS

Enable IAM OIDC provider for an EKS cluster

To allow Vault pods to assume IAM roles in order to access AWS services the IAM OIDC provider needs to be enabled on the cluster.

BANZAI_CURRENT_CLUSTER_NAME="mycluster"

# Enable OIDC provider for the cluster with eksctl
# Follow the docs here to do it manually https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
eksctl utils associate-iam-oidc-provider \
    --cluster ${BANZAI_CURRENT_CLUSTER_NAME} \
    --approve

# Create a KMS key and S3 bucket and enter details here
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
REGION="eu-west-1"
KMS_KEY_ID="9f054126-2a98-470c-9f10-9b3b0cad94a1"
KMS_KEY_ARN="arn:aws:kms:${REGION}:${AWS_ACCOUNT_ID}:key/${KMS_KEY_ID}"
BUCKET="bank-vaults"
OIDC_PROVIDER=$(aws eks describe-cluster --name ${BANZAI_CURRENT_CLUSTER_NAME} --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")
SERVICE_ACCOUNT_NAME="vault"
SERVICE_ACCOUNT_NAMESPACE="vault"

cat > trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

cat > vault-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt"
            ],
            "Resource": [
                "${KMS_KEY_ARN}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::${BUCKET}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::${BUCKET}"
        }
    ]
}
EOF

# AWS IAM role and Kubernetes service account setup
aws iam create-role --role-name vault --assume-role-policy-document file://trust.json
aws iam create-policy --policy-name vault --policy-document file://vault-policy.json
aws iam attach-role-policy --role-name vault --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/vault

# If you are having a ServiceAccount already, only the annotation is needed
kubectl create serviceaccount $SERVICE_ACCOUNT_NAME --namespace $SERVICE_ACCOUNT_NAMESPACE
kubectl annotate serviceaccount $SERVICE_ACCOUNT_NAME  --namespace $SERVICE_ACCOUNT_NAMESPACE eks.amazonaws.com/role-arn="arn:aws:iam::${AWS_ACCOUNT_ID}:role/vault"

# Cleanup
rm vault-policy.json trust.json

Getting the root token

After Vault is successfully deployed, you can query the root-token for admin access.

# Fetch Vault root token, check bucket for actual name based on unsealConfig.aws.s3Prefix
aws s3 cp s3://$s3_bucket_name/vault-root /tmp/vault-root

export VAULT_TOKEN="$(aws kms decrypt \
  --ciphertext-blob fileb:///tmp/vault-root \
  --encryption-context Tool=bank-vaults \
  --query Plaintext --output text | base64 --decode)"

The Instance profile in which the Pod is running has to have the following IAM Policies:

  • KMS: kms:Encrypt, kms:Decrypt
  • S3: s3:GetObject, s3:PutObject on object level and s3:ListBucket on bucket level

An example command how to init and unseal Vault on AWS:

bank-vaults unseal --init --mode aws-kms-s3 --aws-kms-key-id 9f054126-2a98-470c-9f10-9b3b0cad94a1 --aws-s3-region eu-west-1 --aws-kms-region eu-west-1 --aws-s3-bucket bank-vaults

When using existing unseal keys, you need to make sure to kms encrypt these with the proper EncryptionContext. If this is not done, the invocation of bank-vaults will trigger an InvalidCiphertextException from AWS KMS. An example how to encrypt the keys (specify --profile and --region accordingly):

aws kms encrypt --key-id "alias/kms-key-alias" --encryption-context "Tool=bank-vaults"  --plaintext fileb://vault-unseal-0.txt --output text --query CiphertextBlob | base64 -D > vault-unseal-0

From this point on copy the encrypted files to the appropriate S3 bucket. As an additional security measure make sure to turn on encryption of the S3 bucket before uploading the files.

Alibaba Cloud

A CLI example how to run bank-vaults based Vault unsealing on Alibaba Cloud:

bank-vaults unseal --mode alibaba-kms-oss --alibaba-access-key-id ${ALIBABA_ACCESS_KEY_ID} --alibaba-access-key-secret ${ALIBABA_ACCESS_KEY_SECRET} --alibaba-kms-region eu-central-1 --alibaba-kms-key-id ${ALIBABA_KMS_KEY_UUID} --alibaba-oss-endpoint oss-eu-central-1.aliyuncs.com --alibaba-oss-bucket bank-vaults

Kubernetes

The Service Account in which the bank-vaults Pod is running has to have the following Roles rules:

rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs:     ["get", "create", "update"]

9 - Backing up Vault

You can configure the vault-operator to create backups of the Vault cluster with Velero.

Prerequisites

  • The Velero CLI must be installed on your computer.
  • To create Persistent Volume (PV) snapshots, you need access to an object storage. The following example uses an Amazon S3 bucket called bank-vaults-velero in the Stockholm region.

Install Velero

To configure the vault-operator to create backups of the Vault cluster, complete the following steps.

  1. Install Velero on the target cluster with Helm.

    1. Add the Velero Helm repository:

      helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
      
    2. Create a namespace for Velero:

      kubectl create namespace velero
      
    3. Install Velero with Restic so you can create PV snapshots as well:

      BUCKET=bank-vaults-velero
      REGION=eu-north-1
      KMS_KEY_ID=alias/bank-vaults-velero
      SECRET_FILE=~/.aws/credentials
      
      helm upgrade --install velero --namespace velero \
                --set configuration.provider=aws \
                --set-file credentials.secretContents.cloud=${SECRET_FILE} \
                --set deployRestic=true \
                --set configuration.backupStorageLocation.name=aws \
                --set configuration.backupStorageLocation.bucket=${BUCKET} \
                --set configuration.backupStorageLocation.config.region=${REGION} \
                --set configuration.backupStorageLocation.config.kmsKeyId=${KMS_KEY_ID} \
                --set configuration.volumeSnapshotLocation.name=aws \
                --set configuration.volumeSnapshotLocation.config.region=${REGION} \
                --set "initContainers[0].name"=velero-plugin-for-aws \
                --set "initContainers[0].image"=velero/velero-plugin-for-aws:v1.2.1 \
                --set "initContainers[0].volumeMounts[0].mountPath"=/target \
                --set "initContainers[0].volumeMounts[0].name"=plugins \
                vmware-tanzu/velero
      
  2. Install the vault-operator to the cluster:

    helm upgrade --install vault-operator banzaicloud-stable/vault-operator
    
    kubectl apply -f operator/deploy/rbac.yaml
    kubectl apply -f operator/deploy/cr-raft.yaml
    

    Note: The Vault CR in cr-raft.yaml has a special flag called veleroEnabled. This is useful for file-based Vault storage backends (file, raft), see the Velero documentation:

      # Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:
      # https://velero.io/docs/v1.2.0/hooks/
      veleroEnabled: true
    
  3. Create a backup with the Velero CLI or with the predefined Velero Backup CR:

    velero backup create --selector vault_cr=vault vault-1
    
    # OR
    
    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/examples/backup/backup.yaml
    

    Note: For a daily scheduled backup, see schedule.yaml.

  4. Check that the Velero backup got created successfully:

    velero backup describe --details vault-1
    

    Expected output:

    Name:         vault-1
    Namespace:    velero
    Labels:       velero.io/backup=vault-1
                  velero.io/pv=pvc-6eb4d9c1-25cd-4a28-8868-90fa9d51503a
                  velero.io/storage-location=default
    Annotations:  <none>
    
    Phase:  Completed
    
    Namespaces:
      Included:  *
      Excluded:  <none>
    
    Resources:
      Included:        *
      Excluded:        <none>
      Cluster-scoped:  auto
    
    Label selector:  vault_cr=vault
    
    Storage Location:  default
    
    Snapshot PVs:  auto
    
    TTL:  720h0m0s
    
    Hooks:  <none>
    
    Backup Format Version:  1
    
    Started:    2020-01-29 14:17:41 +0100 CET
    Completed:  2020-01-29 14:17:45 +0100 CET
    
    Expiration:  2020-02-28 14:17:41 +0100 CET
    

Test the backup

  1. To emulate a catastrophe, remove Vault entirely from the cluster:

    kubectl delete vault -l vault_cr=vault
    kubectl delete pvc -l vault_cr=vault
    
  2. Now restore Vault from the backup.

    1. Scale down the vault-operator, so it won’t reconcile during the restore process:

      kubectl scale deployment vault-operator --replicas 0
      
    2. Restore all Vault-related resources from the backup:

      velero restore create --from-backup vault-1
      
    3. Check that the restore has finished properly:

      velero restore get
      NAME                    BACKUP   STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
      vault1-20200129142409   vault1   Completed   0          0        2020-01-29 14:24:09 +0100 CET   <none>
      
    4. Check that the Vault cluster got actually restored:

      kubectl get pods
      NAME                                READY   STATUS    RESTARTS   AGE
      vault-0                             4/4     Running   0          1m42s
      vault-1                             4/4     Running   0          1m42s
      vault-2                             4/4     Running   0          1m42s
      vault-configurer-5499ff64cb-g75vr   1/1     Running   0          1m42s
      
    5. Scale the operator back after the restore process:

      kubectl scale deployment vault-operator --replicas 1
      
  3. Delete the backup if you don’t wish to keep it anymore:

    velero backup delete vault-1
    

10 - Running the Bank-Vaults secret webhook alongside Istio

Both the vault-operator and the vault-secrets-webhook can work on Istio enabled clusters quite well.

We support the following three scenarios:

Prerequisites

  1. Install the Istio operator.

  2. Make sure you have mTLS enabled in the Istio mesh through the operator with the following command:

    Enable mTLS if it is not set to STRICT:

    kubectl patch istio -n istio-system mesh --type=json -p='[{"op": "replace", "path": "/spec/meshPolicy/mtlsMode", "value":STRICT}]'
    
  3. Check that mesh is configured with mTLS turned on which applies to all applications in the cluster in Istio-enabled namespaces. You can change this if you would like to use another policy.

    kubectl get meshpolicy default -o yaml
    

    Expected output:

    apiVersion: authentication.istio.io/v1alpha1
    kind: MeshPolicy
    metadata:
      name: default
      labels:
        app: security
    spec:
      peers:
      - mtls: {}
    

Now your cluster is properly running on Istio with mTLS enabled globally.

Install the Bank-Vaults components

  1. You are recommended to create a separate namespace for Bank-Vaults called vault-system. You can enable Istio sidecar injection here as well, but Kubernetes won’t be able to call back the webhook properly since mTLS is enabled (and Kubernetes is outside of the Istio mesh). To overcome this, apply a PERMISSIVE Istio authentication policy to the vault-secrets-webhook Service itself, so Kubernetes can call it back without Istio mutual TLS authentication.

    kubectl create namespace vault-system
    kubectl label namespace vault-system name=vault-system istio-injection=enabled
    
    kubectl apply -f - <<EOF
    apiVersion: authentication.istio.io/v1alpha1
    kind: Policy
    metadata:
      name: vault-secrets-webhook
      namespace: vault-system
      labels:
        app: security
    spec:
      targets:
      - name: vault-secrets-webhook
      peers:
      - mtls:
          mode: PERMISSIVE
    EOF
    
  2. Now you can install the operator and the webhook to the prepared namespace:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm upgrade --install vault-secrets-webhook banzaicloud-stable/vault-secrets-webhook --namespace vault-system
    helm upgrade --install vault-operator banzaicloud-stable/vault-operator --namespace vault-system
    

Soon the webhook and the operator become up and running. Check that the istio-proxy got injected into all Pods in vault-system.

Proceed to the description of your scenario:

10.1 - Scenario 1 - Vault runs outside, the application inside the mesh

In this scenario, Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled.

First, complete the Prerequisites, then install Vault outside the mesh, and finally install an application within the mesh.

Install Vault outside the mesh

  1. Provision a Vault instance with the Bank-Vaults operator in a separate namespace:

    kubectl create namespace vault
    
  2. Apply the RBAC and CR files to the cluster to create a Vault instance in the vault namespace with the operator:

    kubectl apply -f rbac.yaml -f cr-istio.yaml
    
    kubectl get pods -n vault
    

    Expected output:

    NAME                               READY   STATUS    RESTARTS   AGE
    vault-0                            3/3     Running   0          22h
    vault-configurer-6458cc4bf-6tpkz   1/1     Running   0          22h
    

    If you are writing your own Vault CR make sure that istioEnabled: true is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly.

  3. The vault-secrets-webhook can’t inject Vault secrets into initContainers in an Istio-enabled namespace when the STRICT authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase when initContainers are running the Pod doesn’t have a sidecar yet. If you wish to inject into initContainers as well, you need to apply a PERMISSIVE authentication policy in the vault namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).

    kubectl apply -f - <<EOF
    apiVersion: authentication.istio.io/v1alpha1
    kind: Policy
    metadata:
      name: default
      namespace: vault
      labels:
        app: security
    spec:
      peers:
      - mtls:
          mode: PERMISSIVE
    EOF
    

Install the application inside a mesh

In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:

  1. Create a namespace first for the application and enable Istio sidecar injection:

    kubectl create namespace app
    kubectl label namespace app istio-injection=enabled
    
  2. Install the application manifest to the cluster:

    kubectl apply -f app.yaml
    
  3. Check that the application is up and running. It should have two containers, the app itself and the istio-proxy:

    kubectl get pods -n app
    

    Expected output:

    NAME                  READY   STATUS    RESTARTS   AGE
    app-5df5686c4-sl6dz   2/2     Running   0          119s
    
    kubectl logs -f -n app deployment/app app
    

    Expected output:

    time="2020-02-18T14:26:01Z" level=info msg="Received new Vault token"
    time="2020-02-18T14:26:01Z" level=info msg="Initial Vault token arrived"
    s3cr3t
    going to sleep...
    

10.2 - Scenario 2 - Running Vault inside the mesh

To run Vault inside the mesh, complete the following steps.

Note: These instructions assume that you have Scenario 1 up and running, and modifying it to run Vault inside the mesh.

  1. Turn off Istio in the app namespace by removing the istio-injection label:

    kubectl label namespace app istio-injection-
    kubectl label namespace vault istio-injection=enabled
    
  2. Delete the Vault pods in the vault namespace, so they will get recreated with the istio-proxy sidecar:

    kubectl delete pods --all -n vault
    
  3. Check that they both come back with an extra container (4/4 and 2/2 now):

    kubectl get pods -n vault
    

    Expected output:

    NAME                                READY   STATUS    RESTARTS   AGE
    vault-0                             4/4     Running   0          1m
    vault-configurer-6d9b98c856-l4flc   2/2     Running   0          1m
    
  4. Delete the application pods in the app namespace, so they will get recreated without the istio-proxy sidecar:

    kubectl delete pods --all -n app
    

The app pod got recreated with only the app container (1/1) and Vault access still works:

kubectl get pods -n app

Expected output:

NAME                  READY   STATUS    RESTARTS   AGE
app-5df5686c4-4n6r7   1/1     Running   0          71s
kubectl logs -f -n app deployment/app

Expected output:

time="2020-02-18T14:41:20Z" level=info msg="Received new Vault token"
time="2020-02-18T14:41:20Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...

10.3 - Scenario 3 - Both Vault and the app are running inside the mesh

In this scenario, both Vault and the app are running inside the mesh.

  1. Complete the Prerequisites.

  2. Enable sidecar auto-injection for both namespaces:

    kubectl label namespace app   istio-injection=enabled
    kubectl label namespace vault istio-injection=enabled
    
  3. Delete all pods so they are getting injected with the proxy:

    kubectl delete pods --all -n app
    kubectl delete pods --all -n vault
    
  4. Check the logs in the app container. It should sill show success:

    kubectl logs -f -n app deployment/app
    

    Expected output:

    time="2020-02-18T15:04:03Z" level=info msg="Initial Vault token arrived"
    time="2020-02-18T15:04:03Z" level=info msg="Renewed Vault Token"
    s3cr3t
    going to sleep...
    

11 - HSM Support

Bank-Vaults offers a lot of alternatives for encrypting and storing the unseal-keys and the root-token for Vault. One of the encryption technics is the HSM - Hardware Security Module. HSM offers an industry-standard way to encrypt your data in on-premise environments.

You can use a Hardware Security Module (HSM) to generate and store the private keys used by Bank-Vaults. Some articles still point out the speed of HSM devices as their main selling point, but an average PC can do more cryptographic operations. Actually, the main benefit is from the security point of view. An HSM protects your private keys and handles cryptographic operations, which allows the encryption of protected information without exposing the private keys (they are not extractable). Bank-Vaults currently supports the PKCS11 software standard to communicate with an HSM. Fulfilling compliance requirements (for example, PCI DSS) is also a great benefit of HSMs, so from now on you can achieve that with Bank-Vaults.

Implementation in Bank-Vaults

Vault HSM

To support HSM devices for encrypting unseal-keys and root-tokens, Bank-Vaults:

  • implements an encryption/decryption Service named hsm in the bank-vaults CLI,
  • the bank-vaults Docker image now includes the SoftHSM (for testing) and the OpenSC tooling,
  • the operator is aware of HSM and its nature.

The HSM offers an encryption mechanism, but the unseal-keys and root-token have to be stored somewhere after they got encrypted. Currently there are two possible solutions for that:

  • Some HSM devices can store a limited quantity of arbitrary data (like Nitrokey HSM), and Bank-Vaults can store the unseal-keys and root-token here.
  • If the HSM does not support that, Bank-Vaults uses the HSM to encrypt the unseal-keys and root-token, then stores them in Kubernetes Secrets. We believe that it is safe to store these keys in Kubernetes Secrets in encrypted format.

Bank-Vaults offers the ability to use the pre-created the cryptographic encryption keys on the HSM device, or generate a key pair on the fly if there isn’t any with the specified label in the specified slot.

Since Bank-Vaults is written in Go, it uses the github.com/miekg/pkcs11 wrapper to pull in the PKCS11 library, to be more precise the p11 high-level wrapper .

Supported HSM solutions

Bank-Vaults currently supports the following HSM solutions:

  • SoftHSM, recommended for testing
  • NitroKey HSM.
  • AWS CloudHSM supports the PKCS11 API as well, so it probably works, though it needs a custom Docker image.

11.1 - NitroKey HSM support (OpenSC)

Nitrokey HSM is a USB HSM device based on the OpenSC project. We are using NitroKey to develop real hardware-based HSM support for Bank-Vaults. This device is not a cryptographic accelerator, only key generation and the private key operations (sign and decrypt) are supported. Public key operations should be done by extracting the public key and working on the computer, and this is how it is implemented in Bank-Vaults. It is not possible to extract private keys from NitroKey HSM, the device is tamper-resistant.

This device supports only RSA based encryption/decryption, and thus this is implemented in Bank-Vaults currently. It supports ECC keys as well, but only for sign/verification operations.

To start using a NitroKey HSM, complete the following steps.

  1. Install OpenSC and initialize the NitroKey HSM stick:

    brew install opensc
    sc-hsm-tool --initialize --label bank-vaults --pin banzai --so-pin banzaicloud
    pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults
    
  2. Check that you got a keypair object in slot 0:

    pkcs11-tool --list-objects
    
    
    Using slot 0 with a present token (0x0)
    Public Key Object; RSA 2048 bits
      label:      bank-vaults
      ID:         a9548075b20243627e971873826ead172e932359
      Usage:      encrypt, verify, wrap
      Access:     none
    
    pkcs15-tool --list-keys
    
    Using reader with a card: Nitrokey Nitrokey HSM
    Private RSA Key [bank-vaults]
      Object Flags   : [0x03], private, modifiable
      Usage          : [0x0E], decrypt, sign, signRecover
      Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
      ModLength      : 2048
      Key ref        : 1 (0x01)
      Native         : yes
      Auth ID        : 01
      ID             : a9548075b20243627e971873826ead172e932359
      MD:guid        : a6b2832c-1dc5-f4ef-bb0f-7b3504f67015
    
  3. If you are testing the HSM on macOS, setup minikube. Otherwise, continue with the next step.

  4. Configure the operator to use NitroKey HSM for unsealing.

    You must adjust the unsealConfig section in the vault-operator configuration, so the operator can communicate with OpenSC HSM devices correctly. Adjust your configuration based on the following snippet:

      # This example relies on an OpenSC HSM (NitroKey HSM) device initialized and plugged in to the Kubernetes Node.
      unsealConfig:
        hsm:
          # OpenSC daemon is needed in this case to communicate with the device
          daemon: true
          # The HSM SO module path (opensc is built into the bank-vaults image)
          modulePath: /usr/lib/opensc-pkcs11.so
          # For OpenSC slotId is the preferred way instead of tokenLabel
          # (OpenSC appends/prepends some extra stuff to labels)
          slotId: 0
          pin: banzai # This can be specified in the BANK_VAULTS_HSM_PIN environment variable as well, from a Secret
          keyLabel: bank-vaults
    
  5. Configure your Kubernetes node that has the HSM attached so Bank-Vaults can access it.

Setup on Minikube for testing (optional)

On macOS where you run Docker in VMs you need to do some extra steps before mounting your HSM device to Kubernetes. Complete the following steps to mount NitroKey into the minikube Kubernetes cluster:

  1. Make sure that the Oracle VM VirtualBox Extension Pack for USB 2.0 support is installed.

    VBoxManage list extpacks
    
  2. Remove the HSM device from your computer if it is already plugged in.

  3. Specify VirtualBox as the VM backend for Minikube.

    minikube config set vm-driver virtualbox
    
  4. Create a minikube cluster with the virtualbox driver and stop it, so you can modify the VM.

    minikube start
    minikube stop
    
  5. Enable USB 2.0 support for the minikube VM.

    VBoxManage modifyvm minikube --usbehci on
    
  6. Find the vendorid and productid for your Nitrokey HSM device.

    VBoxManage list usbhost
    
    VENDORID=0x20a0
    PRODUCTID=0x4230
    
  7. Create a filter for it.

    VBoxManage usbfilter add 1 --target minikube --name "Nitrokey HSM" --vendorid ${VENDORID} --productid ${PRODUCTID}
    
  8. Restart the minikube VM.

    minikube start
    
  9. Plug in the USB device.

  10. Check that minikube captured your NitorKey HSM.

    minikube ssh lsusb | grep ${VENDORID:2}:${PRODUCTID:2}
    

Now your minikube Kubernetes cluster has access to the HSM device through USB.

Kubernetes node setup

Some HSM vendors offer network daemons to enhance the reach of their HSM equipment to different servers. Unfortunately, there is no networking standard defined for PKCS11 access and thus currently Bank-Vaults has to be scheduled to the same node where the HSM device is attached directly (if not using a Cloud HSM).

Since the HSM is a hardware device connected to a physical node, Bank-Vaults has to find its way to that node. To make this work, create an HSM extended resource on the Kubernetes nodes for which the HSM device is plugged in. Extended resources must be advertised in integer amounts, for example, a Node can advertise four HSM devices, but not 4.5.

  1. You need to patch the node to specify that it has an HSM device as a resource. Because of the integer constraint and because all Bank-Vaults related Pods has to land on a Node where an HSM resource is available we need to advertise two units for 1 device, one will be allocated by each Vault Pod and one by the Configurer. If you would like to run Vault in HA mode - multiple Vault instances in different nodes - you will need multiple HSM devices plugged into those nodes, having the same key and slot setup.

    kubectl proxy &
    
    NODE=minikube
    
    curl --header "Content-Type: application/json-patch+json" \
        --request PATCH \
        --data '[{"op": "add", "path": "/status/capacity/nitrokey.com~1hsm", "value": "2"}]' \
        http://localhost:8001/api/v1/nodes/${NODE}/status
    

    From now on, you can request the nitrokey.com/hsm resource in the PodSpec

  2. Include the nitrokey.com/hsm resource in your PodSpec:

      # If using the NitroKey HSM example, that resource has to be part of the resource scheduling request.
      resources:
        hsmDaemon:
          requests:
            cpu: 100m
            memory: 64Mi
            nitrokey.com/hsm: 1
          limits:
            cpu: 200m
            memory: 128Mi
            nitrokey.com/hsm: 1
    
  3. Apply the modified setup from scratch:

    kubectl delete vault vault
    kubectl delete pvc vault-file-vault-0
    kubectl delete secret vault-unseal-keys
    kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-hsm-nitrokey.yaml
    
  4. Check the logs that unsealing uses the NitroKey HSM device. Run the following command:

    kubectl logs -f vault-0 bank-vaults
    

    The output should be something like:

    time="2020-03-04T13:32:29Z" level=info msg="HSM Information {CryptokiVersion:{Major:2 Minor:20} ManufacturerID:OpenSC Project Flags:0 LibraryDescription:OpenSC smartcard framework LibraryVersion:{Major:0 Minor:20}}"
    time="2020-03-04T13:32:29Z" level=info msg="HSM Searching for slot in HSM slots [{ctx:0xc0000c0318 id:0}]"
    time="2020-03-04T13:32:29Z" level=info msg="found HSM slot 0 in HSM by slot ID"
    time="2020-03-04T13:32:29Z" level=info msg="HSM TokenInfo {Label:bank-vaults (UserPIN)\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ManufacturerID:www.CardContact.de Model:PKCS#15 emulated SerialNumber:DENK0200074 Flags:1037 MaxSessionCount:0 SessionCount:0 MaxRwSessionCount:0 RwSessionCount:0 MaxPinLen:15 MinPinLen:6 TotalPublicMemory:18446744073709551615 FreePublicMemory:18446744073709551615 TotalPrivateMemory:18446744073709551615 FreePrivateMemory:18446744073709551615 HardwareVersion:{Major:24 Minor:13} FirmwareVersion:{Major:3 Minor:3} UTCTime:\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}"
    time="2020-03-04T13:32:29Z" level=info msg="HSM SlotInfo for slot 0: {SlotDescription:Nitrokey Nitrokey HSM (DENK02000740000         ) 00 00 ManufacturerID:Nitrokey Flags:7 HardwareVersion:{Major:0 Minor:0} FirmwareVersion:{Major:0 Minor:0}}"
    time="2020-03-04T13:32:29Z" level=info msg="found objects with label \"bank-vaults\" in HSM"
    time="2020-03-04T13:32:29Z" level=info msg="this HSM device doesn't support encryption, extracting public key and doing encrytion on the computer"
    time="2020-03-04T13:32:29Z" level=info msg="no storage backend specified for HSM, using on device storage"
    time="2020-03-04T13:32:29Z" level=info msg="joining leader vault..."
    time="2020-03-04T13:32:29Z" level=info msg="vault metrics exporter enabled: :9091/metrics"
    [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
    - using env:	export GIN_MODE=release
    - using code:	gin.SetMode(gin.ReleaseMode)
    
    [GIN-debug] GET    /metrics                  --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)
    [GIN-debug] Listening and serving HTTP on :9091
    time="2020-03-04T13:32:30Z" level=info msg="initializing vault..."
    time="2020-03-04T13:32:30Z" level=info msg="initializing vault"
    time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-0
    time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-1
    time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-2
    time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-3
    time="2020-03-04T13:32:33Z" level=info msg="unseal key stored in key store" key=vault-unseal-4
    time="2020-03-04T13:32:33Z" level=info msg="root token stored in key store" key=vault-root
    time="2020-03-04T13:32:33Z" level=info msg="vault is sealed, unsealing"
    time="2020-03-04T13:32:39Z" level=info msg="successfully unsealed vault"
    
  5. Find the unseal keys and the root token on the HSM:

    pkcs11-tool --list-objects
    

    Expected output:

    Using slot 0 with a present token (0x0)
    Public Key Object; RSA 2048 bits
      label:      bank-vaults
      ID:         a9548075b20243627e971873826ead172e932359
      Usage:      encrypt, verify, wrap
      Access:     none
    Data object 2168561792
      label:          'vault-test'
      application:    'vault-test'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168561168
      label:          'vault-unseal-0'
      application:    'vault-unseal-0'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168561264
      label:          'vault-unseal-1'
      application:    'vault-unseal-1'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168561360
      label:          'vault-unseal-2'
      application:    'vault-unseal-2'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168562304
      label:          'vault-unseal-3'
      application:    'vault-unseal-3'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168562400
      label:          'vault-unseal-4'
      application:    'vault-unseal-4'
      app_id:         <empty>
      flags:           modifiable
    Data object 2168562496
      label:          'vault-root'
      application:    'vault-root'
      app_id:         <empty>
      flags:           modifiable
    
  6. If you don’t need the encryption keys or the unseal keys on the HSM anymore, you can delete them with the following commands:

    PIN=banzai
    
    # Delete the unseal keys and the root token
    for label in "vault-test" "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
    do
      pkcs11-tool --delete-object --type data --label ${label} --pin ${PIN}
    done
    
    # Delete the encryption key
    pkcs11-tool --delete-object --type privkey --label bank-vaults --pin ${PIN}
    

11.2 - SoftHSM support for testing

You can use SoftHSMv2 to implement and test software interacting with PKCS11 implementations. You can install it on macOS by running the following commands:

# Initializing SoftHSM to be able to create a working example (only for dev),
# sharing the HSM device is emulated with a pre-created keypair in the image.
brew install softhsm
softhsm2-util --init-token --free --label bank-vaults --so-pin banzai --pin banzai
pkcs11-tool --module /usr/local/lib/softhsm/libsofthsm2.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

To interact with SoftHSM when using the vault-operator, include the following unsealConfig snippet in the Vault CR:

  # This example relies on the SoftHSM device initialized in the Docker image.
  unsealConfig:
    hsm:
      # The HSM SO module path (softhsm is built into the bank-vaults image)
      modulePath: /usr/lib/softhsm/libsofthsm2.so 
      tokenLabel: bank-vaults
      pin: banzai
      keyLabel: bank-vaults

To run the whole SoftHSM based example in Kubernetes, run the following commands:

kubectl create namespace vault-infra
helm upgrade --install vault-operator banzaicloud-stable/vault-operator --namespace vault-infra
kubectl apply -f https://raw.githubusercontent.com/banzaicloud/bank-vaults/master/operator/deploy/cr-hsm-softhsm.yaml

12 - The CLI tool

The bank-vaults CLI tool is to help automate the setup and management of HashiCorp Vault.

Features:

  • Initializes Vault and stores the root token and unseal keys in one of the followings:
    • AWS KMS keyring (backed by S3)
    • Azure Key Vault
    • Google Cloud KMS keyring (backed by GCS)
    • Alibaba Cloud KMS (backed by OSS)
    • Kubernetes Secrets (should be used only for development purposes)
    • Dev Mode (useful for vault server -dev dev mode Vault servers)
    • Files (backed by files, should be used only for development purposes)
  • Automatically unseals Vault with these keys
  • In addition to the standard Vault configuration, the operator and CLI can continuously configure Vault using an external YAML/JSON configuration
    • If the configuration is updated Vault will be reconfigured
    • It supports configuring Vault secret engines, plugins, auth methods, and policies

The bank-vaults CLI command needs certain cloud permissions to function properly (init, unseal, configuration).

13 - The Go library

The Bank-Vaults repository contains several Go packages for interacting with Vault, these packages are organized into the sdk Go module, which can be pulled in with go get github.com/bank-vaults/bank-vaults/pkg/sdk and is versioned by the pkg/sdk/vX.Y.Z Git tags:

  • pkg/sdk/auth

    Stores JWT bearer tokens in Vault.

    (NOTE: The Gin handler has been moved out to gin-utilz )

    authn

  • pkg/sdk/vault

    A wrapper for the official Vault client with automatic token renewal, and Kubernetes support.

    token

  • pkg/sdk/db

    A helper for creating database source strings (MySQL/PostgreSQL) with database credentials dynamically based on configured Vault roles (instead of username:password).

    token

  • pkg/sdk/tls

    A simple package to generate self-signed TLS certificates. Useful for bootstrapping situations, when you can’t use Vault’s PKI secret engine.

Examples for using the library part

Some examples are in cmd/examples/main.go

14 - Monitoring

At Banzai Cloud we prefer Prometheus for monitoring and use it also for Vault. If you configure, Vault can expose metrics through statsd. Both the Helm chart and the Vault Operator installs the Prometheus StatsD exporter and annotates the pods correctly with Prometheus annotations so Prometheus can discover and scrape them. All you have to do is to put the telemetry stanza into your Vault configuration:

    telemetry:
      statsd_address: localhost:9125

You may find the generic Prometheus kubernetes client Go Process runtime monitoring dashboard useful for monitoring the webhook or any other Go process.

To monitor the mutating webhook, see Monitoring the Webhook with Grafana and Prometheus.

15 - Annotations and labels

Annotations

The Vault Operator supports annotating most of the resources it creates using a set of fields in the Vault Specs:

Common Vault Resources annotations

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      annotations:
          example.com/test: "something"

These annotations are common to all Vault Created resources

  • Vault Statefulset
  • Vault Pods
  • Vault Configurer Deployment
  • Vault Configurer Pod
  • Vault Services
  • Vault Configurer Service
  • Vault TLS Secret

Vault Statefulset Resources annotations

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      vaultAnnotations:
          example.com/vault: "true"

These annotations are common to all Vault Statefulset Created resources

  • Vault Statefulset
  • Vault Pods
  • Vault Services
  • Vault TLS Secret

These annotations will override any annotation defined in the common set

Vault Configurer deployment Resources annotations

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      vaultConfigurerAnnotations:
          example.com/vaultConfigurer: "true"

These annotations are common to all Vault Configurer Deployment Created resources

  • Vault Configurer Deployment
  • Vault Configurer Pod
  • Vault Configurer Service

These annotations will override any annotation defined in the common set

ETCD CRD Annotations

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      etcdAnnotations:
          etcd.database.coreos.com/scope: clusterwide

These annotations are set only on the etcdcluster resource

ETCD PODs Annotations

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      etcdPodAnnotations:
          backup.velero.io/backup-volumes: "YOUR_VOLUME_NAME"

These annotations are set only on the etcd pods created by the etcd-operator

Labels

The Vault Operator support labelling most of the resources it creates using a set of fields in the Vault Specs:

Vault Statefulset Resources labels

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      vaultLabels:
          example.com/log-format: "json"

These Labels are common to all Vault Statefulset Created resources

  • Vault Statefulset
  • Vault Pods
  • Vault Services
  • Vault TLS Secret

Vault Configurer deployment Resources labels

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
    spec:
      vaultConfigurerLabels:
          example.com/log-format: "string"

These labels are common to all Vault Configurer Deployment Created resources

  • Vault Configurer Deployment
  • Vault Configurer Pod
  • Vault Configurer Service

16 - Watching External Secrets

In some cases, you might have to restart the Vault Statefulset when secrets that are not managed by the operator control are changed. For example:

  • Cert-Manager managing a public Certificate for vault using Let’s Encrypt.
  • Cloud IAM Credentials created with an external tool (like terraform) to allow vault to interact with the cloud services.

The operator can watch a set of secrets in the namespace of the Vault resource using either a list of label selectors or an annotations selector. When the content of any of those secrets changes, the operator updates the statefulset and triggers a rolling restart.

Configure label selectors

Set the secrets to watch using the watchedSecretsAnnotations and watchedSecretsLabels fields in your Vault custom resource.

Note: For cert-manager 0.11 or newer, use the watchedSecretsAnnotations field.

In the following example, the Vault StatefulSet is restarted when:

  • A secret with label certmanager.k8s.io/certificate-name: vault-letsencrypt-cert changes its contents (cert-manager 0.10 and earlier).
  • A secret with label test.com/scope: gcp AND test.com/credentials: vault changes its contents.
  • A secret with annotation cert-manager.io/certificate-name: vault-letsencrypt-cert changes its contents (cert-manager 0.11 and newer).
watchedSecretsLabels:
  - certmanager.k8s.io/certificate-name: vault-letsencrypt-cert
  - test.com/scope: gcp
    test.com/credentials: vault

watchedSecretsAnnotations:
  - cert-manager.io/certificate-name: vault-letsencrypt-cert

The operator controls the restart of the statefulset by adding an annotation to the spec.template of the vault resource

kubectl get -n vault statefulset vault -o json | jq .spec.template.metadata.annotations
{
  "prometheus.io/path": "/metrics",
  "prometheus.io/port": "9102",
  "prometheus.io/scrape": "true",
  "vault.banzaicloud.io/watched-secrets-sum": "ff1f1c79a31f76c68097975977746be9b85878f4737b8ee5a9d6ee3c5169b0ba"
}

17 - Tips and tricks

The following section lists some questions, problems, and tips that came up on the Bank-Vaults Slack channel.

Login to the Vault web UI

To login to the Vault web UI, you can use the root token, or any configured authentication backend.

Can changing the vault CR delete the Vault instance and data?

Bank-Vaults never ever deletes the Vault instance from the cluster. However, if you delete the Vault CR, then the Kubernetes garbage controller deletes the vault pods. You are recommended to keep backups.

Set default for vault.security.banzaicloud.io/vault-addr

You can set the default settings for vault.security.banzaicloud.io/vault-addr so you don’t have to specify it in every PodSpec. Just set the VAULT_ADDR in the env section: https://github.com/bank-vaults/bank-vaults/blob/3c89e831bdb21b2733680f13ceec7ac4b6e3f892/charts/vault-secrets-webhook/values.yaml#L37

18 - Support

Bank-Vaults is a Vault swiss-army knife: a K8s operator, Go client with automatic token renewal, automatic configuration, multiple unseal options and more. A CLI tool to init, unseal and configure Vault (auth methods, secret engines). Direct secret injection into Pods.

Community support

If you encounter problems while using Bank-Vaults that the documentation does not address, open an issue or talk to us on the Emerging Tech Community Slack channel #bank-vaults.

19 - Contributing

If you find this project useful here’s how you can help:

Development environment

In your development environment you can use file mode for testing bank-vaults cli-tool:

vault server -config vault.hcl

example vault.hcl:

api_addr = "http://localhost:8200"

storage "file" {
  path = "/tmp/vault"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = true
}

Now you have a running vault server which is uninitialized and unsealed you can init and unseal it with bank-vaults cli-tool and unseal keys will be stored to a local file:

VAULT_ADDR=http://127.0.0.1:8200 bank-vaults unseal --init --mode file

The unseal keys and root token are stored your working directory:

vault-root
vault-unseal-0
vault-unseal-1
vault-unseal-2
vault-unseal-3
vault-unseal-4

Operator

Developing the operator requires a working Kubernetes cluster, minikube and Docker for Mac Kubernetes will suffice.

The operator consists of two parts, the bank-vaults sidecar running inside a container and the operator itself.

You can fire up the operator on your machine, so you can debug it locally (yes you don’t have to build a container from it), if your kube context points to the development cluster:

make operator-up

This installs all the necessary RBAC rules and other CRDs that you need to create a Vault instance. If you change the code of the operator you have to CTRL + C this make command and rerun it again.

Now it is time create a Vault instance for yourself, which you can work on:

kubectl apply -f operator/deploy/cr.yaml

If you change the bank-vaults sidecar code you have to build a new Docker image from it:

DOCKER_LATEST=1 make docker

There are at least four ways to distribute this image in your Kubernetes cluster, by default IfNotPresent image pull policy is used:

  • If you are using Docker for Mac, you don’t have to anything, the Kubernetes cluster and your host shares the same Docker daemon.
  • If you are using Minikube with --vm-driver=none (you are probably using Linux) the same applies as for Docker for Mac
  • If you are using Minikube with some real vm-driver you have to run eval $(minikube docker-env) before building the Docker image with the make command so you build it with the minikube Docker daemon and the image will be stored there
  • Build and re-tag the image and push it to the Docker registry of your choice, dont forget to change the bankVaultsImage attribute in the the Vault Custom Resource YAML file (cr.yaml in this case).

Restart the containers using the bank-vaults image: Vault instances and the configurer.

WebHook

This will deploy the webhook via the Helm chart, scale it to 0, start it locally and proxy it into the cluster (somehow similar to operator-up but a bit more complex).

You will need Helm and kurun installed to run this:

make webhook-up -j

Now you can try out with mutating a Deployment:

kubectl apply -f deploy/test-deployment.yaml