Vault operator

The Vault operator builds on Bank-Vaults features such as:

  • external, API based configuration (secret engines, auth methods, policies) to automatically re/configure a Vault cluster
  • automatic unsealing (AWS, GCE, Azure, Alibaba, Kubernetes Secrets (for dev purposes), Oracle)
  • TLS support

The operator flow is the following:

operator

The source code can be found in the vault-operator repository.

The operator requires the following cloud permissions.

Deploy a local Vault operator

This is the simplest scenario: you install the Vault operator on a simple cluster. The following commands install a single-node Vault instance that stores unseal and root tokens in Kubernetes secrets. If you want to customize the Helm chart, see the list of vault-operator Helm chart values.

  1. Install the Bank-Vaults operator:

    helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
    

    Expected output:

    Release "vault-operator" does not exist. Installing it now.
    Pulled: ghcr.io/bank-vaults/helm-charts/vault-operator:1.20.0
    Digest: sha256:46045be1c3b215f0c734908bb1d4022dc91eae48d2285382bb71d63f72c737d1
    NAME: vault-operator
    LAST DEPLOYED: Thu Jul 27 11:22:55 2023
    NAMESPACE: default
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    
  2. Create a Vault instance using the Vault custom resources. This will create a Kubernetes CustomResource called vault and a PersistentVolumeClaim for it:

    kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
        

    Expected output:

    serviceaccount/vault created
        role.rbac.authorization.k8s.io/vault created
        role.rbac.authorization.k8s.io/leader-election-role created
        rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
        rolebinding.rbac.authorization.k8s.io/vault created
        clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
        
    kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml
        

    Expected output:

    vault.vault.banzaicloud.com/vault created
        

    Note: If needed, you can install the latest CustomResource from the main branch, but that’s usually under development and might not be stable.

    kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/crd | kubectl apply -f -
        
  3. Wait a few seconds, then check the operator and the vault pods:

    kubectl get pods
    

    Expected output:

    NAME                                                        READY     STATUS    RESTARTS   AGE
    vault-0                                                     3/3       Running   0          10s
    vault-configurer-6c545cb6b4-dmvb5                           1/1       Running   0          10s
    vault-operator-788559bdc5-kgqkg                             1/1       Running   0          23s
    
  4. Configure your Vault client to access the Vault instance running in the vault-0 pod.

    1. Port-forward into the pod:

      kubectl port-forward vault-0 8200 &
      
    2. Set the address of the Vault instance.

      export VAULT_ADDR=https://127.0.0.1:8200
      
    3. Import the CA certificate of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors):

      kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
      export VAULT_CACERT=$PWD/vault-ca.crt
      

      Alternatively, you can instruct the Vault client to skip verifying the certificate of Vault by running: export VAULT_SKIP_VERIFY=true

    4. If you already have the Vault CLI installed, check that you can access the Vault:

      vault status
      

      Expected output:

      Key             Value
      ---             -----
      Seal Type       shamir
      Initialized     true
      Sealed          false
      Total Shares    5
      Threshold       3
      Version         1.5.4
      Cluster Name    vault-cluster-27ecd0e6
      Cluster ID      ed5492f3-7ef3-c600-aef3-bd77897fd1e7
      HA Enabled      false
      
    5. To authenticate to Vault, you can access its root token by running:

      export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
      

      Note: Using the root token is recommended only in test environments. In production environment, create dedicated, time-limited tokens.

    6. Now you can interact with Vault. For example, add a secret by running vault kv put secret/demosecret/aws AWS_SECRET_ACCESS_KEY=s3cr3t If you want to access the Vault web interface, open https://127.0.0.1:8200 in your browser using the root token (to reveal the token, run echo $VAULT_TOKEN).

For other configuration examples of the Vault CustomResource, see the YAML files in the deploy/examples and test/deploy directories of the vault-operator repository. After you are done experimenting with Bank-Vaults and you want to delete the operator, you can delete the related CRs:

kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl delete -f -
kubectl delete -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml

HA setup with Raft

In a production environment you want to run Vault as a cluster. The following CR creates a 3-node Vault instance that uses the Raft storage backend:

  1. Install the Bank-Vaults operator:

    helm upgrade --install --wait vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
    
  2. Create a Vault instance using the cr-raft.yaml custom resource. This will create a Kubernetes CustomResource called vault that uses the Raft backend:

    kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/rbac | kubectl apply -f -
        

    Expected output:

    serviceaccount/vault created
        role.rbac.authorization.k8s.io/vault created
        role.rbac.authorization.k8s.io/leader-election-role created
        rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
        rolebinding.rbac.authorization.k8s.io/vault created
        clusterrolebinding.rbac.authorization.k8s.io/vault-auth-delegator created
        
    kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-raft.yaml
        

    Expected output:

    vault.vault.banzaicloud.com/vault created
        

    Note: If needed, you can install the latest CustomResource from the main branch, but that’s usually under development and might not be stable.

    kubectl kustomize https://github.com/bank-vaults/vault-operator/deploy/crd | kubectl apply -f -
        

CAUTION:

Make sure to set up a solution for backing up the storage backend to prevent data loss. Bank-Vaults doesn’t do this automatically. We recommend using Velero for backups.

Pod anti-affinity

If you want to setup pod anti-affinity, you can set podAntiAffinity vault with a topologyKey value. For example, you can use failure-domain.beta.kubernetes.io/zone to force K8S deploy vault on multi AZ.

Delete a resource created by the operator

If you manually delete a resource that the Bank-Vaults operator has created (for example, the Ingress resource), the operator automatically recreates it every 30 seconds. If it doesn’t, then something went wrong, or the operator is not running. In this case, check the logs of the operator.