Backing up Vault

You can configure the vault-operator to create backups of the Vault cluster with Velero.

Prerequisites

• The Velero CLI must be installed on your computer.
• To create Persistent Volume (PV) snapshots, you need access to an object storage. The following example uses an Amazon S3 bucket called bank-vaults-velero in the Stockholm region.

Install Velero

To configure the vault-operator to create backups of the Vault cluster, complete the following steps.

1. Install Velero on the target cluster with Helm.

   1. Add the Velero Helm repository:

      helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts
      

   2. Create a namespace for Velero:

      kubectl create namespace velero
      

   3. Install Velero with Restic so you can create PV snapshots as well:

      BUCKET=bank-vaults-velero
      REGION=eu-north-1
      KMS_KEY_ID=alias/bank-vaults-velero
      SECRET_FILE=~/.aws/credentials
      
      helm upgrade --install velero --namespace velero \
                --set "configuration.backupStorageLocation[0].name"=aws \
                --set "configuration.backupStorageLocation[0].provider"=aws \
                --set "configuration.backupStorageLocation[0].bucket"=YOUR_BUCKET_NAME \
                --set "configuration.backupStorageLocation[0].config.region"=${REGION} \
                --set "configuration.backupStorageLocation[0].config.kmsKeyId"=${KMS_KEY_ID} \
                --set "configuration.volumeSnapshotLocation[0].name"=aws \
                --set "configuration.volumeSnapshotLocation[0].provider"=aws \
                --set "configuration.volumeSnapshotLocation[0].config.region"=${REGION} \
                --set "initContainers[0].name"=velero-plugin-for-aws \
                --set "initContainers[0].image"=velero/velero-plugin-for-aws:v1.7.0 \
                --set "initContainers[0].volumeMounts[0].mountPath"=/target \
                --set "initContainers[0].volumeMounts[0].name"=plugins \
                vmware-tanzu/velero
      

2. Install the vault-operator to the cluster:

   helm upgrade --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator
   
   kubectl apply -f operator/deploy/rbac.yaml
   kubectl apply -f operator/deploy/cr-raft.yaml
   

   Note: The Vault CR in cr-raft.yaml has a special flag called veleroEnabled. This is useful for file-based Vault storage backends (file, raft), see the Velero documentation:

     # Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:
     # https://velero.io/docs/v1.2.0/hooks/
     veleroEnabled: true
   

3. Create a backup with the Velero CLI or with the predefined Velero Backup CR:

   velero backup create --selector vault_cr=vault vault-1
   
   # OR
   
   kubectl apply -f https://raw.githubusercontent.com/bank-vaults/bank-vaults.dev/main/content/docs/operator/backup/backup.yaml
   

   Note: For a daily scheduled backup, see schedule.yaml.

4. Check that the Velero backup got created successfully:

   velero backup describe --details vault-1
   

   Expected output:

   Name:         vault-1
   Namespace:    velero
   Labels:       velero.io/backup=vault-1
                 velero.io/pv=pvc-6eb4d9c1-25cd-4a28-8868-90fa9d51503a
                 velero.io/storage-location=default
   Annotations:  <none>
   
   Phase:  Completed
   
   Namespaces:
     Included:  *
     Excluded:  <none>
   
   Resources:
     Included:        *
     Excluded:        <none>
     Cluster-scoped:  auto
   
   Label selector:  vault_cr=vault
   
   Storage Location:  default
   
   Snapshot PVs:  auto
   
   TTL:  720h0m0s
   
   Hooks:  <none>
   
   Backup Format Version:  1
   
   Started:    2020-01-29 14:17:41 +0100 CET
   Completed:  2020-01-29 14:17:45 +0100 CET
   
   Expiration:  2020-02-28 14:17:41 +0100 CET
   

Test the backup

1. To emulate a catastrophe, remove Vault entirely from the cluster:

   kubectl delete vault -l vault_cr=vault
   kubectl delete pvc -l vault_cr=vault
   

2. Now restore Vault from the backup.

   1. Scale down the vault-operator, so it won’t reconcile during the restore process:

      kubectl scale deployment vault-operator --replicas 0
      

   2. Restore all Vault-related resources from the backup:

      velero restore create --from-backup vault-1
      

   3. Check that the restore has finished properly:

      velero restore get
      NAME                    BACKUP   STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
      vault1-20200129142409   vault1   Completed   0          0        2020-01-29 14:24:09 +0100 CET   <none>
      

   4. Check that the Vault cluster got actually restored:

      kubectl get pods
      NAME                                READY   STATUS    RESTARTS   AGE
      vault-0                             4/4     Running   0          1m42s
      vault-1                             4/4     Running   0          1m42s
      vault-2                             4/4     Running   0          1m42s
      vault-configurer-5499ff64cb-g75vr   1/1     Running   0          1m42s
      

   5. Scale the operator back after the restore process:

      kubectl scale deployment vault-operator --replicas 1
      

3. Delete the backup if you don’t wish to keep it anymore:

   velero backup delete vault-1