Backing up Vault

You can configure the vault-operator to create backups of the Vault cluster with Velero.

Prerequisites

The Velero CLI must be installed on your computer.
To create Persistent Volume (PV) snapshots, you need access to an object storage. The following example uses an Amazon S3 bucket called bank-vaults-velero in the Stockholm region.

Install Velero

To configure the vault-operator to create backups of the Vault cluster, complete the following steps.

Install Velero on the target cluster with Helm.

Add the Velero Helm repository:

helm repo add vmware-tanzu https://vmware-tanzu.github.io/helm-charts

Create a namespace for Velero:
```
kubectl create namespace velero
```

Install Velero with Restic so you can create PV snapshots as well:

BUCKET=bank-vaults-velero
REGION=eu-north-1
KMS_KEY_ID=alias/bank-vaults-velero
SECRET_FILE=~/.aws/credentials

helm upgrade --install velero --namespace velero \
          --set "configuration.backupStorageLocation[0].name"=aws \
          --set "configuration.backupStorageLocation[0].provider"=aws \
          --set "configuration.backupStorageLocation[0].bucket"=YOUR_BUCKET_NAME \
          --set "configuration.backupStorageLocation[0].config.region"=${REGION} \
          --set "configuration.backupStorageLocation[0].config.kmsKeyId"=${KMS_KEY_ID} \
          --set "configuration.volumeSnapshotLocation[0].name"=aws \
          --set "configuration.volumeSnapshotLocation[0].provider"=aws \
          --set "configuration.volumeSnapshotLocation[0].config.region"=${REGION} \
          --set "initContainers[0].name"=velero-plugin-for-aws \
          --set "initContainers[0].image"=velero/velero-plugin-for-aws:v1.7.0 \
          --set "initContainers[0].volumeMounts[0].mountPath"=/target \
          --set "initContainers[0].volumeMounts[0].name"=plugins \
          vmware-tanzu/velero

Install the vault-operator to the cluster:

helm upgrade --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator

kubectl apply -f operator/deploy/rbac.yaml
kubectl apply -f operator/deploy/cr-raft.yaml

Note: The Vault CR in cr-raft.yaml has a special flag called veleroEnabled. This is useful for file-based Vault storage backends (file, raft), see the Velero documentation:

  # Add Velero fsfreeze sidecar container and supporting hook annotations to Vault Pods:
  # https://velero.io/docs/v1.2.0/hooks/
  veleroEnabled: true

Create a backup with the Velero CLI or with the predefined Velero Backup CR:

velero backup create --selector vault_cr=vault vault-1

# OR

kubectl apply -f https://raw.githubusercontent.com/bank-vaults/bank-vaults.dev/main/content/docs/operator/backup/backup.yaml

Note: For a daily scheduled backup, see schedule.yaml.

Check that the Velero backup got created successfully:

velero backup describe --details vault-1

Expected output:

Name:         vault-1
Namespace:    velero
Labels:       velero.io/backup=vault-1
              velero.io/pv=pvc-6eb4d9c1-25cd-4a28-8868-90fa9d51503a
              velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  vault_cr=vault

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-01-29 14:17:41 +0100 CET
Completed:  2020-01-29 14:17:45 +0100 CET

Expiration:  2020-02-28 14:17:41 +0100 CET

Test the backup

To emulate a catastrophe, remove Vault entirely from the cluster:

kubectl delete vault -l vault_cr=vault
kubectl delete pvc -l vault_cr=vault

Now restore Vault from the backup.

Scale down the vault-operator, so it won’t reconcile during the restore process:
```
kubectl scale deployment vault-operator --replicas 0
```
Restore all Vault-related resources from the backup:
```
velero restore create --from-backup vault-1
```

Check that the restore has finished properly:

velero restore get
NAME                    BACKUP   STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
vault1-20200129142409   vault1   Completed   0          0        2020-01-29 14:24:09 +0100 CET   <none>

Check that the Vault cluster got actually restored:

kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
vault-0                             4/4     Running   0          1m42s
vault-1                             4/4     Running   0          1m42s
vault-2                             4/4     Running   0          1m42s
vault-configurer-5499ff64cb-g75vr   1/1     Running   0          1m42s

Scale the operator back after the restore process:

kubectl scale deployment vault-operator --replicas 1

Delete the backup if you don’t wish to keep it anymore:
```
velero backup delete vault-1
```