This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

HSM Support

Bank-Vaults offers a lot of alternatives for encrypting and storing the unseal-keys and the root-token for Vault. One of the encryption technics is the HSM - Hardware Security Module. HSM offers an industry-standard way to encrypt your data in on-premise environments.

You can use a Hardware Security Module (HSM) to generate and store the private keys used by Bank-Vaults. Some articles still point out the speed of HSM devices as their main selling point, but an average PC can do more cryptographic operations. Actually, the main benefit is from the security point of view. An HSM protects your private keys and handles cryptographic operations, which allows the encryption of protected information without exposing the private keys (they are not extractable). Bank-Vaults currently supports the PKCS11 software standard to communicate with an HSM. Fulfilling compliance requirements (for example, PCI DSS) is also a great benefit of HSMs, so from now on you can achieve that with Bank-Vaults.

Implementation in Bank-Vaults

Vault HSM

To support HSM devices for encrypting unseal-keys and root-tokens, Bank-Vaults:

  • implements an encryption/decryption Service named hsm in the bank-vaults CLI,
  • the bank-vaults Docker image now includes the SoftHSM (for testing) and the OpenSC tooling,
  • the operator is aware of HSM and its nature.

The HSM offers an encryption mechanism, but the unseal-keys and root-token have to be stored somewhere after they got encrypted. Currently there are two possible solutions for that:

  • Some HSM devices can store a limited quantity of arbitrary data (like Nitrokey HSM), and Bank-Vaults can store the unseal-keys and root-token here.
  • If the HSM does not support that, Bank-Vaults uses the HSM to encrypt the unseal-keys and root-token, then stores them in Kubernetes Secrets. We believe that it is safe to store these keys in Kubernetes Secrets in encrypted format.

Bank-Vaults offers the ability to use the pre-created the cryptographic encryption keys on the HSM device, or generate a key pair on the fly if there isn’t any with the specified label in the specified slot.

Since Bank-Vaults is written in Go, it uses the github.com/miekg/pkcs11 wrapper to pull in the PKCS11 library, to be more precise the p11 high-level wrapper .

Supported HSM solutions

Bank-Vaults currently supports the following HSM solutions:

  • SoftHSM, recommended for testing
  • NitroKey HSM.
  • AWS CloudHSM supports the PKCS11 API as well, so it probably works, though it needs a custom Docker image.

1 - NitroKey HSM support (OpenSC)

Nitrokey HSM is a USB HSM device based on the OpenSC project. We are using NitroKey to develop real hardware-based HSM support for Bank-Vaults. This device is not a cryptographic accelerator, only key generation and the private key operations (sign and decrypt) are supported. Public key operations should be done by extracting the public key and working on the computer, and this is how it is implemented in Bank-Vaults. It is not possible to extract private keys from NitroKey HSM, the device is tamper-resistant.

This device supports only RSA based encryption/decryption, and thus this is implemented in Bank-Vaults currently. It supports ECC keys as well, but only for sign/verification operations.

To start using a NitroKey HSM, complete the following steps.

  1. Install OpenSC and initialize the NitroKey HSM stick:

    brew install opensc
sc-hsm-tool --initialize --label bank-vaults --pin banzai --so-pin banzaicloud
pkcs11-tool --module /usr/local/lib/opensc-pkcs11.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

  2. Check that you got a keypair object in slot 0:

    pkcs11-tool --list-objects


Using slot 0 with a present token (0x0)
Public Key Object; RSA 2048 bits
  label:      bank-vaults
  ID:         a9548075b20243627e971873826ead172e932359
  Usage:      encrypt, verify, wrap
  Access:     none

    pkcs15-tool --list-keys

Using reader with a card: Nitrokey Nitrokey HSM
Private RSA Key [bank-vaults]
  Object Flags   : [0x03], private, modifiable
  Usage          : [0x0E], decrypt, sign, signRecover
  Access Flags   : [0x1D], sensitive, alwaysSensitive, neverExtract, local
  ModLength      : 2048
  Key ref        : 1 (0x01)
  Native         : yes
  Auth ID        : 01
  ID             : a9548075b20243627e971873826ead172e932359
  MD:guid        : a6b2832c-1dc5-f4ef-bb0f-7b3504f67015

  3. If you are testing the HSM on macOS, setup minikube. Otherwise, continue with the next step.

  4. Configure the operator to use NitroKey HSM for unsealing.

    You must adjust the unsealConfig section in the vault-operator configuration, so the operator can communicate with OpenSC HSM devices correctly. Adjust your configuration based on the following snippet:

      # This example relies on an OpenSC HSM (NitroKey HSM) device initialized and plugged in to the Kubernetes Node.
  unsealConfig:
    hsm:
      # OpenSC daemon is needed in this case to communicate with the device
      daemon: true
      # The HSM SO module path (opensc is built into the bank-vaults image)
      modulePath: /usr/lib/opensc-pkcs11.so
      # For OpenSC slotId is the preferred way instead of tokenLabel
      # (OpenSC appends/prepends some extra stuff to labels)
      slotId: 0
      pin: banzai # This can be specified in the BANK_VAULTS_HSM_PIN environment variable as well, from a Secret
      keyLabel: bank-vaults

  5. Configure your Kubernetes node that has the HSM attached so Bank-Vaults can access it.

Setup on Minikube for testing (optional)

On macOS where you run Docker in VMs you need to do some extra steps before mounting your HSM device to Kubernetes. Complete the following steps to mount NitroKey into the minikube Kubernetes cluster:

  1. Make sure that the Oracle VM VirtualBox Extension Pack for USB 2.0 support is installed.

    VBoxManage list extpacks

  2. Remove the HSM device from your computer if it is already plugged in.

  3. Specify VirtualBox as the VM backend for Minikube.

    minikube config set vm-driver virtualbox

  4. Create a minikube cluster with the virtualbox driver and stop it, so you can modify the VM.

    minikube start
minikube stop

  5. Enable USB 2.0 support for the minikube VM.

    VBoxManage modifyvm minikube --usbehci on

  6. Find the vendorid and productid for your Nitrokey HSM device.

    VBoxManage list usbhost

VENDORID=0x20a0
PRODUCTID=0x4230

  7. Create a filter for it.

    VBoxManage usbfilter add 1 --target minikube --name "Nitrokey HSM" --vendorid ${VENDORID} --productid ${PRODUCTID}

  8. Restart the minikube VM.

    minikube start

  9. Plug in the USB device.

  10. Check that minikube captured your NitorKey HSM.

    minikube ssh lsusb | grep ${VENDORID:2}:${PRODUCTID:2}

Now your minikube Kubernetes cluster has access to the HSM device through USB.

Kubernetes node setup

Some HSM vendors offer network daemons to enhance the reach of their HSM equipment to different servers. Unfortunately, there is no networking standard defined for PKCS11 access and thus currently Bank-Vaults has to be scheduled to the same node where the HSM device is attached directly (if not using a Cloud HSM).

Since the HSM is a hardware device connected to a physical node, Bank-Vaults has to find its way to that node. To make this work, create an HSM extended resource on the Kubernetes nodes for which the HSM device is plugged in. Extended resources must be advertised in integer amounts, for example, a Node can advertise four HSM devices, but not 4.5.

  1. You need to patch the node to specify that it has an HSM device as a resource. Because of the integer constraint and because all Bank-Vaults related Pods has to land on a Node where an HSM resource is available we need to advertise two units for 1 device, one will be allocated by each Vault Pod and one by the Configurer. If you would like to run Vault in HA mode - multiple Vault instances in different nodes - you will need multiple HSM devices plugged into those nodes, having the same key and slot setup.

    kubectl proxy &

NODE=minikube

curl --header "Content-Type: application/json-patch+json" \
    --request PATCH \
    --data '[{"op": "add", "path": "/status/capacity/nitrokey.com~1hsm", "value": "2"}]' \
    http://localhost:8001/api/v1/nodes/${NODE}/status

    From now on, you can request the nitrokey.com/hsm resource in the PodSpec

  2. Include the nitrokey.com/hsm resource in your PodSpec:

      # If using the NitroKey HSM example, that resource has to be part of the resource scheduling request.
  resources:
    hsmDaemon:
      requests:
        cpu: 100m
        memory: 64Mi
        nitrokey.com/hsm: 1
      limits:
        cpu: 200m
        memory: 128Mi
        nitrokey.com/hsm: 1

  3. Apply the modified setup from scratch:

    kubectl delete vault vault
kubectl delete pvc vault-file-vault-0
kubectl delete secret vault-unseal-keys
kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.20.0/deploy/examples/cr-hsm-nitrokey.yaml

  4. Check the logs that unsealing uses the NitroKey HSM device. Run the following command:

    kubectl logs -f vault-0 bank-vaults

    The output should be something like:

    time="2020-03-04T13:32:29Z" level=info msg="HSM Information {CryptokiVersion:{Major:2 Minor:20} ManufacturerID:OpenSC Project Flags:0 LibraryDescription:OpenSC smartcard framework LibraryVersion:{Major:0 Minor:20}}"
time="2020-03-04T13:32:29Z" level=info msg="HSM Searching for slot in HSM slots [{ctx:0xc0000c0318 id:0}]"
time="2020-03-04T13:32:29Z" level=info msg="found HSM slot 0 in HSM by slot ID"
time="2020-03-04T13:32:29Z" level=info msg="HSM TokenInfo {Label:bank-vaults (UserPIN)\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ManufacturerID:www.CardContact.de Model:PKCS#15 emulated SerialNumber:DENK0200074 Flags:1037 MaxSessionCount:0 SessionCount:0 MaxRwSessionCount:0 RwSessionCount:0 MaxPinLen:15 MinPinLen:6 TotalPublicMemory:18446744073709551615 FreePublicMemory:18446744073709551615 TotalPrivateMemory:18446744073709551615 FreePrivateMemory:18446744073709551615 HardwareVersion:{Major:24 Minor:13} FirmwareVersion:{Major:3 Minor:3} UTCTime:\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}"
time="2020-03-04T13:32:29Z" level=info msg="HSM SlotInfo for slot 0: {SlotDescription:Nitrokey Nitrokey HSM (DENK02000740000         ) 00 00 ManufacturerID:Nitrokey Flags:7 HardwareVersion:{Major:0 Minor:0} FirmwareVersion:{Major:0 Minor:0}}"
time="2020-03-04T13:32:29Z" level=info msg="found objects with label \"bank-vaults\" in HSM"
time="2020-03-04T13:32:29Z" level=info msg="this HSM device doesn't support encryption, extracting public key and doing encrytion on the computer"
time="2020-03-04T13:32:29Z" level=info msg="no storage backend specified for HSM, using on device storage"
time="2020-03-04T13:32:29Z" level=info msg="joining leader vault..."
time="2020-03-04T13:32:29Z" level=info msg="vault metrics exporter enabled: :9091/metrics"
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env:  export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /metrics                  --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers)
[GIN-debug] Listening and serving HTTP on :9091
time="2020-03-04T13:32:30Z" level=info msg="initializing vault..."
time="2020-03-04T13:32:30Z" level=info msg="initializing vault"
time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-0
time="2020-03-04T13:32:31Z" level=info msg="unseal key stored in key store" key=vault-unseal-1
time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-2
time="2020-03-04T13:32:32Z" level=info msg="unseal key stored in key store" key=vault-unseal-3
time="2020-03-04T13:32:33Z" level=info msg="unseal key stored in key store" key=vault-unseal-4
time="2020-03-04T13:32:33Z" level=info msg="root token stored in key store" key=vault-root
time="2020-03-04T13:32:33Z" level=info msg="vault is sealed, unsealing"
time="2020-03-04T13:32:39Z" level=info msg="successfully unsealed vault"

  5. Find the unseal keys and the root token on the HSM:

    pkcs11-tool --list-objects

    Expected output:

    Using slot 0 with a present token (0x0)
Public Key Object; RSA 2048 bits
  label:      bank-vaults
  ID:         a9548075b20243627e971873826ead172e932359
  Usage:      encrypt, verify, wrap
  Access:     none
Data object 2168561792
  label:          'vault-test'
  application:    'vault-test'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561168
  label:          'vault-unseal-0'
  application:    'vault-unseal-0'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561264
  label:          'vault-unseal-1'
  application:    'vault-unseal-1'
  app_id:         <empty>
  flags:           modifiable
Data object 2168561360
  label:          'vault-unseal-2'
  application:    'vault-unseal-2'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562304
  label:          'vault-unseal-3'
  application:    'vault-unseal-3'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562400
  label:          'vault-unseal-4'
  application:    'vault-unseal-4'
  app_id:         <empty>
  flags:           modifiable
Data object 2168562496
  label:          'vault-root'
  application:    'vault-root'
  app_id:         <empty>
  flags:           modifiable

  6. If you don’t need the encryption keys or the unseal keys on the HSM anymore, you can delete them with the following commands:

    PIN=banzai

# Delete the unseal keys and the root token
for label in "vault-test" "vault-root" "vault-unseal-0" "vault-unseal-1" "vault-unseal-2" "vault-unseal-3" "vault-unseal-4"
do
  pkcs11-tool --delete-object --type data --label ${label} --pin ${PIN}
done

# Delete the encryption key
pkcs11-tool --delete-object --type privkey --label bank-vaults --pin ${PIN}

2 - SoftHSM support for testing

You can use SoftHSMv2 to implement and test software interacting with PKCS11 implementations. You can install it on macOS by running the following commands:

# Initializing SoftHSM to be able to create a working example (only for dev),
# sharing the HSM device is emulated with a pre-created keypair in the image.
brew install softhsm
softhsm2-util --init-token --free --label bank-vaults --so-pin banzai --pin banzai
pkcs11-tool --module /usr/local/lib/softhsm/libsofthsm2.so --keypairgen --key-type rsa:2048 --pin banzai --token-label bank-vaults --label bank-vaults

To interact with SoftHSM when using the vault-operator, include the following unsealConfig snippet in the Vault CR:

  # This example relies on the SoftHSM device initialized in the Docker image.
  unsealConfig:
    hsm:
      # The HSM SO module path (softhsm is built into the bank-vaults image)
      modulePath: /usr/lib/softhsm/libsofthsm2.so 
      tokenLabel: bank-vaults
      pin: banzai
      keyLabel: bank-vaults

To run the whole SoftHSM based example in Kubernetes, run the following commands:

kubectl create namespace vault-infra
helm upgrade --install vault-operator oci://ghcr.io/bank-vaults/helm-charts/vault-operator --namespace vault-infra
kubectl apply -f https://raw.githubusercontent.com/bank-vaults/vault-operator/v1.21.0/deploy/examples/cr-hsm-softhsm.yaml