Running the Bank-Vaults secret webhook alongside Istio

• 1: Scenario 1 - Vault runs outside, the application inside the mesh
• 2: Scenario 2 - Running Vault inside the mesh
• 3: Scenario 3 - Both Vault and the app are running inside the mesh

1 - Scenario 1 - Vault runs outside, the application inside the mesh

In this scenario, Vault runs outside an Istio mesh, whereas the namespace where the application runs and the webhook injects secrets has Istio sidecar injection enabled.

First, complete the Prerequisites, then install Vault outside the mesh, and finally install an application within the mesh.

Install Vault outside the mesh

1. Provision a Vault instance with the Bank-Vaults operator in a separate namespace:

   kubectl create namespace vault
   

2. Apply the RBAC and CR files to the cluster to create a Vault instance in the vault namespace with the operator:

   kubectl apply -f rbac.yaml -f cr-istio.yaml
   

   kubectl get pods -n vault
   

   Expected output:

   NAME                               READY   STATUS    RESTARTS   AGE
   vault-0                            3/3     Running   0          22h
   vault-configurer-6458cc4bf-6tpkz   1/1     Running   0          22h
   

   If you are writing your own Vault CR make sure that istioEnabled: true is configured, this influences port naming so the Vault service port protocols are detected by Istio correctly.

3. The vault-secrets-webhook can’t inject Vault secrets into initContainers in an Istio-enabled namespace when the STRICT authentication policy is applied to the Vault service, because Istio needs a sidecar container to do mTLS properly, and in the phase when initContainers are running the Pod doesn’t have a sidecar yet. If you wish to inject into initContainers as well, you need to apply a PERMISSIVE authentication policy in the vault namespace, since it has its own TLS certificate outside of Istio scope (so this is safe to do from networking security point of view).

   kubectl apply -f - <<EOF
   apiVersion: authentication.istio.io/v1alpha1
   kind: Policy
   metadata:
     name: default
     namespace: vault
     labels:
       app: security
   spec:
     peers:
     - mtls:
         mode: PERMISSIVE
   EOF
   

Install the application inside a mesh

In this scenario Vault is running outside the Istio mesh (as we have installed it in the previous steps and our demo application runs within the Istio mesh. To install the demo application inside the mesh, complete the following steps:

1. Create a namespace first for the application and enable Istio sidecar injection:

   kubectl create namespace app
   kubectl label namespace app istio-injection=enabled
   

2. Install the application manifest to the cluster:

   kubectl apply -f app.yaml
   

3. Check that the application is up and running. It should have two containers, the app itself and the istio-proxy:

   kubectl get pods -n app
   

   Expected output:

   NAME                  READY   STATUS    RESTARTS   AGE
   app-5df5686c4-sl6dz   2/2     Running   0          119s
   

   kubectl logs -f -n app deployment/app app
   

   Expected output:

   time="2020-02-18T14:26:01Z" level=info msg="Received new Vault token"
   time="2020-02-18T14:26:01Z" level=info msg="Initial Vault token arrived"
   s3cr3t
   going to sleep...
   

2 - Scenario 2 - Running Vault inside the mesh

To run Vault inside the mesh, complete the following steps.

Note: These instructions assume that you have Scenario 1 up and running, and modifying it to run Vault inside the mesh.

1. Turn off Istio in the app namespace by removing the istio-injection label:

   kubectl label namespace app istio-injection-
   kubectl label namespace vault istio-injection=enabled
   

2. Delete the Vault pods in the vault namespace, so they will get recreated with the istio-proxy sidecar:

   kubectl delete pods --all -n vault
   

3. Check that they both come back with an extra container (4/4 and 2/2 now):

   kubectl get pods -n vault
   

   Expected output:

   NAME                                READY   STATUS    RESTARTS   AGE
   vault-0                             4/4     Running   0          1m
   vault-configurer-6d9b98c856-l4flc   2/2     Running   0          1m
   

4. Delete the application pods in the app namespace, so they will get recreated without the istio-proxy sidecar:

   kubectl delete pods --all -n app
   

The app pod got recreated with only the app container (1/1) and Vault access still works:

kubectl get pods -n app

Expected output:

NAME                  READY   STATUS    RESTARTS   AGE
app-5df5686c4-4n6r7   1/1     Running   0          71s

kubectl logs -f -n app deployment/app

Expected output:

time="2020-02-18T14:41:20Z" level=info msg="Received new Vault token"
time="2020-02-18T14:41:20Z" level=info msg="Initial Vault token arrived"
s3cr3t
going to sleep...

3 - Scenario 3 - Both Vault and the app are running inside the mesh

In this scenario, both Vault and the app are running inside the mesh.

1. Complete the Prerequisites.

2. Enable sidecar auto-injection for both namespaces:

   kubectl label namespace app   istio-injection=enabled
   kubectl label namespace vault istio-injection=enabled
   

3. Delete all pods so they are getting injected with the proxy:

   kubectl delete pods --all -n app
   kubectl delete pods --all -n vault
   

4. Check the logs in the app container. It should sill show success:

   kubectl logs -f -n app deployment/app
   

   Expected output:

   time="2020-02-18T15:04:03Z" level=info msg="Initial Vault token arrived"
   time="2020-02-18T15:04:03Z" level=info msg="Renewed Vault Token"
   s3cr3t
   going to sleep...