Secrets Provider - init container/sidecar - Kubernetes Secrets mode

This topic describes how to set up the CyberArk Secrets Provider for Kubernetes as an init container or sidecar, cyberark-secrets-provider-for-k8s, to populate Kubernetes Secrets with secrets stored in Conjur.

As an init container/sidecar, the Secrets Provider for Kubernetes is deployed in the same Pod as the application container, and serves only that application.

As a sidecar, secrets rotation and updates are supported.

How it works

This section shows how the Secrets Provider works.

The cyberark-secrets-provider-for-k8s sidecar/init container, starts and authenticates to Conjur using the Kubernetes Authenticator (authn-k8s).
The cyberark-secrets-provider-for-k8s sidecar/init container reads all Kubernetes Secrets required by the Pod.
For each mapped Kubernetes Secret, the cyberark-secrets-provider-for-k8s sidecar/init container:
1. Fetches the Conjur secret
2. Updates the Kubernetes Secret with the Conjur secret value
If the Secrets Provider is an init container, it runs to completion.

If the Secrets Provider is a sidecar and is configured to refresh secrets, after a specified interval the Secrets Provider checks if there are any updates to the secrets in Conjur. If changes are detected, the Secrets Provider sidecar updates the secrets files in Kubernetes Secrets.

Moreover, the application can optionally delete secret files after consuming them. In this case, if changes are detected in the secrets in Conjur, the Secrets Provider sidecar recreates and rewrites the secret files to Kubernetes Secrets.
The application container starts and consumes the Kubernetes Secrets.

Secret rotation and update detection

Supported by the Secrets Provider sidecar only.

If secrets are updated or if secrets rotation is enabled, the Secrets Provider sidecar container updates the secrets in the Kubernetes Secrets.

After a specified interval (default or specified by conjur.org/secrets-refresh-interval annotation in the Secrets Provider manifest), the Secrets Provider checks if the secrets have changed by comparing the SHA-256 checksums of the secrets with the previous checksums. The Secrets Provider does not save any of the unencrypted secrets.

If the time needed to fetch the secrets is longer than is specified for the interval, then the internal will be the actual time to retrieve the secrets.

For example: if the duration is set to two seconds, but retrieving the secrets takes three seconds, then the secrets are updated every three seconds.

If a secret is deleted from Conjur or if access to the secret has been revoked, the Secrets Provider deletes the secret from Kubernetes Secrets.

Set up Secrets Provider as an init container/sidecar

This section describes how to set up the Secrets Provider for Kubernetes as an init container/sidecar.

Before you begin
- This procedure assumes you have a configured Kubernetes namespace with a service account for your application. The namespace must be prepared with the relevant resources, as described in Prepare the application namespace.
  
  It also assumes that you are familiar with loading manifests into your namespace.
  
  In this section, we use test-app-namespace for the namespace, and test-app-sa for the service account.
- Make sure that a Kubernetes Authenticator endpoint has been configured and allowlisted. For more information, contact your Conjur admin, or see Certificate-based Kubernetes authentication.

Define the application as a Conjur host in policy

Conjur admin: To enable the Secrets Provider for Kubernetes (cyberark-secrets-provider-for-k8s) to retrieve Conjur secrets, it first needs to authenticate to Conjur.

In this step, you define a Conjur host used to authenticate the cyberark-secrets-provider-for-k8s container with the Kubernetes Authenticator.

The Secrets Provider for Kubernetes must be defined by its namespace and authentication container name, and can also be defined by its service account. These definitions are defined in the host annotations in the policy. For guidelines on how to define annotations, see Workload identity for Kubernetes (cert-based authentication).

The following policy:

defines a Conjur identity for test-app by its namespace, test-app-namespace, authentication container name, cyberark-secrets-provider-for-k8s, as well as by its service account, test-app-sa

gives test-app permissions to authenticate to Conjur using the dev-clusterKubernetes Authenticator

Save the policy as apps.yml:

- !host
  id: test-app
  annotations:
    authn-k8s/namespace: test-app-namespace
    authn-k8s/service-account: test-app-sa
    authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s
   
- !grant
  roles:
  - !group conjur/authn-k8s/dev-cluster/consumers
  members:
  - !host test-app

The value of the host's authn-k8s/authentication-container-name annotation states the container name from which it authenticates to Conjur. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container/sidecar has the same name.

Load the policy file to root:

$ conjur policy load -f apps.yml -b root

Grant access to secrets

Conjur admin: In this step, you grant the Secrets Provider access to secrets. These secrets might be synchronized from the Vault, or defined in Conjur, or both.

Secrets synchronized from the Vault

If you have a Safe with an account that holds secrets for your application (see Accounts and Safes), grant permissions to access secrets from Vault/Privilege Cloud:

Save the following policy as safe-access.yml.

This policy adds test-app to the Vault/LOB_user/Safe's consumers group, giving the test-app permission to fetch all secrets from the (Vault/LOB_user/Safe) Safe in the Vault/Privilege Cloud.

- !grant
  role: !group Vault/LOB_user/Safe/delegation/consumers
  member: !host test-app

Load the policy file into root:

$ conjur policy load -f safe-access.yml -b root

Secrets in Conjur

Define variables (secrets) that the Secrets Provider needs access to:

Save the following policy as app-secrets.yml.

This policy defines Conjur variables and a group that has permissions on the variables.

In the following example, all members of the consumers group are granted permissions on the username and password variables:

- !policy
  id: secrets
  body:
    - !group consumers
    - &variables
      - !variable username
      - !variable password
    - !permit
      role: !group consumers
      privilege: [ read, execute ]
      resource: *variables
- !grant
  role: !group secrets/consumers
  member: !host test-app

Load the policy file into root:

$ conjur policy load -f app-secrets.yml -b root

Populate the variables with secrets, for example myUser and MyP@ssw0rd!:

$ conjur variable set -i secrets/username -v myUser

$ conjur variable set -i secrets/password -v MyP@ssw0rd!

Map Conjur variables to Kubernetes Secrets

Application developer: To populate variables (secrets) from Conjur to Kubernetes Secrets, modify the yaml file that defines the Kubernetes Secret.

Add a conjur-map data entry to specify the mapping between the Kubernetes Secrets data key and the path to the variable in Conjur.

To create the mapping, update each Kubernetes Secret to include a conjur-map, for example:

---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
stringData:
  conjur-map: |-   
    username: secrets/username
    password: secrets/password

When loaded, "stringData" values are encoded.

Create and bind a role to the application service account

Application developer: In this step, you create a role, secrets-access and bind it to the test-app-sa service account. This allows the Secrets Provider for Kubernetes to update Kubernetes Secrets with values from Conjur variables (secrets).

Copy the following manifest and load it into your application's namespace, test-app-namespace:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secrets-access
  namespace: test-app-namespace
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: [ "get", "update" ]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: test-app-namespace
  name: secrets-access-binding
subjects:
  - kind: ServiceAccount
    namespace: test-app-namespace
    name: test-app-sa
roleRef:
  kind: Role
  apiGroup: rbac.authorization.k8s.io
  name: secrets-access

Set up the application deployment manifest

Application developer: In this step you set up the application deployment manifest using the ConfigMap you created in the previous step. The manifest defines a deployment which includes an application container, myorg/test-app, and an init container or sidecar of the cyberark/secrets-provider-for-k8s image.

Copy the relevant manifest, under the env section fill in the required environment variables, and load the manifest to the application namespace, test-app-namespace. For details about the required environment variables, see the Secrets Provider configuration reference.

Sidecar
Init container

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-app-secrets-provider-k8s
  name: test-app-secrets-provider-k8s
  namespace: test-app-namespace
spec:
  selector:
    matchLabels:
      app: test-app-secrets-provider-k8s
  replicas: 1
  template:
    metadata:
      annotations:
        conjur.org/container-mode: sidecar
        conjur.org/secrets-refresh-enabled: "true"
        conjur.org/secrets-refresh-interval: 10s
      labels:
        app: test-app-secrets-provider-k8s
    spec:
      serviceAccountName: test-app-sa
      containers:
      - name: test-app
        image: myorg/test-app
        command: [ "sleep" ]
        args: [ "infinity" ]
        env:
          - name: DB_USERNAME
            valueFrom:
              secretKeyRef:
                name: test-app-secrets-provider-k8s-secret
                key: DB_USERNAME
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: test-app-secrets-provider-k8s-secret
                key: DB_PASSWORD
      - image: cyberark/secrets-provider-for-k8s
        imagePullPolicy: Always
        name: cyberark-secrets-provider-for-k8s
        env:
          - name: CONJUR_AUTHN_LOGIN
            value: host/conjur/authn-k8s/my-authenticator-id/apps/test-app-secrets-provider-k8s
          - name: CONTAINER_MODE
            value: sidecar
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: K8S_SECRETS
            value: test-app-secrets-provider-k8s-secret
          - name: SECRETS_DESTINATION
            value: k8s_secrets
        envFrom:
          - configMapRef:
              name: conjur-connect
        volumeMounts:
        - mountPath: /run/conjur
          name: conjur-access-token
        - mountPath: /etc/conjur/ssl
          name: conjur-certs
        - mountPath: /conjur/podinfo
          name: podinfo
      volumes:
      - emptyDir:
          medium: Memory
        name: conjur-access-token
      - emptyDir:
          medium: Memory
        name: conjur-certs
      - downwardAPI:
          defaultMode: 420
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations
            path: annotations
        name: podinfo

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test-app-secrets-provider-k8s
  name: test-app-secrets-provider-k8s
  namespace: test-app-namespace
spec:
  selector:
    matchLabels:
      app: test-app-secrets-provider-k8s
  replicas: 1
  template:
    metadata:
      labels:
        app: test-app-secrets-provider-k8s
    spec:
      serviceAccountName: test-app-sa
      containers:
      - name: test-app
        image: myorg/test-app
        command: [ "sleep" ]
        args: [ "infinity" ]
        env:
          - name: DB_USERNAME
            valueFrom:
              secretKeyRef:
                name: test-app-secrets-provider-k8s-secret
                key: DB_USERNAME
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: test-app-secrets-provider-k8s-secret
                key: DB_PASSWORD
      initContainers:
      - image: cyberark/secrets-provider-for-k8s
        imagePullPolicy: Always
        name: cyberark-secrets-provider-for-k8s
        env:
          - name: CONJUR_AUTHN_LOGIN
            value: host/conjur/authn-k8s/my-authenticator-id/apps/test-app-secrets-provider-k8s
          - name: CONTAINER_MODE
            value: init
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: K8S_SECRETS
            value: test-app-secrets-provider-k8s-secret
          - name: SECRETS_DESTINATION
            value: k8s_secrets
        envFrom:
          - configMapRef:
              name: conjur-connect

Decode Base64-encoded secrets

If the secrets stored in Conjur are Base64-encoded strings, Secrets Provider can decode the secrets before providing them to the application. You configure Secrets Provider to decode these strings using annotations and configuring the Kubernetes Secret with a conjur-map.

To add Base64 decoding to a Kubernetes Secret, modify the conjur-map by adding an id: and content-type: base64 to the string to decode.

For example, the following Kubernetes Secret configuration doesn't use Base64 decoding.

apiVersion: v1
kind: Secret
metadata:
  name: test-app-secrets-provider-k8s-secret
type: Opaque
stringData:
  conjur-map: |-
    DB_URL: test-secrets-provider-k8s-app-db/url
    DB_USERNAME: test-secrets-provider-k8s-app-db/username
    DB_PASSWORD: test-secrets-provider-k8s-app-db/password

In the following example, Base64 decoding is enabled for the DB_PASSWORD value using the id: and content-type: base64 annotations.

apiVersion: v1
kind: Secret
metadata:
  name: test-app-secrets-provider-k8s-secret
type: Opaque
stringData:
  conjur-map: |-
    DB_URL: test-secrets-provider-k8s-app-db/url
    DB_USERNAME: test-secrets-provider-k8s-app-db/username
    DB_PASSWORD:
      id: test-secrets-provider-k8s-app-db/password
      content-type: base64

If the secret can't be decoded, a warning is added to the log files.

Secrets Provider only decodes secrets in Base64 format.

Check Secrets Provider status using sentinel files

Supports Secrets Provider v1.4.1 and later.

Secrets Provider allows for its status to be monitored through the creation of two empty sentinel files: CONJUR_SECRETS_PROVIDED and CONJUR_SECRETS_UPDATED.

The CONJUR_SECRETS_PROVIDED file is created when the Secrets Provider has completed its first round of providing secrets via Kubernetes Secrets. It creates or recreates the CONJUR_SECRETS_UPDATED file whenever it has updated Kubernetes Secrets. If needed, application containers can mount these files via a shared volume.

The Pod would need a Volume defined as follows:

 volumes:
    - name: conjur-status
      emptyDir:
        medium: Memory

The application container and Secrets Provider container would need to include volumeMounts as follows:

volumeMounts:
    - mountPath: /conjur/status
      name: conjur-status

The sentinel files delay the start of the application until after the Secrets Provider has started up and written the secrets. Kubelet starts the Pod containers in the order they are listed in the manifest. A postStart lifecycle hook can be added to the Secrets Provider manifest to delay the start of the application container until the postStart lifecycle hook is complete.

        lifecycle:
          postStart:
            exec:
              command:
              - /usr/local/bin/conjur-secrets-provided.sh

For an example of the Secrets Provider script, see conjur-secrets-provided.

If a livenessProbe is not already being used by the container as a health probe, you can use a livenessProbe as a "file watcher" and can cause the application to be restarted after the secrets have been updated. The livenessProbe could look as follows:

livenessProbe:
          exec:
            command:
            - /conjur/status/conjur-secrets-unchanged.sh
          failureThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1