Secrets Provider - init container/sidecar - Kubernetes Secrets mode
This topic describes how to set up the CyberArk Secrets Provider for Kubernetes as an init container or sidecar,
cyberark-secrets-provider-for-k8s, to populate Kubernetes Secrets with secrets stored in Conjur.
As an init container/sidecar, the Secrets Provider for Kubernetes is deployed in the same Pod as the application container, and serves only that application.
As a sidecar, secrets rotation and updates are supported.
How it works
This section shows how the Secrets Provider works.
cyberark-secrets-provider-for-k8ssidecar/init container, starts and authenticates to Conjur using the
Kubernetes Authenticator (.
cyberark-secrets-provider-for-k8ssidecar/init container reads all Kubernetes Secrets required by the Pod.
For each mapped Kubernetes Secret, the
Fetches the Conjur secret
Updates the Kubernetes Secret with the Conjur secret value
If the Secrets Provider is an init container, it runs to completion.
If the Secrets Provider is a sidecar and is configured to refresh secrets, after a specified interval the Secrets Provider checks if there are any updates to the secrets in Conjur. If changes are detected, the Secrets Provider sidecar updates the secrets files in Kubernetes Secrets.
Moreover, the application can optionally delete secret files after consuming them. In this case, if changes are detected in the secrets in Conjur, the Secrets Provider sidecar recreates and rewrites the secret files to Kubernetes Secrets.
The application container starts and consumes the Kubernetes Secrets.
Secret rotation and update detection
Supported by the Secrets Provider sidecar only.
If secrets are updated or if secrets rotation is enabled, the Secrets Provider sidecar container updates the secrets in the
After a specified interval (default or specified by
conjur.org/secrets-refresh-interval annotation in the Secrets Provider manifest), the Secrets Provider checks if the secrets have changed by comparing the SHA-256 checksums of the secrets with the previous checksums. The Secrets Provider does not save any of the unencrypted secrets.
If the time needed to fetch the secrets is longer than is specified for the interval, then the internal will be the actual time to retrieve the secrets.
For example: if the duration is set to two seconds, but retrieving the secrets takes three seconds, then the secrets are updated every three seconds.
If a secret is deleted from Conjur or if access to the secret has been revoked, the Secrets Provider deletes the
Set up Secrets Provider as an init container/sidecar
This section describes how to set up the Secrets Provider for Kubernetes as an init container/sidecar.
Before you begin
This procedure assumes you have a configured Kubernetes namespace with a service account for your application. The namespace must be prepared with the relevant resources, as described in Prepare the application namespace.
It also assumes that you are familiar with loading manifests into your namespace.
In this section, we use
test-app-namespacefor the namespace, and
test-app-safor the service account.
Make sure that a Kubernetes Authenticator endpoint has been configured and allowlisted. For more information, contact your Conjur admin, or see Certificate-based Kubernetes authentication.
Define the application as a Conjur host in policy
Conjur admin: To enable the Secrets Provider for Kubernetes (
cyberark-secrets-provider-for-k8s) to retrieve Conjur secrets, it first needs to authenticate to Conjur.
In this step, you define a Conjur host used to authenticate the
cyberark-secrets-provider-for-k8scontainer with the Kubernetes Authenticator.
The Secrets Provider for Kubernetes must be defined by its namespace and authentication container name, and can also be defined by its service account. These definitions are defined in the host annotations in the policy. For guidelines on how to define annotations, see Workload identity for Kubernetes (cert-based authentication).
The following policy:
defines a Conjur identity for
test-appby its namespace,
test-app-namespace, authentication container name,
cyberark-secrets-provider-for-k8s, as well as by its service account,
test-apppermissions to authenticate to Conjur using the
- !host id: test-app annotations: authn-k8s/namespace: test-app-namespace authn-k8s/service-account: test-app-sa authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s - !grant roles: - !group conjur/authn-k8s/dev-cluster/consumers members: - !host test-app
The value of the host's
authn-k8s/authentication-container-nameannotation states the container name from which it authenticates to Conjur. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container/sidecar has the same name.
Load the policy file to root:
conjur policy load -f apps.yml -b root
Grant access to secrets
Conjur admin: In this step, you grant the Secrets Provider access to secrets. These secrets might be synchronized from the Vault, or defined in Conjur, or both.Secrets synchronized from the Vault
If you have a Safe with an account that holds secrets for your application (see Accounts and Safes), grant permissions to access secrets from Vault/Privilege Cloud:
Secrets in Conjur
Save the following policy as safe-access.yml.
This policy adds
Vault/LOB_user/Safe's consumers group, giving the
test-apppermission to fetch all secrets from the (
Vault/LOB_user/Safe) Safe in the Vault/Privilege Cloud.
- !grant role: !group Vault/LOB_user/Safe/delegation/consumers member: !host test-app
Load the policy file into root:
conjur policy load -f safe-access.yml -b root
Define variables (secrets) that the Secrets Provider needs access to:
Save the following policy as app-secrets.yml.
This policy defines Conjur variables and a group that has permissions on the variables.
In the following example, all members of the
consumersgroup are granted permissions on the
- !policy id: secrets body: - !group consumers - &variables - !variable username - !variable password - !permit role: !group consumers privilege: [ read, execute ] resource: *variables - !grant role: !group secrets/consumers member: !host test-app
Load the policy file into root:
conjur policy load -f app-secrets.yml -b root
Populate the variables with secrets, for example
conjur variable set -i secrets/username -v myUser
conjur variable set -i secrets/password -v MyP@ssw0rd!
Map Conjur variables to Kubernetes Secrets
Application developer: To populate variables (secrets) from Conjur to Kubernetes Secrets, modify the yaml file that defines the Kubernetes Secret.
To create the mapping, update each Kubernetes Secret to include a
conjur-map, for example:
--- apiVersion: v1 kind: Secret metadata: name: db-credentials type: Opaque data: DBName: bXlhcHBEQg== stringData: conjur-map: |- username: secrets/username password: secrets/password
When loaded, "stringData" values are encoded.
Create and bind a role to the application service account
Application developer: In this step, you create a role,
secrets-accessand bind it to the
test-app-saservice account. This allows the Secrets Provider for Kubernetes to update Kubernetes Secrets with values from Conjur variables (secrets).
Copy the following manifest and load it into your application's namespace,
--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: secrets-access namespace: test-app-namespace rules: - apiGroups: [""] resources: ["secrets"] verbs: [ "get", "update" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: test-app-namespace name: secrets-access-binding subjects: - kind: ServiceAccount namespace: test-app-namespace name: test-app-sa roleRef: kind: Role apiGroup: rbac.authorization.k8s.io name: secrets-access
Set up the application deployment manifest
Application developer: In this step you set up the application deployment manifest using the ConfigMap you created in the previous step. The manifest defines a deployment which includes an application container,
myorg/test-app, and an init container or sidecar of the
Copy the relevant manifest, under the
envsection fill in the required environment variables, and load the manifest to the application namespace,
test-app-namespace. For details about the required environment variables, see the Secrets Provider configuration reference.
apiVersion: apps/v1 kind: Deployment metadata: labels: app: test-app-secrets-provider-k8s name: test-app-secrets-provider-k8s namespace: test-app-namespace spec: selector: matchLabels: app: test-app-secrets-provider-k8s replicas: 1 template: metadata: annotations: conjur.org/container-mode: sidecar conjur.org/secrets-refresh-enabled: "true" conjur.org/secrets-refresh-interval: 10s labels: app: test-app-secrets-provider-k8s spec: serviceAccountName: test-app-sa containers: - name: test-app image: myorg/test-app command: [ "sleep" ] args: [ "infinity" ] env: - name: DB_USERNAME valueFrom: secretKeyRef: name: test-app-secrets-provider-k8s-secret key: DB_USERNAME - name: DB_PASSWORD valueFrom: secretKeyRef: name: test-app-secrets-provider-k8s-secret key: DB_PASSWORD - image: cyberark/secrets-provider-for-k8s imagePullPolicy: Always name: cyberark-secrets-provider-for-k8s env: - name: CONJUR_AUTHN_LOGIN value: host/conjur/authn-k8s/my-authenticator-id/apps/test-app-secrets-provider-k8s - name: CONTAINER_MODE value: sidecar - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: K8S_SECRETS value: test-app-secrets-provider-k8s-secret - name: SECRETS_DESTINATION value: k8s_secrets envFrom: - configMapRef: name: conjur-connect volumeMounts: - mountPath: /run/conjur name: conjur-access-token - mountPath: /etc/conjur/ssl name: conjur-certs - mountPath: /conjur/podinfo name: podinfo volumes: - emptyDir: medium: Memory name: conjur-access-token - emptyDir: medium: Memory name: conjur-certs - downwardAPI: defaultMode: 420 items: - fieldRef: apiVersion: v1 fieldPath: metadata.annotations path: annotations name: podinfo
apiVersion: apps/v1 kind: Deployment metadata: labels: app: test-app-secrets-provider-k8s name: test-app-secrets-provider-k8s namespace: test-app-namespace spec: selector: matchLabels: app: test-app-secrets-provider-k8s replicas: 1 template: metadata: labels: app: test-app-secrets-provider-k8s spec: serviceAccountName: test-app-sa containers: - name: test-app image: myorg/test-app command: [ "sleep" ] args: [ "infinity" ] env: - name: DB_USERNAME valueFrom: secretKeyRef: name: test-app-secrets-provider-k8s-secret key: DB_USERNAME - name: DB_PASSWORD valueFrom: secretKeyRef: name: test-app-secrets-provider-k8s-secret key: DB_PASSWORD initContainers: - image: cyberark/secrets-provider-for-k8s imagePullPolicy: Always name: cyberark-secrets-provider-for-k8s env: - name: CONJUR_AUTHN_LOGIN value: host/conjur/authn-k8s/my-authenticator-id/apps/test-app-secrets-provider-k8s - name: CONTAINER_MODE value: init - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: K8S_SECRETS value: test-app-secrets-provider-k8s-secret - name: SECRETS_DESTINATION value: k8s_secrets envFrom: - configMapRef: name: conjur-connect
If the secrets stored in Conjur are Base64-encoded strings, Secrets Provider can decode the secrets before providing them to the application. You configure Secrets Provider to decode these strings using annotations
To add Base64 decoding to a Kubernetes Secret, modify the
conjur-map by adding an
content-type: base64 to the string to decode.
For example, the following Kubernetes Secret configuration doesn't use Base64 decoding.
apiVersion: v1 kind: Secret metadata: name: test-app-secrets-provider-k8s-secret type: Opaque stringData: conjur-map: |- DB_URL: test-secrets-provider-k8s-app-db/url DB_USERNAME: test-secrets-provider-k8s-app-db/username DB_PASSWORD: test-secrets-provider-k8s-app-db/password
In the following example, Base64 decoding is enabled for the
DB_PASSWORD value using the
content-type: base64 annotations.
apiVersion: v1 kind: Secret metadata: name: test-app-secrets-provider-k8s-secret type: Opaque stringData: conjur-map: |- DB_URL: test-secrets-provider-k8s-app-db/url DB_USERNAME: test-secrets-provider-k8s-app-db/username DB_PASSWORD: id: test-secrets-provider-k8s-app-db/password content-type: base64
If the secret can't be decoded, a warning is added to the log files.
Supports Secrets Provider v1.4.1 and later.
Secrets Provider allows for its status to be monitored through the creation of two empty sentinel files: CONJUR_SECRETS_PROVIDED and CONJUR_SECRETS_UPDATED.
The CONJUR_SECRETS_PROVIDED file is created when the Secrets Provider has completed its first round of providing secrets via
The Pod would need a Volume defined as follows:
volumes: - name: conjur-status emptyDir: medium: Memory
The application container and Secrets Provider container would need to include
volumeMounts as follows:
volumeMounts: - mountPath: /conjur/status name: conjur-status
The sentinel files delay the start of the application until after the Secrets Provider has started up and written the secrets. Kubelet starts the Pod containers in the order they are listed in the manifest. A
postStart lifecycle hook can be added to the Secrets Provider manifest to delay the start of the application container until the
postStart lifecycle hook is complete.
lifecycle: postStart: exec: command: - /usr/local/bin/conjur-secrets-provided.sh
For an example of the Secrets Provider script, see conjur-secrets-provided.
If a livenessProbe is not already being used by the container as a health probe, you can use a livenessProbe as a "file watcher" and can cause the application to be restarted after the secrets have been updated. The livenessProbe could look as follows:
livenessProbe: exec: command: - /conjur/status/conjur-secrets-unchanged.sh failureThreshold: 1 initialDelaySeconds: 5 periodSeconds: 5 successThreshold: 1 timeoutSeconds: 1