Secrets Provider - init container/sidecar - Push-to-File mode

This topic describes how to deploy the CyberArk Secrets Provider for Kubernetes as an init container or sidecar, cyberark-secrets-provider-for-k8s, in Push-to-File mode.

In Push-to-File mode, the Secrets Provider pushes Conjur secrets to one or more secrets files in a shared volume in the same Pod as your application container.

How it works

In Push-to-File mode, the Secrets Provider enables your application to consume Conjur secrets from secrets files in a shared volume in the same Pod as the application container.

Secret rotation and update detection

If secrets are updated or if secrets rotation is enabled, the Secrets Provider sidecar container updates the secrets in the secrets files.

After a specified interval (default or specified by conjur.org/secrets-refresh-interval annotation in the Secrets Provider manifest), the Secrets Provider checks if the secrets have changed by comparing the SHA-256 checksums of the secrets with the previous checksums. The Secrets Provider does not save any of the unencrypted secrets.

If the time needed to fetch the secrets is longer than is specified for the interval, then the internal will be the actual time to retrieve the secrets.

For example: if the duration is set to two seconds, but retrieving the secrets takes three seconds, then the secrets are updated every three seconds.

If a secret is deleted from Conjur or if access to the secret has been revoked, the Secrets Provider deletes the secrets file from the shared volume.

Set up Secrets Provider as an init container/sidecar in Push-to-File mode

This section describes how to set up the Secrets Provider for Kubernetes as an init container/sidecar in Push-to-File mode.

1. Before you begin

   • This procedure assumes you have a configured Kubernetes namespace with a service account for your application. The namespace must be prepared with the relevant resources, as described in Prepare the application namespace.

     It also assumes that you are familiar with loading manifests into your namespace.

     In this section, we use test-app-namespace for the namespace, and test-app-sa for the service account.

   • Make sure that a Kubernetes Authenticator endpoint has been configured and allowlisted. For more information, contact your Conjur admin, or see Certificate-based Kubernetes authentication.

2. Define the application as a Conjur host in policy

   Conjur admin: To enable the Secrets Provider for Kubernetes init container/sidecar (cyberark-secrets-provider-for-k8s) to retrieve Conjur secrets, it first needs to authenticate to Conjur.

   1. In this step, you define a Conjur host used to authenticate the cyberark-secrets-provider-for-k8s with the Kubernetes Authenticator.

      The Secrets Provider for Kubernetes must be defined by its namespace and authentication container name, and can also be defined by its service account. These definitions are defined in the host annotations in the policy. For guidelines on how to define annotations, see Workload identity for Kubernetes (cert-based authentication).

      The following policy:

      • defines a Conjur identity for test-app by its namespace, test-app-namespace, authentication container name, cyberark-secrets-provider-for-k8s, as well as by its service account, test-app-sa

      • gives test-app permissions to authenticate to Conjur using the dev-clusterKubernetes Authenticator

      Save the policy as apps.yml:

      - !host id: test-app annotations: authn-k8s/namespace: test-app-namespace authn-k8s/service-account: test-app-sa authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s - !grant roles: - !group conjur/authn-k8s/dev-cluster/consumers members: - !host test-app

      The value of the host's authn-k8s/authentication-container-name annotation states the container name from which it authenticates to Conjur. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container/sidecar has the same name.

   2. Load the policy file to root:

      $ conjur policy load -f apps.yml -b root

3. Grant access to secrets

   Conjur admin: In this step, you grant the Secrets Provider access to secrets. These secrets might be synchronized from the Vault, or defined in Conjur, or both.

   Secrets synchronized from the Vault

   If you have a Safe with an account that holds secrets for your application (see Accounts and Safes), grant permissions to access secrets from Vault/Privilege Cloud:

   1. Save the following policy as safe-access.yml.

      This policy adds test-app to the Vault/LOB_user/Safe's consumers group, giving the test-app permission to fetch all secrets from the (Vault/LOB_user/Safe) Safe in the Vault/Privilege Cloud.

      - !grant role: !group Vault/LOB_user/Safe/delegation/consumers member: !host test-app

   2. Load the policy file into root:

      $ conjur policy load -f safe-access.yml -b root

   Secrets in Conjur

   Define variables (secrets) that the Secrets Provider needs access to:

   1. Save the following policy as app-secrets.yml.

      This policy defines Conjur variables and a group that has permissions on the variables.

      In the following example, all members of the consumers group are granted permissions on the username and password variables:

      - !policy id: secrets body: - !group consumers - &variables - !variable username - !variable password - !permit role: !group consumers privilege: [ read, execute ] resource: *variables - !grant role: !group secrets/consumers member: !host test-app

   2. Load the policy file into root:

      $ conjur policy load -f app-secrets.yml -b root

   3. Populate the variables with secrets, for example myUser and MyP@ssw0rd!:

      $ conjur variable set -i secrets/username -v myUser

      $ conjur variable set -i secrets/password -v MyP@ssw0rd!

4. Set up the application deployment manifest

   Application developer: In this step you set up the application deployment manifest

   You set up an application deployment manifest that includes an application container, myorg/test-app, and an init container/sidecar that uses the cyberark/secrets-provider-for-k8s image.

   The deployment manifest also includes Pod annotations and environment variables to configure the Secrets Provider in Push-to-File mode. These direct the Secrets Provider to generate and write a secrets file containing YAML key/value settings into a volume that is shared with the application container.

   When you define the manifest, make sure to include the following:

   • Pod annotations, which must include:

     • conjur.org/authn-identity, , conjur.org/secrets-destination

     • conjur.org/container-mode: Value: init or sidecar

     • At least one secrets group with Conjur secret paths, for example conjur.org/secret-file-path.test-app

     • Sidecar only - To support secrets rotation:

       • conjur.org/secrets-refresh-enabled. Value: true

       • conjur.org/secrets-refresh-interval to define how often to check for updated secret values.

         This annotation depends on conjur.org/secrets-refresh-enabled being true. In this case, the interval defaults to 5 minutes unless otherwise defined.

     For details about these annotations as well as other optional annotations, see the Secrets Provider configuration reference.

   • Environment variables

     • The MY_POD_NAME and MY_POD_NAMESPACE environment variables which will be populated automatically by the Kubernetes Downward API upon the application's deployment.

     • The contents of the local conjur-connect ConfigMap which provides Conjur connection details from the Golden ConfigMap must be exposed as an environment variable.

   • Volume definitions

     • The init container/sidecar added to the Pod, with volume mounts.

     • Secrets volume mount added to the application container for the secrets file.

     The mountPath values used for the cyberark-secrets-provider-for-k8s container must appear exactly as shown in the manifest below:

     • /conjur/podinfo for the podinfo volume

     • /conjur/secrets for the conjur-secrets volume

     You can also include a volume mount to check the status of the Secrets Provider. For details, see Check Secrets Provider status using sentinel files.

   The following is an example of how to set up the application deployment manifest to push the Conjur secrets to a secrets file.

   When loaded to the application namespace, the Secrets Provider creates the credentials.yaml secrets file in /opt/secrets/conjur/, in the test-app Pod's conjur-secrets shared volume.

   • Sidecar
   • Init

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: test-app
     name: test-app
     namespace: test-app-namespace
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: test-app
     template:
       metadata:
         labels:
           app: test-app
         annotations:
           conjur.org/authn-identity: host/test-app
           conjur.org/container-mode: sidecar
           conjur.org/secrets-destination: file
           conjur.org/conjur-secrets-policy-path.test-app: secrets/
           conjur.org/conjur-secrets.test-app: |
             - admin-username: username
             - admin-password: password
           conjur.org/secret-file-path.test-app: "./credentials.yaml"
           conjur.org/secret-file-format.test-app: "yaml"
           conjur.org/secrets-refresh-enabled: "true"
           conjur.org/secrets-refresh-interval: 10m
       spec:
         serviceAccountName: test-app-sa
         containers:
         - name: test-app
           image: myorg/test-app
           volumeMounts:
             - name: conjur-secrets
               mountPath: /opt/secrets/conjur
               readOnly: true
         - name: cyberark-secrets-provider-for-k8s
           image: 'cyberark/secrets-provider-for-k8s:latest'
           imagePullPolicy: Always
           env:
           - name: MY_POD_NAME
             valueFrom:
               fieldRef:
                 fieldPath: metadata.name
           - name: MY_POD_NAMESPACE
             valueFrom:
               fieldRef:
                 fieldPath: metadata.namespace
           envFrom:
           - configMapRef:
               name: conjur-connect
           volumeMounts:
             - name: podinfo
               mountPath: /conjur/podinfo
             - name: conjur-secrets
               mountPath: /conjur/secrets
         volumes:
           - name: podinfo
             downwardAPI:
               items:
                 - path: "annotations"
                   fieldRef:
                     fieldPath: metadata.annotations
           - name: conjur-secrets
             emptyDir:
               medium: Memory

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: test-app
     name: test-app
     namespace: test-app-namespace
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: test-app
     template:
       metadata:
         labels:
           app: test-app
         annotations:
           conjur.org/authn-identity: host/test-app
           conjur.org/container-mode: init
           conjur.org/secrets-destination: file
           conjur.org/conjur-secrets.test-app: |
             - admin-username: username
             - admin-password: password
           conjur.org/conjur-secrets-policy-path.test-app: secrets/
           conjur.org/secret-file-path.test-app: "./credentials.yaml"
           conjur.org/secret-file-format.test-app: "yaml"
       spec:
         serviceAccountName: test-app-sa
         containers:
           - name: test-app
             image: myorg/test-app
             volumeMounts:
               - name: conjur-secrets
                 mountPath: /opt/secrets/conjur
                 readOnly: true
         initContainers:
           - name: cyberark-secrets-provider-for-k8s
             image: 'cyberark/secrets-provider-for-k8s:latest'
             imagePullPolicy: Always
             env:
               - name: MY_POD_NAME
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.name
               - name: MY_POD_NAMESPACE
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
             envFrom:
               - configMapRef:
                   name: conjur-connect
             volumeMounts:
               - name: conjur-certs
                 mountPath: /etc/conjur/ssl
               - name: podinfo
                 mountPath: /conjur/podinfo
               - name: conjur-secrets
                 mountPath: /conjur/secrets
         volumes:
           - name: conjur-certs
             emptyDir:
               medium: Memory
           - name: podinfo
             downwardAPI:
               items:
                 - path: "annotations"
                   fieldRef:
                     fieldPath: metadata.annotations
           - name: conjur-secrets
             emptyDir:
               medium: Memory

   The credentials.yaml secrets file contains the following secrets:

   "admin-username": "myUser"
   "admin-password": "myP@ssw0rd!"

Check Secrets Provider status using sentinel files

Secrets Provider allows for its status to be monitored through the creation of two empty sentinel files: CONJUR_SECRETS_PROVIDED and CONJUR_SECRETS_UPDATED.

The CONJUR_SECRETS_PROVIDED file is created when the Secrets Provider has completed its first round of providing secrets via secret files. It creates or recreates the CONJUR_SECRETS_UPDATED file whenever it has updated secret files. If needed, application containers can mount these files via a shared volume.

The Pod would need a Volume defined as follows:

 volumes:
    - name: conjur-status
      emptyDir:
        medium: Memory

The application container and Secrets Provider container would need to include volumeMounts as follows:

volumeMounts:
    - mountPath: /conjur/status
      name: conjur-status

The sentinel files delay the start of the application until after the Secrets Provider has started up and written the secrets. Kubelet starts the Pod containers in the order they are listed in the manifest. A postStart lifecycle hook can be added to the Secrets Provider manifest to delay the start of the application container until the postStart lifecycle hook is complete.

        lifecycle:
          postStart:
            exec:
              command:
              - /usr/local/bin/conjur-secrets-provided.sh

For an example of the Secrets Provider script, see conjur-secrets-provided.

If a livenessProbe is not already being used by the container as a health probe, you can use a livenessProbe as a "file watcher" and can cause the application to be restarted after the secrets have been updated. The livenessProbe could look as follows:

livenessProbe:
          exec:
            command:
            - /conjur/status/conjur-secrets-unchanged.sh
          failureThreshold: 1
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1

Secrets files deletion after consumption