CyberArk Secrets Provider for Kubernetes

This topic describes how to populate Kubernetes secrets with secrets stored in DAP.

How it works

The CyberArk Secrets Provider for Kubernetes enables applications to use secrets stored and managed in the CyberArk Vault using DAP and consume them as Kubernetes secrets in your application containers. To do so, the CyberArk Secrets Provider for Kubernetes image (cyberark/secrets-provider-for-k8s) runs as an init container (cyberark-secrets-provider-for-k8s) and populates the Kubernetes secrets required by the pod, from DAP.

  1. The cyberark-secrets-provider-for-k8s init container starts and authenticates with the DAP Follower using the Kubernetes Authenticator (authn-k8s).​

  2. The cyberark-secrets-provider-for-k8s init container reads all Kubernetes secrets required by the pod.

  3. For each mapped Kubernetes secret, the cyberark-secrets-provider-for-k8s init container retrieves DAP secrets and updates the Kubernetes secret with the DAP secret value.

  4. The cyberark-secrets-provider-for-k8s init container runs to completion.

  5. The application container consumes the Kubernetes secrets and starts.

Example: Kubernetes secret mapped to DAP secrets

The following example demonstrates why you would update a Kubernetes secret with secrets from DAP, and how to do this.

Let's say you have a Kubernetes secret, db-credentials, defined in a yaml file as follows:

 
# mysecrets.yml
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
  username: dGhlLXVzZXItbmFtZQ==
  password: dGhlLXBhc3N3b3Jk

You determine that the username and password values are sensitive and should be stored in DAP. You decide that the DBName value is not as sensitive, and can remain in Kubernetes.

In the following policy, the DAP admin defines the username and password values as DAP variables:

 
- !policy
  id: db_credentials
  body:
  - !variable username
  - !variable password

and loads the values to DAP:

 
conjur variable values add db_credentials/username the-user-name
conjur variable values add db_credentials/password the-password

To consume these secrets as Kubernetes secrets, you update the original yaml file as follows:

 
# mysecrets.yml
---
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  DBName:   bXlhcHBEQg==
stringData:
  conjur-map: |-   
    username: db_credentials/username   
    password: db_credentials/password
 

When loaded, "stringData" values are encoded.

Finally, you update the secret in Kubernetes:

 
# Kubernetes
kubectl apply -f mysecrets.yml
 
# OpenShift
oc apply -f mysecrets.yml

The Kubernetes secret can now be used to retrieve the secrets from DAP using the CyberArk Secrets Provider for Kubernetes.

Step 1: Configure DAP resources

To enable the CyberArk Secrets Provider for Kubernetes to retrieve DAP secrets, it needs to authenticate to DAP.

To do so, you need to assign the CyberArk Secrets Provider for Kubernetes a DAP host identity that authenticates against the Kubernetes Authenticator:

  1. Prerequisite: Verify that a Kubernetes Authenticator has been enabled

    For details, see Enable Authenticators for Applications.

  2. Enable DAP hosts to authenticate using the Kubernetes Authenticator

    In this step, define a DAP host used to authenticate the cyberark-secrets-provider-for-k8s container with the Kubernetes Authenticator:

    1. Use the following policy to define a DAP host. Remember to specify the Kubernetes namespace (APP_NAMESPACE_NAME) where you are deploying your app.

      The service provider host should follow Application Identity in OpenShift/Kubernetes guidelines.

      For more details, see Policy for authentication identities.

      For example, to control authentication and secret access by namespace < TEST_APP_NAMESPACE_NAME >, create the following policy:

       
      - !policy
        id: apps
        body:
          - !layer
          - &hosts
            - !host
              id: namespace-based-app
              annotations:
                authn-k8s/namespace: < TEST_APP_NAMESPACE_NAME >
                authn-k8s/authentication-container-name: cyberark-secrets-provider-for-k8s
          - !grant
            role: !layer
            members: *hosts
       

      The value of the authn-k8s/authentication-container-name annotation of the host states the container name from which it authenticates with DAP. When you create the application deployment manifest below, verify that the CyberArk Secrets Provider for Kubernetes init container has the same name.

    2. Save the policy as apps.yml.

    3. Load the policy file to root or to any other branch where you want the policy to reside.

      For example, if you are loading the policy to conjur/authn-k8s/<authenticator-id>, run:

       
      conjur policy load conjur/authn-k8s/<authenticator-id> apps.yml
     

    For the rest of this topic, all the examples refer to the host, conjur/authn-k8s/<authenticator-id>/apps/namespace-based-app.

  3. Define variables to hold the secrets for your application

    This section describes how to define the secrets that the cyberark-secrets-provider-for-k8s container needs access to.

    1. Use the following sample policy to define the database credentials as DAP variables and grant access to these credentials to the host defined in the previous step.

       
      - !policy
        id: db_credentials
        body:
        - &variables
            - !variable password
            - !variable url
            - !variable username
       
      - !permit
        resources: *variables
        role: !layer conjur/authn-k8s/<authenticator-id>/apps
        privileges: [ read, execute ]
    2. Save the policy as a yaml file using the following file naming convention: db-credentials.yml

    3. Load the policy file into root:

       
      conjur policy load root db-credentials.yml

Step 2: Map Kubernetes secrets to DAP secrets

To populate secrets from DAP to Kubernetes secrets, create or modify the yaml file that defines the Kubernetes secrets.

Create the mapping

Add a conjur-map data entry to specify the mapping between the Kubernetes secret data key and the secret in DAP.

To create the mapping:

  1. Edit each Kubernetes secret to include a conjur-map entry for each key stored in DAP. For example:

     
    # authn-k8s-<authenticator_id>-secrets.yml
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: db-credentials
    type: Opaque
    data:
      DBName:   bXlhcHBEQg==
    stringData:
      conjur-map: |-   
        username: db_credentials/username   
        password: db_credentials/password
  2. Update the secret in Kubernetes:

     
    # Kubernetes
    kubectl apply -f authn-k8s-<authenticator_id>-secrets.yml
     
    # OpenShift
    oc apply -f authn-k8s-<authenticator_id>-secrets.yml

Step 3: Configure RBAC permissions

Create a ServiceAccount, and bind a Role or a ClusterRole to that ServiceAccount to grant privileges to it. These permissions enable the CyberArk Secrets Provider for Kubernetes to update Kubernetes secrets with DAP secrets.

The following manifest defines a ServiceAccount. The ServiceAccount is bound to the role secrets-access that defines the RBAC permissions on the Kubernetes secrets resource:

To have the DAP ServiceAccount authenticate your application, you also need to add permissions. For details, see Kubernetes RBAC Permissions.

Load this manifest by running the following:

 
# Kubernetes
kubectl apply -f conjur-role.yml
  
  
# OpenShift
oc apply -f conjur-role.yml

Step 4: Add the CyberArk Secrets Provider for Kubernetes image to the application deployment manifest

The CyberArk Secrets Provider for Kubernetes is a public image available on DockerHub or on Red Hat.

  1. Add the CyberArk Secrets Provider for Kubernetes image as an init container to the application deployment manifest. The manifest defines a deployment which includes an application container, test-app, and an init container of the cyberark/secrets-provider-for-k8s image.

    Configure the CyberArk Secrets Provider for Kubernetes with the following variables:

     

    The value of serviceAccountName should be identical to the name of the ServiceAccount you defined in the Step 3: Configure RBAC permissions section.

    Parameter

    Description

    CONTAINER_MODE

    Specify how the container should run.

    Supported value: init

    MY_POD_NAME

    This variable identifies the pod where your application runs. Use the Downward API to insert this information into the manifest. In the example above, this value is auto-populated from the app container info.

    MY_POD_NAMESPACE

    This variable identifies the namespace where your application runs. Use the Downward API to insert this information into the manifest. In the example above, this value is auto-populated from the app container info.

    CONJUR_AUTHN_URL

    Specify the credential service used to log into DAP.

    Syntax:

     
    https://dap-follower.<CONJUR_NAMESPACE_NAME>.svc.cluster.local/authn-k8s/<AUTHENTICATOR_ID>

    CONJUR_APPLIANCE_URL

    The URL of the service defined for the DAP Follower.

    Syntax:

     
    https://< CONJUR_SERVICE_NAME >.< CONJUR_NAMESPACE >.svc/api

    CONJUR_ACCOUNT

    The account name designated to the DAP Follower during initial configuration.

    CONJUR_AUTHN_LOGIN

    Specify the DAP host that authenticates to DAP. Set this value to a host id that you defined in the policy above.

    For more information, see Application Identity in OpenShift/Kubernetes.

    CONJUR_SSL_CERTIFICATE

    The public SSL certificate value required for connecting to the DAP Follower.

    The SSL certificate is generated during DAP Follower configuration and stored in a .pem file located in the root folder where the DAP Follower was created. The file name is conjur-account.pem, where account is the account name provided for the DAP Follower.

    We recommend using a ConfigMap to store this value.

    Example

    • Kubernetes:

       
      kubectl create configmap conjur-cert --from-file=ssl-certificate="/path/to/ssl/cert"
    • OpenShift:

       
      oc create configmap conjur-cert --from-file=ssl-certificate="/path/to/ssl/cert"
      

    The above commands create a ConfigMap that loads the certificate value into it. Once the certificate value is populated, you can add the following configuration to the manifest:

     
    configMapKeyRef:
        name: conjur-cert        
        key: ssl_certificate

    DEBUG

    Enable logs.

    Supported value: true

     

    Do not include this parameter in a production environment.

    SECRETS_DESTINATION

    Required destination for the DAP secrets, to be retrieved by the application container.

    Supported values: k8s_secrets

    K8S_SECRETS

    A comma-separated list of Kubernetes secret names that the applications in the pod consume and their value is sourced from DAP.

    For example:

     
    name: K8S_SECRETS
    value: db-credentials,other-secret
  2. Load the manifest by running the following:

    • Kubernetes:

       
      kubectl apply -f deployment-with-conjur.yml
    • OpenShift:

       
      oc apply -f deployment-with-conjur.yml

Step 5: Start the application

To start the application:

  • Kubernetes:

     
    kubectl create -f deployment-with-conjur.yml
  • OpenShift:

     
    oc create -f deployment-with-conjur.yml

Troubleshooting the CyberArk Secrets Provider for Kubernetes

This section describes how to troubleshoot common CyberArk Secrets Provider for Kubernetes issues.

Enable logs

To enable logs, add the debug parameter to the application deployment manifest. For details, see Enable debug logs.

Display logs

To display the CyberArk Secrets Provider for Kubernetes init container logs:

Troubleshooting

The table below describes common issues and their resolution:

Issue

Error code

Resolution

SECRETS_DESTINATION=k8s_secrets is not configured as an init container

CSPFK007E

Validate that the SECRETS_DESTINATION variable is defined as k8s_secrets in the application manifest.

The Service Account doesn’t have permissions to fetch Kubernetes secrets data

CSPFK020E

Validate that the Service Account assigned to the pod has get permissions on the secrets resource.

For details, see Step 3: Configure RBAC permissions.

The Service Account doesn’t have permissions to update Kubernetes secrets data

CSPFK022E

Validate that the Service Account assigned to the pod has update permissions on the secret's resource.

For details, see Step 3: Configure RBAC permissions.

CyberArk Secrets Provider for Kubernetes is unable to authenticate

CSPFK010E

First, validate that you provided the correct host name in the manifest for the init container that authenticates to DAP. For details, see CONJUR_AUTHN_LOGIN.

Second, validate that you assigned the correct permissions to the authn-k8s service.

Kubernetes secrets resources that are defined in the K8S_SECRETS environment variable are not defined in Kubernetes

CSPFK020E

Validate that all Kubernetes secrets resources defined for the K8S_SECRETS environment variable exist in Kubernetes

For details, see Step 2: Map Kubernetes secrets to DAP secrets

DAP secrets are not accessible

CSPFK034E

Validate all DAP variable paths defined in the conjur-map data entry can be retrieved by the DAP host identity.

For details, see Step 1: Configure DAP resources.

Client application failed because of invalid or non existing Kubernetes secrets data

No error is raised by the cyberark-secrets-provider-for-k8s init container

Validate that the secrets-provider-for-k8s image is configured as an init container.

For details, see Step 3: Configure RBAC permissions.

Kubernetes secrets resources defined in the K8S_SECRETS environment variable are not properly configured.

CSPFK028E

Validate all Kubernetes secrets resources are defined in the K8S_SECRETS environment variable are configured with a conjur-map data entry.

For details, see Step 2: Map Kubernetes secrets to DAP secrets.

Contribute to the CyberArk Secrets Provider for Kubernetes

We welcome contributions. Visit the secrets-provider-for-k8s repo on GitHub.