Conjur Follower inside OpenShift/Kubernetes cluster

This topic describes how to set up a Conjur Follower inside your OpenShift/Kubernetes cluster using JWT-based or certificate(cert)-based authentication.

For all the options for deploying Followers inside OpenShift/Kubernetes cluster, see Deploy Follower for OpenShift/Kubernetes integration.
This procedure involves tasks for both the Conjur admin and the Kubernetes cluster admin. Make sure that both persona are available when performing these tasks.
Unless specifically noted otherwise, all references to Kubernetes apply to native Kubernetes as well as RedHat OpenShift (using JWT-based authentication only), GKE, and other supported Kubernetes-based implementations.
When using JWT-based authentication, note that all references to the Kubernetes namespaces intentionally include the OpenShift concept of project.
Followers for OpenShift/ Kubernetes integrations do not support selective replication.

Before you begin

Make sure that you have the following:

A configured and enabled JWT Authenticator endpoint. For details, see JWT-based Kubernetes authentication.

The examples in this section follow on from the example JWT Authenticator configuration described in JWT-based Kubernetes authentication.

Ensure that you have a certificate for the Conjur Follower.

If you are using self-signed certificates, the seed generation webservice generates this certificate.

If you are using certificates signed by a third party, a Conjur Follower certificate with a subject name matching the FQDN of the service in Kubernetes needs to be loaded on the Leader. For more information, contact your CyberArk Professional Services representative.

The Conjur Follower uses the Conjur Server image (conjur-appliance) which is available for download from the CyberArk Support Vault. Contact your CyberArk representative for access to the CyberArk Conjur Safe.

The Seed Fetcher can be downloaded from the CyberArk Conjur Safe in the Support Vault or it can be pulled from DockerHub.com.

Download these images and copy them to your registry.

Image	Name in support Vault	Name on DockerHub.com
Conjur Server	conjur-appliance-vXX.X.tar.gz	n/a
Seed Fetcher	seed-fetcher.tar.gz	cyberark/dap-seedfetcher

Access to the kubectl CLI (or oc CLI for OpenShift)
Make sure you have the Conjur CLI (v7.0.1+) installed and that you are logged in. For details, see Set up the Conjur CLI.

Define an identity in Conjur for the Conjur Follower
Conjur admin: To enable the Conjur Follower to connect to Conjur, it first needs to authenticate to Conjur.
In this step, you give the Conjur Follower an identity in Conjur using policy.
Define the host using the same value as the token-app-property defined in the JWT Authenticator.

The examples in this section follow on from the example JWT Authenticator configuration described in JWT-based Kubernetes authentication.

We recommend using the sub claim and naming the host using the following syntax: system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT>.

For more information, see Workload identity for Kubernetes (JWT-based authentication).

In the following policy, the annotations include information about the namespace and the service account. Save the policy as k8s-follower.yml, and adjust the values accordingly:

- !policy id: app-path body: - !host id: system:serviceaccount:cyberark-conjur:authn-jwt-sa annotations: authn-jwt/dev-cluster/kubernetes.io/namespace: cyberark-conjur authn-jwt/dev-cluster/kubernetes.io/serviceaccount/name: authn-jwt-sa

Load the policy file to root.

$ conjur policy load -f k8s-follower.yml -b root

Give the Conjur Follower permission to authenticate to Conjur and to use the seed service using the JWT Authenticator endpoint.

Save the following policy as grant-follower-access.yml:

- !grant roles: - !group conjur/authn-jwt/dev-cluster/apps - !group conjur/seed-generation/consumers members: - !host app-path/system:serviceaccount:cyberark-conjur:authn-jwt-sa

Load the policy file to root:

$ conjur policy load -f grant-follower-access.yml -b root

Set up a ConfigMap

Kubernetes cluster admin (using input from the Conjur admin): In this step you create a ConfigMap for populating Conjur connection details into the Conjur Follower deployment manifest.

Ask your Conjur admin for:

the Conjur cluster load balancer URL (https://<conjur-lb-dns>). Note that the URL starts with https:// For example: https://conjur-lb.example.com
the Kubernetes Authenticator service ID (SERVICE_ID) for your Kubernetes cluster (for example, dev-cluster)

the public certificate for Conjur

Conjur admin: To retrieve the public certificate, run the following command using the URL (excluding the https:// prefix):

$ openssl s_client -showcerts -connect conjur-lb.example.com:443 < /dev/null 2> /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > conjur.pem

Create a conjur.pem file using the certificate you got from the Conjur admin. Then create the CONJUR_SSL_CERTIFICATE environment variable:

$ CONJUR_SSL_CERTIFICATE=conjur.pem

Create the CONJUR_APPLIANCE_URL environment variable using the Conjur cluster load balancer URL that you got from the Conjur admin:

$ CONJUR_APPLIANCE_URL=https://conjur-lb.example.com

Create the AUTHENTICATOR_ID environment variable that you got from the Conjur admin

$ AUTHENTICATOR_ID=dev-cluster

Retrieve the Conjur account:

$ curl -k $CONJUR_APPLIANCE_URL/info | grep account

Create the following environment variable using the account name you retrieved, for example:

$ CONJUR_ACCOUNT=myorg

Create the CONJUR_SEED_FILE_URL environment variable using the service ID that you got from the Conjur admin:

$ CONJUR_SEED_FILE_URL=$CONJUR_APPLIANCE_URL/configuration/$CONJUR_ACCOUNT/seed/follower

Create a ConfigMap, follower-cm, in the cyberark-conjur namespace using the variables you just created:

For OpenShift, replace kubectl with oc.

Do a dry run first to check that the values are correct:

$ kubectl create configmap follower-cm -n cyberark-conjur \
  -o yaml \
  --dry-run \
  --from-literal CONJUR_ACCOUNT=${CONJUR_ACCOUNT} \
  --from-literal CONJUR_APPLIANCE_URL=${CONJUR_APPLIANCE_URL} \
  --from-literal CONJUR_SEED_FILE_URL=${CONJUR_SEED_FILE_URL} \
  --from-literal AUTHENTICATOR_ID=${AUTHENTICATOR_ID} \
  --from-file "CONJUR_SSL_CERTIFICATE=${CONJUR_SSL_CERTIFICATE}"

When you are satisfied, create and apply the ConfigMap:

$ kubectl create configmap follower-cm -n cyberark-conjur \
  -o yaml \
  --dry-run \
  --from-literal CONJUR_ACCOUNT=${CONJUR_ACCOUNT} \
  --from-literal CONJUR_APPLIANCE_URL=${CONJUR_APPLIANCE_URL} \
  --from-literal CONJUR_SEED_FILE_URL=${CONJUR_SEED_FILE_URL} \
  --from-literal AUTHENTICATOR_ID=${AUTHENTICATOR_ID} \
  --from-file "CONJUR_SSL_CERTIFICATE=${CONJUR_SSL_CERTIFICATE}" | kubectl apply -f -

Set up the Follower service and deployment manifest

Kubernetes cluster admin: In this step you set up the Follower service and deployment manifest using the ConfigMap you created in the previous step.

Copy the following manifest, and change the mycorp-registry/conjur-appliance:version to match the internal registry where you pushed the conjur appliance image.

The manifest instructs Kubernetes to project a service account token (JWT) to the application file system. Under volumes provide the following projection information for jwt-token:

Field	Description
`path`	A token path that can be used to identify the token, for example `jwt`
`expirationSeconds`	(Optional) The expiration of the token in seconds, for example `6000`
`audience`	Tthe token's audience. This is required only if the JWT Authenticator endpoint was defined with an `audience` variable, and the value must match the value that the variable was populated with, for example `https://conjur.host.name/`. For details, see Populate the policy variables. Otherwise `audience` is optional.

Field

Description

path

A token path that can be used to identify the token, for example jwt

expirationSeconds

(Optional) The expiration of the token in seconds, for example 6000

audience

Tthe token's audience. This is required only if the JWT Authenticator endpoint was defined with an audience variable, and the value must match the value that the variable was populated with, for example https://conjur.host.name/. For details, see Populate the policy variables.

Otherwise audience is optional.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: conjur-follower
spec:
  replicas: 1
  selector:
    matchLabels:
      app: conjur-follower
  template:
    metadata:
      labels:
        app: conjur-follower
        name: conjur-follower
        role: follower
    spec:
      serviceAccountName: authn-jwt-sa
      containers:
      - name: conjur-appliance
        image: conjur-appliance:version
        command: ["/tmp/seedfile/start-follower.sh"]
        imagePullPolicy: Always
        env:
          - name: SEEDFILE_DIR
            value: /tmp/seedfile
          - name: CONJUR_AUTHENTICATORS
            value: authn-jwt/dev-cluster
        ports:
        - containerPort: 443
          name: https
        readinessProbe:
          httpGet:
            path: /health
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 15
          timeoutSeconds: 5
        volumeMounts:
          - name: seedfile
            mountPath: /tmp/seedfile
            readOnly: true
      initContainers:
      - name: authenticator
        image: cyberark/dap-seedfetcher
        imagePullPolicy: Always
        env:
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: FOLLOWER_HOSTNAME
            value: conjur-follower
          - name: SEEDFILE_DIR
            value: /tmp/seedfile
          - name: JWT_TOKEN_PATH
            value: /var/run/secrets/tokens/jwt
          - name: AUTHENTICATOR
            value: authn-jwt
        envFrom:
          - configMapRef:
              name: follower-cm
        volumeMounts:
          - name: seedfile
            mountPath: /tmp/seedfile
          - name: conjur-token
            mountPath: /run/conjur
          - name: jwt-token
            mountPath: /var/run/secrets/tokens
      volumes:
      - name: seedfile
        emptyDir:
          medium: Memory
      - name: conjur-token
        emptyDir:
          medium: Memory
      - name: jwt-token
        projected:
            sources:
              - serviceAccountToken:
                  audience: https://conjur.host.name/
                  path: jwt
                  expirationSeconds: 6000

Deploy the Conjur Follower

Load the Conjur Follower manifest into the cyberark-conjur namespace.

The environment is now ready for the app owner to set up applications to retrieve secrets from Conjur. For more information, see Set up workloads (JWT-based authentication).

Before you begin

Make sure that you have the following:

A configured and enabled Kubernetes Authenticator endpoint. For details, see Certificate-based Kubernetes authentication.

Ensure that you have a certificate for the Conjur Follower.

If you are using self-signed certificates, the seed generation webservice generates this certificate.

The Seed Fetcher can be downloaded from the CyberArk Conjur Safe in the Support Vault or it can be pulled from DockerHub.com.

Download these images and copy them to your registry.

Image	Name in support Vault	Name on DockerHub.com
Conjur Server	conjur-appliance-vXX.X.tar.gz	n/a
Seed Fetcher	seed-fetcher.tar.gz	cyberark/dap-seedfetcher

Access to the kubectl CLI
Make sure you have the Conjur CLI (v7.0.1+) installed and that you are logged in. For details, see Set up the Conjur CLI.

Define an identity in Conjur for the Conjur Follower

Conjur admin: To enable the Conjur Follower to connect to Conjur, it first needs to authenticate to Conjur.

In this step, you give the Conjur Follower an identity in Conjur using policy. The policy:

Defines the Conjur Follower by its namespace, cyberark-conjur.

The Conjur Follower can also be defined by its service account, authn-k8s-sa. These definitions are defined in the host annotations in the policy. For guidelines on how to define annotations, see Workload identity for Kubernetes (cert-based authentication).

Gives the Conjur Follower permission to authenticate to Conjur and to use the seed service using the dev-cluster Kubernetes Authenticator.

Save the following policy as k8s-follower.yml:

- !host
  id: k8s-follower
  annotations:
    authn-k8s/namespace: cyberark-conjur
    authn-k8s/service-account: authn-k8s-sa
   
- !grant
  roles:
  - !group conjur/authn-k8s/dev-cluster/consumers
  - !group conjur/seed-generation/consumers
  members:
  - !host k8s-follower

Load the policy file to root.

$ conjur policy load -f k8s-follower.yml -b root

Bind the role to the Kubernetes Authenticator service account

Kubernetes cluster admin: During the Kubernetes Authenticator configuration, the conjur-authn-role cluster role and authn-k8s-sa service account were created. In this step, you bind them together in the cyberark-conjur namespace.

Copy the following manifest and load it into the cyberark-conjur namespace:

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: conjur-authn-rolebinding
  namespace: cyberark-conjur
subjects:
- kind: ServiceAccount
  name: authn-k8s-sa
  namespace: cyberark-conjur
roleRef:
  kind: ClusterRole
  name: conjur-authn-role
  apiGroup: rbac.authorization.k8s.io

Set up a ConfigMap

Kubernetes cluster admin (using input from the Conjur admin): In this step you create a ConfigMap for populating Conjur connection details into the Conjur Follower deployment manifest.

Ask your Conjur admin for:

the Conjur cluster load balancer URL (https://<conjur-lb-dns>). Note that the URL starts with https:// For example: https://conjur-lb.example.com
the Kubernetes Authenticator service ID (SERVICE_ID) for your Kubernetes cluster (for example, dev-cluster)

the public certificate for Conjur

Conjur admin: To retrieve the public certificate, run the following command using the URL (excluding the https:// prefix):

$ openssl s_client -showcerts -connect conjur-lb.example.com:443 < /dev/null 2> /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > conjur.pem

Create a conjur.pem file using the certificate you got from the Conjur admin. Then create the CONJUR_SSL_CERTIFICATE environment variable:

$ CONJUR_SSL_CERTIFICATE=conjur.pem

Create the CONJUR_APPLIANCE_URL environment variable using the Conjur cluster load balancer URL that you got from the Conjur admin:

$ CONJUR_APPLIANCE_URL=https://conjur-lb.example.com

Create the AUTHENTICATOR_ID environment variable that you got from the Conjur admin

$ AUTHENTICATOR_ID=dev-cluster

Retrieve the Conjur account:

$ curl -k $CONJUR_APPLIANCE_URL/info | grep account

Create the following environment variable using the account name you retrieved, for example:

$ CONJUR_ACCOUNT=myorg

Create the CONJUR_SEED_FILE_URL environment variable using the service ID that you got from the Conjur admin:

$ CONJUR_SEED_FILE_URL=$CONJUR_APPLIANCE_URL/configuration/$CONJUR_ACCOUNT/seed/follower

Create a ConfigMap, follower-cm, in the cyberark-conjur namespace using the variables you just created:

Do a dry run first to check that the values are correct:

$ kubectl create configmap follower-cm -n cyberark-conjur \
  -o yaml \
  --dry-run \
  --from-literal CONJUR_ACCOUNT=${CONJUR_ACCOUNT} \
  --from-literal CONJUR_APPLIANCE_URL=${CONJUR_APPLIANCE_URL} \
  --from-literal CONJUR_SEED_FILE_URL=${CONJUR_SEED_FILE_URL} \
  --from-literal AUTHENTICATOR_ID=${AUTHENTICATOR_ID} \
  --from-file "CONJUR_SSL_CERTIFICATE=${CONJUR_SSL_CERTIFICATE}"

When you are satisfied, create and apply the ConfigMap:

$ kubectl create configmap follower-cm -n cyberark-conjur \
  -o yaml \
  --dry-run \
  --from-literal CONJUR_ACCOUNT=${CONJUR_ACCOUNT} \
  --from-literal CONJUR_APPLIANCE_URL=${CONJUR_APPLIANCE_URL} \
  --from-literal CONJUR_SEED_FILE_URL=${CONJUR_SEED_FILE_URL} \
  --from-literal AUTHENTICATOR_ID=${AUTHENTICATOR_ID} \
  --from-file "CONJUR_SSL_CERTIFICATE=${CONJUR_SSL_CERTIFICATE}" | kubectl apply -f -

Set up the Follower service and deployment manifest

Kubernetes cluster admin: In this step you set up the Follower service and deployment manifest using the ConfigMap you created in the previous step.

Copy the following manifest, and change the mycorp-registry/conjur-appliance:version to match the internal registry where you pushed the conjur appliance image.

If your Conjur Follower DNS name does not follow the default format of $FOLLOWER_HOSTNAME.$MY_POD_NAMESPACE.svc.cluster.local, you can use additional variables in the manifest to customize it to ensure that Conjur can locate the corresponding certificate. For instructions, see Customize Conjur Follower in Kubernetes DNS name.

---
apiVersion: v1
kind: Service
metadata:
  name: conjur-follower
  labels:
    app: conjur-follower
spec:
  ports:
  - port: 443
    name: https
  selector:
    app: conjur-follower
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: conjur-follower
spec:
  replicas: 1
  selector:
    matchLabels:
      app: conjur-follower
  template:
    metadata:
      labels:
        app: conjur-follower
        name: conjur-follower
        role: follower
    spec:
      serviceAccountName: authn-k8s-sa
      volumes:
      - name: seedfile
        emptyDir:
          medium: Memory
      - name: conjur-token
        emptyDir:
          medium: Memory
      initContainers:
      - name: authenticator
        image: cyberark/dap-seedfetcher
        imagePullPolicy: Always
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: FOLLOWER_HOSTNAME
            value: conjur-follower
          - name: SEEDFILE_DIR
            value: /tmp/seedfile
          - name: CONJUR_AUTHN_LOGIN
            value: host/k8s-follower
        envFrom:
          - configMapRef:
              name: follower-cm
        volumeMounts:
          - name: seedfile
            mountPath: /tmp/seedfile
          - name: conjur-token
            mountPath: /run/conjur
      containers:
      - name: conjur-appliance
        image: mycorp-registry/conjur-appliance:version
        command: ["/tmp/seedfile/start-follower.sh"]
        imagePullPolicy: Always
        env:
          - name: SEEDFILE_DIR
            value: /tmp/seedfile
          - name: CONJUR_AUTHENTICATORS
            value: authn-k8s/dev-cluster
        ports:
        - containerPort: 443
          name: https
        readinessProbe:
          httpGet:
            path: /health
            port: 443
            scheme: HTTPS
          initialDelaySeconds: 15
          timeoutSeconds: 5
        volumeMounts:
          - name: seedfile
            mountPath: /tmp/seedfile
            readOnly: true

Deploy the Conjur Follower

Load the Conjur Follower manifest into the cyberark-conjur namespace.

Customize Conjur Follower in Kubernetes DNS name

If your Conjur Follower DNS name does not follow the default format of $FOLLOWER_HOSTNAME.$MY_POD_NAMESPACE.svc.cluster.local, you can use additional variables in the manifest to customize it. You must ensure that the Conjur Follower DNS name is correctly configured so that Conjur can identify the corresponding TLS certificate.

Use only one of the options from the following table to configure your Conjur Follower DNS name.

Variables to customize Conjur Follower domain name
Variable	Details
`K8S_DNS_SUFFIX`	Replaces the default suffix added to internal Kubernetes URLs. Add this variable to the manifest if the Conjur Follower URL ends with a suffix other than `svc.cluster.local` and set its value to the correct suffix. Do not remove any variables from the manifest when using this option. Example manifest excerpt with `K8S_DNS_SUFFIX` In the following example, the Conjur Follower DNS name uses the suffix `k8s.mycompany.net`, which is specified in the `K8S_DNS_SUFFIX` variable. Note that the `FOLLOWER_HOSTNAME` variable remains in the manifest. initContainers: - name: authenticator image: cyberark/dap-seedfetcher ... env: ... - name: FOLLOWER_HOSTNAME value: conjur-follower # This provides the namespace name to # the seed fetcher container. There is # nothing user-configured in this config line. # # This is the Kubernetes downward API: # https://kubernetes.io/docs/concepts/workloads/pods/downward-api/ - name: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace # Optional, defaults to 'svc.cluster.local' - name: K8S_DNS_SUFFIX value: k8s.mycompany.net
`FOLLOWER_FQDN`	Replaces the entire fully qualified domain name (FQDN) of the Conjur Follower, including the hostname and suffix. Add this variable to the manifest only if the Conjur Follower DNS name is not in the format `$FOLLOWER_HOSTNAME.$POD_NAMESPACE.$K8S_DNS_SUFFIX`. If you add the `FOLLOWER_FQDN` variable: Remove the `FOLLOWER_HOSTNAME` and `K8S_DNS_SUFFIX` variables from the manifest. Keep the `MY_POD_NAMESPACE` variable in the manifest, even if it does not correspond with the `FOLLOWER_FQDN` value. The `MY_POD_NAMESPACE` variable is necessary for the configuration. Example manifest excerpt with `FOLLOWER_FQDN` In the following example, the Conjur Follower DNS name is `my-k8s-follower.conjur.mycompany.net`, which does not have the typical format of `$FOLLOWER_HOSTNAME.$POD_NAMESPACE.$K8S_DNS_SUFFIX`. The manifest uses the `FOLLOWER_FQDN` variable to specify the full Conjur Follower DNS name instead of using the `FOLLOWER_HOSTNAME` and `K8S_DNS_SUFFIX` variables. `initContainers: - name: authenticator image: cyberark/dap-seedfetcher ... env: ... - name: FOLLOWER_FQDN value: my-k8s-follower.conjur.mycompany.net`

The environment is now ready for the app owner to set up applications to retrieve secrets from Conjur. For more information, see Set up workloads (cert-based authentication).

Troubleshoot the deployment

When deployed, a Conjur Follower Pod self-initializes from the Leader, a process that takes upwards of 30 seconds depending on processor speed and the amount of CPU resources specified in the manifest. The Readiness probe shows that the Pod is ready once initialization completes.

If the Conjur Follower shows errors or fails to start, start from the Pod and work backwards to the Leader, checking the following:

On the Conjur Follower configuration host:

Check	Description
`kubectl get events -n cyberark-conjur`	Ensure that the seed-fetcher init container started without errors. Image pull errors are common. Check the image name spelling in the manifest and ensure the image was pushed to the registry and referenced correctly. Ensure the user deploying the Conjur Follower has correct privileges to and can log in to the registry.
`kubectl logs <follower-pod-name> -c authenticator -n cyberark-conjur`	Ensure that the seed-fetcher started and authenticates successfully.

Check

Description

kubectl get events -n cyberark-conjur

Ensure that the seed-fetcher init container started without errors. Image pull errors are common.

Check the image name spelling in the manifest and ensure the image was pushed to the registry and referenced correctly.

Ensure the user deploying the Conjur Follower has correct privileges to and can log in to the registry.

kubectl logs <follower-pod-name> -c authenticator -n cyberark-conjur

Ensure that the seed-fetcher started and authenticates successfully.

On the Leader host:

Check	Description
`docker logs <MASTER_CONTAINER_NAME> --since 1m`	Display last minute’s worth of the Leader log to trace seed-fetcher authentication messages in the Leader log. Ensure that authentication requests appear in the log. If not, check possible network connection issue (firewall, security groups, etc.) or certificate validation failure.
`docker exec -it <MASTER_CONTAINER_NAME> evoke variable list CONJUR_AUTHENTICATORS`	Ensure `authn-k8s/<SERVICE_ID>`is present in list and spelled correctly, for example: `authn-k8s/dev-cluster>`

Check

Description

docker logs <MASTER_CONTAINER_NAME> --since 1m

Display last minute’s worth of the Leader log to trace seed-fetcher authentication messages in the Leader log.

Ensure that authentication requests appear in the log. If not, check possible network connection issue (firewall, security groups, etc.) or certificate validation failure.

docker exec -it <MASTER_CONTAINER_NAME> evoke variable list CONJUR_AUTHENTICATORS

Ensure authn-k8s/<SERVICE_ID>is present in list and spelled correctly, for example: authn-k8s/dev-cluster>