Configure the Conjur cluster

This topic describes the process for setting up a high-availability Conjur cluster. A basic cluster includes one Leader and at least two Standbys.

Prerequisites

For information about system requirements, see System requirements for Conjur Server.

Before you begin your Conjur Enterprise configuration, you need to prepare your Conjur Enterprise environment:

Configure Conjur Leader

This section describes how to configure the Conjur Leader. The following host names will be used in the examples:

Step 1: Configure the Leader

This step describes how to set up the Leader Conjur Server.

1. Connect to the Leader via SSH.

2. Configure the node, including setting the admin password:

   $ docker exec <container-name> evoke configure master \ --accept-eula \ --hostname <load-balancer-dns> \ --master-altnames <leader-dns>,<standby1-dns>,<standby2-dns> \ --admin-password <password> \ <account-name>

   Attribute	Description
   container-name	The name the container is given when it is started.   If you are deploying Conjur Enterprise on AWS, the container name must be defined as conjur-appliance.If you are deploying Conjur Enterprise on a VM you can choose any container name, for example mycontainer.
   load-balancer-dns	The DNS resolvable name assigned to the load balancer that sits in front of the HA cluster. Clients connect to this load balancer to communicate with the HA cluster. For example, lb.production.example.com
   master-altnames	Omit this parameter if you are using 3rd-party certificates. When using self-signed certificates, Conjur Enterprise generates certificates that include the names listed in this parameter plus the DNS name of the Leader. The --master-altnames should include all possible FQDNs of each node within the Conjur cluster (Leader and each Standby). Provide a string that is a comma-separated list of alternate names to include in the generated certificate. It must include: The DNS resolvable name for the Leader. For example, node1.production.example.com The DNS names for all of the Standbys. For example, node2.production.example.com and node3.production.example.com.
   admin-password	The password to assign to the built-in admin account. The password must be 12-128 characters, and must include: 2 uppercase letters 2 lowercase letters 1 digit 1 special character For example: MySecretP@ss1
   account-name	The organizational account name that identifies this Conjur Enterprise deployment. This is typically the name of your organization, without punctuation, for example: default. All database records will include this organizational account label.

   Between the --hostname flag and the --master-altnames flag, all FQDNs of each need node in the Conjur Enterprise cluster must be included at setup. Including all possible node FQDNs ensures that certificates remain valid following a failover event. It is possible to add additional Standby names after initial deployment, but waiting requires redeploying each Standby. This is necessary because each Standby's certificate is a copy of the Leader's certificate. This may lead to problems if a Standby needs to be promoted.

   Example

   Deployment on VMs

   docker exec mycontainer evoke configure master \ --accept-eula \ --hostname lb.production.example.com \ --master-altnames node1.production.example.com,node2.production.example.com,node3.production.example.com \ --admin-password MySecretP@ss1 \ default

   Deployment on AWS

   docker exec conjur-appliance evoke configure master \ --accept-eula \ --hostname lb.production.example.com \ --master-altnames node1.production.example.com,node2.production.example.com,node3.production.example.com \ --admin-password MySecretP@ss1 \ default

Step 2: Optional - Import 3rd-party certificates

If you are using self-signed certificates, you can skip this step.

This step is optional and describes how to import 3rd-party certificates for the HA cluster. For more information, see Certificate requirements.

Conjur Enterprise requires certificates for communication between the Leader and Standbys in the cluster. It is important to gather all of the HA cluster certificates onto the Leader.

Assumptions

• Certificates are placed in a folder /opt/cyberark/dap/certificates on the host machine.

• Certificate files are named:

  • ca-chain.pem

  • master-key.pem

  • master-cert.pem

  • follower-key.pem

  • follower-cert.pem

• The certificate destination is /opt/cyberark/dap/certificates inside the running container.

To import 3rd-party certificates from a 3rd-party issuer into the Leader:

1. Copy all of the certificate files received from the 3rd-party issuer into a directory on the Leader.

   Be sure to copy all of the following files: 

   Purpose	Required files
   Root/CA certificate	Root certificate or a combined certificate chain (ca-chain.cert.pem) file.
   Server certificate	Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the Leader.
   Common certificate for all Standbys	Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the load balancer that sits in front of the HA cluster.
   Follower load balancer certificates	A server key file (key.pem) and certificate (cert.pem) for each Follower load balancer. Each group of Followers shares the certificate for its respective load balancer.

2. Copy each certificate into the running Conjur Enterprise container. On the host machine, run:

   # to ensure the folder exists in the container $ docker exec <container-name> mkdir -p /opt/cyberark/dap/certificates $ docker cp /opt/cyberark/dap/certificates/ca-chain.pem <container-name>:/opt/cyberark/dap/certificates/ca-chain.pem $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/master-key.pem $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/master-cert.pem $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/follower-key.pem $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/follower-cert.pem

3. On the Leader, import the Root/CA certificate:

   $ docker exec <container-name> evoke ca import --no-restart --root <root-cert-name>

   where:

   • <root-cert-name> is the Root/CA certificate file (.pem)

   For example:

   $ docker exec mycontainer evoke ca import --no-restart --root /opt/cyberark/dap/certificates/ca-chain.pem

   This stops all Conjur Enterprise services on the Leader, removes all existing certificates, and installs the new Root/CA certificate.

4. On the Leader, import the key file and server certificate pair for the HA cluster certificate received from the 3rd party.

   $ docker exec <container-name> evoke ca import --no-restart --key <master-key-file> --set <master-server-cert>

   For example:

   $ docker exec mycontainer evoke ca import \ --no-restart \ --key /opt/cyberark/dap/certificates/master-key.pem \ --set /opt/cyberark/dap/certificates/master-cert.pem

5. On the Leader, import the key file and server certificate pairs of the Follower load balancers:

   $ docker exec <container-name> evoke ca import --no-restart --key <key-file-name> <server-cert-name>

   For example:

   $ docker exec mycontainer evoke ca import \ --no-restart \ --key /opt/cyberark/dap/certificates/follower-key.pem \ /opt/cyberark/dap/certificates/follower-cert.pem

6. Restart the Conjur services by running:

   docker exec <container-name> sv restart conjur nginx pg seed

Step 3: Optional - Encrypt the server keys on the Leader

This step is optional and describes how to encrypt the Conjur Enterprise server keys (data key, Conjur Enterprise UI key, and SSL keys) using a master key.

1. Generate a master key using one of the following methods:

   Method	Description
   Native key encryption	This method uses a command line command to generate a master key. You manually manage access to the master key file or preferably store it offline in a safe and retrievable place.
   HSM	Conjur Enterprise supports the use of a PKCS#11-compliant Hardware Security Module (HSM) to securely manage a master key.
   AWS KMS	If you are running Conjur Enterprise in an AWS environment, Conjur Enterprise supports using the AWS Key Management Service (KMS) for generating and storing a master key.

   For details on encrypting server keys using these methods, see Server key encryption methods.

2. Copy the file containing the generated master key (the master key file) to the Leader container.

3. On the Leader, apply the master key:

   $ docker exec <container-name> evoke keys encrypt <path to master key file>

4. Immediately unlock the master key on the Leader. This step allows the Conjur services to access the server keys.

   $ docker exec <container-name> evoke keys unlock <path to master key file>

   The following example shows the command sequence using a simple native key encryption:

   # Generate a master key named master.key $ openssl rand 32 > /etc/dap-secrets/master.key # Copy master.key to Leader container $ docker cp /etc/dap-secrets/master.key dap:/etc/dap-secrets/master.key # Encrypt server keys using master.key $ docker exec mycontainer evoke keys encrypt /etc/dap-secrets/master.key # Unlock server keys to allow access to Conjur Enterprise $ docker exec mycontainer evoke keys unlock /etc/dap-secrets/master.key

Step 4: Verify the Conjur Enterprise Leader configuration

This step describes how to verify the Conjur Enterprise Leader configuration.

1. Verify health

   Successful setup can be verified by checking the Leader endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

   { "services": { "ldap-sync": "disabled", "ui": "ok", "possum": "ok", "ok": true }, "database": { "ok": true, "connect": { "main": "ok" }, "free_space": { "main": { "kbytes": 40749192, "inodes": 3245299 } }, "replication_status": { "pg_current_xlog_location": "0/1803F18", "pg_current_xlog_location_bytes": 25181976 } }, "ok": true }

   Check for a value of "ok": true at the bottom.

2. Validate UI

   Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

After you have configured the Leader, continue the setup of the Conjur cluster by configuring the Standbys. For more information, see Configure Standbys.

Configure Standbys

After you have configured the Conjur Enterprise Leader, continue the setup of the Conjur cluster by configuring the Standbys.

Step 1: Create and unpack the Standby seed files

This step describes how to create a seed file for Standbys, and unpack the seed file on the corresponding Standby container.

A seed file is an archive file, generated from the Leader, that contains all the files necessary for configuring a Standby. These seed files contain secrets, such as the data key, Conjur Enterprise UI key, and SSL keys.

For each Standby, run the following command from your local terminal (or a terminal with access to both nodes):

$ ssh -i "<path to Leader .pem file>" <user@leader-dns> "sudo docker exec <leader-container-name> evoke seed standby <standby-dns><leader-dns>" | ssh -i "<path to Standby .pem file>" <user@standby-dns> "sudo docker exec -i <standby-container-name> evoke unpack seed -"

$ ssh -i "dap.pem" core@node1.production.example.com "sudo docker exec mycontainer evoke seed standby node2.production.example.com node1.production.example.com" | ssh -i "standby.pem" core@node2.production.example.com "sudo docker exec -i mycontainer evoke unpack seed -"

Step 2: Configure the Standbys

This step configures the Standby. The command requirements are different depending on whether or not your server keys are encrypted (For more information about encryption, see Optional - Encrypt the server keys on the Leader).

$ docker exec <standby-container-name> evoke keys exec -m <path to master key file> -- evoke configure standby

$ docker exec mycontainer evoke keys exec -m /dap_files/master.key -- evoke configure standby"

$ docker exec <standby-container-name> evoke configure standby

$ docker exec mycontainer evoke configure standby

Step 3: Enable synchronous replication

Synchronous replication ensures all writes are written to the Leader and one Standby successfully. This protects against potential data loss in the event the Leader fails.

SSH onto the Conjur Enterprise Leader and enable Synchronous replication:

$ docker exec <container-name> sh -c "
  evoke replication sync start
"

Step 4: Verify the Conjur cluster configuration

This step describes how to verify the Conjur Enterprise cluster configuration.

1. Verify health

   Successful setup can be verified by checking the Leader endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

   { ... "database": { ... "replication_status": { "pg_stat_replication": [ { "usename": "lb.production.example.com", "application_name": "node2.production.example.com", "client_addr": "10.10.0.2", ... "sync_state": "sync", ... }, { "usename": "lb.production.example.com", "application_name": "node3.production.example.com", "client_addr": "10.10.0.3", ... "sync_state": "potential", ... }, { "usename": "lb-follower.production.example.com", "application_name": "node4.production.example.com", "client_addr": "10.10.1.4", ... "sync_state": "async", ... } ], ... } }, "ok": true }

   Check for a value of "ok": true at the bottom.

   This page provides some additional information about the state of the cluster and includes the following keys:

   Key	Explanation
   "sync_state": "sync"	This node handles synchronous replication.
   "sync_state": "potential"	This node can take over synchronous replication if the current synchronous node fails.
   "sync_state": "async"	This node is either a Follower, or is a disaster recovery Standby (which cannot handle synchronous replication).

2. Validate UI

   Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

Step 5: Optional - Configure auto-failover

In an auto-failover event, if the Conjur Enterprise Leader becomes unavailable, auto-failover enables the automatic promotion of a Standby to Leader.

To add auto-failover onto the Conjur cluster, see Configure auto-failover.

Troubleshoot the setup

To troubleshoot the Leader or Standby setup, add the --debug flag to the evoke configure command. For more information, see Troubleshoot Conjur Enterprise configuration.