Configure the Conjur cluster

This topic describes the process for setting up a high-availability Conjur cluster. A basic cluster includes one Leader and at least two Standbys.

Prerequisites

For information about system requirements, see System requirements for Conjur Server.

Before you begin your Conjur Enterprise configuration, you need to prepare your Conjur Enterprise environment:

Deployment environment

Preparation instructions

Provisioned VMs

Prepare Conjur environment for Docker or Podman deployment

AWS

Prepare Conjur environment for Docker or Podman deployment

Configure Conjur Leader

This section describes how to configure the Conjur Leader. The following host names will be used in the examples:

Component

Example Host Name

Conjur cluster Load balancer

lb.production.example.com

Leader

node1.production.example.com

Standby 1

node2.production.example.com

Standby 2

node3.production.example.com

Step 1: Configure the Leader

This step describes how to set up the Leader Conjur Server.

 

To configure the Leader, you must accept the Conjur end user license agreement (EULA) that is included in the installation package. To display and read the EULA, run evoke eula show.

To accept the EULA, include the --accept-eula flag with the evoke configure master command.

If the flag is not included:

  • When run interactively, you are prompted to accept the EULA.

  • When run non-interactively, the configuration fails.

  1. Connect to the Leader via SSH.

  2. Configure the node, including setting the admin password:

     
    $ docker exec <container-name> evoke configure master \
      --accept-eula \
      --hostname <load-balancer-dns> \
      --master-altnames <leader-dns>,<standby1-dns>,<standby2-dns> \
      --admin-password <password> \
      <account-name>

    Attribute

    Description

    container-name

    The name the container is given when it is started.

     

    If you are deploying Conjur Enterprise on AWS, the container name must be defined as conjur-appliance.If you are deploying Conjur Enterprise on a VM you can choose any container name, for example mycontainer.

    load-balancer-dns

    The DNS resolvable name assigned to the load balancer that sits in front of the HA cluster. Clients connect to this load balancer to communicate with the HA cluster. For example, lb.production.example.com

    master-altnames

     

    Omit this parameter if you are using 3rd-party certificates.

    When using self-signed certificates, Conjur Enterprise generates certificates that include the names listed in this parameter plus the DNS name of the Leader.

    The --master-altnames should include all possible FQDNs of each node within the Conjur cluster (Leader and each Standby).

    Provide a string that is a comma-separated list of alternate names to include in the generated certificate. It must include:

    • The DNS resolvable name for the Leader. For example, node1.production.example.com

    • The DNS names for all of the Standbys. For example, node2.production.example.com and node3.production.example.com.

    admin-password

    The password to assign to the built-in admin account.

    The password must be 12-128 characters, and must include:

    • 2 uppercase letters
    • 2 lowercase letters
    • 1 digit
    • 1 special character

    For example: MySecretP@ss1

    account-name

    The organizational account name that identifies this Conjur Enterprise deployment. This is typically the name of your organization, without punctuation, for example: default.

    All database records will include this organizational account label.

     
    • Between the --hostname flag and the --master-altnames flag, all FQDNs of each need node in the Conjur Enterprise cluster must be included at setup. Including all possible node FQDNs ensures that certificates remain valid following a failover event.

    • It is possible to add additional Standby names after initial deployment, but waiting requires redeploying each Standby. This is necessary because each Standby's certificate is a copy of the Leader's certificate. This may lead to problems if a Standby needs to be promoted.

    Example

Step 2: Optional - Import 3rd-party certificates

If you are using self-signed certificates, you can skip this step.

This step is optional and describes how to import 3rd-party certificates for the HA cluster. For more information, see Certificate requirements.

Conjur Enterprise requires certificates for communication between the Leader and Standbys in the cluster. It is important to gather all of the HA cluster certificates onto the Leader.

 

A self-signed certificate for the Leader and all the Standbys in the cluster was already generated by the evoke configure master command during the step, Configure the Leader. It contains the name of the Leader, and all of the DNS names provided in the -master-altnames parameter.

Assumptions

  • Certificates are placed in a folder /opt/cyberark/dap/certificates on the host machine.

  • Certificate files are named:

    • ca-chain.pem

    • master-key.pem

    • master-cert.pem

    • follower-key.pem

    • follower-cert.pem

  • The certificate destination is /opt/cyberark/dap/certificates inside the running container.

To import 3rd-party certificates from a 3rd-party issuer into the Leader:

  1. Copy all of the certificate files received from the 3rd-party issuer into a directory on the Leader.

    Be sure to copy all of the following files: 

    Purpose

    Required files

    Root/CA certificate

    Root certificate or a combined certificate chain (ca-chain.cert.pem) file.

    Server certificate

    Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the Leader.

    Common certificate for all Standbys

    Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the load balancer that sits in front of the HA cluster.

    Follower load balancer certificates

    A server key file (key.pem) and certificate (cert.pem) for each Follower load balancer. Each group of Followers shares the certificate for its respective load balancer.

  2. Copy each certificate into the running Conjur Enterprise container. On the host machine, run:

     
    # to ensure the folder exists in the container   
    $ docker exec <container-name> mkdir -p /opt/cyberark/dap/certificates
    $ docker cp /opt/cyberark/dap/certificates/ca-chain.pem <container-name>:/opt/cyberark/dap/certificates/ca-chain.pem
    $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/master-key.pem
    $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/master-cert.pem
    $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/follower-key.pem
    $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/follower-cert.pem
  3. On the Leader, import the Root/CA certificate:

     
    $ docker exec <container-name> evoke ca import --no-restart --root <root-cert-name>

    where:

    • <root-cert-name> is the Root/CA certificate file (.pem)

    For example:

     
    $ docker exec mycontainer evoke ca import --no-restart --root /opt/cyberark/dap/certificates/ca-chain.pem
     

    This stops all Conjur Enterprise services on the Leader, removes all existing certificates, and installs the new Root/CA certificate.

  4. On the Leader, import the key file and server certificate pair for the HA cluster certificate received from the 3rd party.

     
    $ docker exec <container-name> evoke ca import --no-restart --key <master-key-file> --set <master-server-cert>

    For example:

     
    $ docker exec mycontainer evoke ca import \
        --no-restart \
        --key /opt/cyberark/dap/certificates/master-key.pem \
        --set /opt/cyberark/dap/certificates/master-cert.pem
  5. On the Leader, import the key file and server certificate pairs of the Follower load balancers:

     
    $ docker exec <container-name> evoke ca import --no-restart --key <key-file-name> <server-cert-name>

    For example:

     
    $ docker exec mycontainer evoke ca import \
        --no-restart \
        --key /opt/cyberark/dap/certificates/follower-key.pem \
        /opt/cyberark/dap/certificates/follower-cert.pem
  6. Restart the Conjur services by running:

    docker exec <container-name> sv restart conjur nginx pg seed

Step 3: Optional - Encrypt the server keys on the Leader

This step is optional and describes how to encrypt the Conjur Enterprise server keys (data key, Conjur Enterprise UI key, and SSL keys) using a master key.

 

It is strongly recommended to encrypt your server keys.

 

Later in this section, you will generate seed files for Standbys. A seed file is an archive file, generated from the Leader, that contains all the files necessary for configuring a Standby. These seed files contain secrets, such as the data key, Conjur Enterprise UI key, and SSL keys.

If you choose not to encrypt these keys, the keys are stored on the Leader in plain text. It is very important to protect access to the seed files.

  1. Generate a master key using one of the following methods:

    Method

    Description

    Native key encryption This method uses a command line command to generate a master key. You manually manage access to the master key file or preferably store it offline in a safe and retrievable place.
    HSM Conjur Enterprise supports the use of a PKCS#11-compliant Hardware Security Module (HSM) to securely manage a master key.
    AWS KMS If you are running Conjur Enterprise in an AWS environment, Conjur Enterprise supports using the AWS Key Management Service (KMS) for generating and storing a master key.

    For details on encrypting server keys using these methods, see Server key encryption methods.

  2. Copy the file containing the generated master key (the master key file) to the Leader container.

  3. On the Leader, apply the master key:

     
    $ docker exec <container-name> evoke keys encrypt <path to master key file>
  4. Immediately unlock the master key on the Leader. This step allows the Conjur services to access the server keys.

     
    $ docker exec <container-name> evoke keys unlock <path to master key file>

    The following example shows the command sequence using a simple native key encryption:

     
    # Generate a master key named master.key
    $ openssl rand 32 > /etc/dap-secrets/master.key
    
    # Copy master.key to Leader container
    $ docker cp /etc/dap-secrets/master.key dap:/etc/dap-secrets/master.key
    
    # Encrypt server keys using master.key
    $ docker exec mycontainer evoke keys encrypt /etc/dap-secrets/master.key
    
    # Unlock server keys to allow access to Conjur Enterprise
    $ docker exec mycontainer evoke keys unlock /etc/dap-secrets/master.key
    

Step 4: Verify the Conjur Enterprise Leader configuration

This step describes how to verify the Conjur Enterprise Leader configuration.

  1. Verify health

    Successful setup can be verified by checking the Leader endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

     
    {
      "services": {
        "ldap-sync": "disabled",
        "ui": "ok",
        "possum": "ok",
        "ok": true
      },
      "database": {
        "ok": true,
        "connect": {
          "main": "ok"
        },
        "free_space": {
          "main": {
            "kbytes": 40749192,
            "inodes": 3245299
          }
        },
        "replication_status": {
          "pg_current_xlog_location": "0/1803F18",
          "pg_current_xlog_location_bytes": 25181976
        }
      },
      "ok": true
    }

    Check for a value of "ok": true at the bottom.

  2. Validate UI

    Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

After you have configured the Leader, continue the setup of the Conjur cluster by configuring the Standbys. For more information, see Configure Standbys.

Configure Standbys

After you have configured the Conjur Enterprise Leader, continue the setup of the Conjur cluster by configuring the Standbys.

Step 1: Create and unpack the Standby seed files

This step describes how to create a seed file for Standbys, and unpack the seed file on the corresponding Standby container.

A seed file is an archive file, generated from the Leader, that contains all the files necessary for configuring a Standby. These seed files contain secrets, such as the data key, Conjur Enterprise UI key, and SSL keys.

 
  • The steps in this section should be run on each of the Standby nodes. For example, node2.production.example.com and node3.production.example.com

  • The following step copies the seed file by streaming it via SSH. If you prefer to transport the file manually, see Seeding a node.

For each Standby, run the following command from your local terminal (or a terminal with access to both nodes):

 
$ ssh -i "<path to Leader .pem file>" <user@leader-dns> "sudo docker exec <leader-container-name> evoke seed standby <standby-dns><leader-dns>" | ssh -i "<path to Standby .pem file>" <user@standby-dns> "sudo docker exec -i <standby-container-name> evoke unpack seed -"

For example:

 
$ ssh -i "dap.pem" core@node1.production.example.com "sudo docker exec mycontainer evoke seed standby node2.production.example.com node1.production.example.com" | ssh -i "standby.pem" core@node2.production.example.com "sudo docker exec -i mycontainer evoke unpack seed -"

Step 2: Configure the Standbys

This step configures the Standby. The command requirements are different depending on whether or not your server keys are encrypted (For more information about encryption, see Optional - Encrypt the server keys on the Leader).

Step 3: Enable synchronous replication

Synchronous replication ensures all writes are written to the Leader and one Standby successfully. This protects against potential data loss in the event the Leader fails.

 

At least two Standbys are required for synchronous replication. This is a Conjur Enterprise system requirement.

SSH onto the Conjur Enterprise Leader and enable Synchronous replication:

 
$ docker exec <container-name> sh -c "
  evoke replication sync start
"

Step 4: Verify the Conjur cluster configuration

This step describes how to verify the Conjur Enterprise cluster configuration.

  1. Verify health

    Successful setup can be verified by checking the Leader endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

     
    {
      ...
      "database": {
      ...
        "replication_status": {
          "pg_stat_replication": [
            {
              "usename": "lb.production.example.com",
              "application_name": "node2.production.example.com",
              "client_addr": "10.10.0.2",
              ...
              "sync_state": "sync",
              ...
            },
            {
              "usename": "lb.production.example.com",
              "application_name": "node3.production.example.com",
              "client_addr": "10.10.0.3",
              ...
              "sync_state": "potential",
              ...
            },
            {
              "usename": "lb-follower.production.example.com",
              "application_name": "node4.production.example.com",
              "client_addr": "10.10.1.4",
              ...
              "sync_state": "async",
              ...
            }
          ],
          ...
        }
      },
      "ok": true
    }

    Check for a value of "ok": true at the bottom.

    This page provides some additional information about the state of the cluster and includes the following keys:

    Key

    Explanation

    "sync_state": "sync"

    This node handles synchronous replication.

    "sync_state": "potential"

    This node can take over synchronous replication if the current synchronous node fails.

    "sync_state": "async"

    This node is either a Follower, or is a disaster recovery Standby (which cannot handle synchronous replication).

  2. Validate UI

    Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

Step 5: Optional - Configure auto-failover

In an auto-failover event, if the Conjur Enterprise Leader becomes unavailable, auto-failover enables the automatic promotion of a Standby to Leader.

To add auto-failover onto the Conjur cluster, see Configure auto-failover.

Troubleshoot the setup

To troubleshoot the Leader or Standby setup, add the --debug flag to the evoke configure command. For more information, see Troubleshoot Conjur Enterprise configuration.