Configure the Conjur cluster (Docker)

This topic describes the process for setting up a high-availability Conjur cluster. A basic cluster includes 1 Master and at least 2 Standbys.

Prerequisites

For information about system requirements, see System Requirements for Conjur Server

Before you begin your Conjur Enterprise configuration, you need to prepare your Conjur Enterprise environment:

Deployment environment

Preparation instructions

Provisioned VMs

Prepare VMs for Conjur deployment and configuration

AWS

Prepare AWS image for Conjur deployment and configuration

Configure Conjur Master

This section describes how to configure the Conjur Master. The following host names will be used in the examples:

Component

Example Host Name

Conjur cluster Load balancer

lb.production.example.com

Master

node1.production.example.com

Standby 1

node2.production.example.com

Standby 2

node3.production.example.com

Step 1: Configure the Master

This step describes how to set up the Master Conjur Server.

 

To configure the Master, you must accept the Conjur end user license agreement (EULA) that is included in the installation package. To display and read the EULA, run evoke eula show.

To accept the EULA, include the --accept-eula flag with the evoke configure master command.

If the flag is not included:

  • When run interactively, you are prompted to accept the EULA.

  • When run non-interactively, the configuration fails.

  1. Connect to the Master via SSH.

  2. Configure the node, including setting the admin password:

     
    $ docker exec <container-name> evoke configure master \
      --accept-eula \
      --hostname <load-balancer-dns> \
      --master-altnames <master-dns>,<standby1-dns>,<standby2-dns> \
      --admin-password <password> \
      <account-name>

    Attribute

    Description

    container-name

    The name the container is given when it is started.

     

    If you are deploying Conjur Enterprise on AWS, the container name must be defined as conjur-appliance.If you are deploying Conjur Enterprise on a VM you can choose any container name, for example mycontainer.

    load-balancer-dns

    The DNS resolvable name assigned to the load balancer that sits in front of the HA cluster. Clients connect to this load balancer to communicate with the HA cluster. For example, lb.production.example.com

    master-altnames

     

    Omit this parameter if you are using 3rd-party certificates.

    When using self-signed certificates, Conjur Enterprise generates certificates that include the names listed in this parameter plus the DNS name of the Master.

    The --master-altnames should include all possible FQDNs of each node within the Conjur cluster (Master and each Standby).

    Provide a string that is a comma-separated list of alternate names to include in the generated certificate. It must include:

    • The DNS resolvable name for the Master. For example, node1.production.example.com

    • The DNS names for all of the Standbys. For example, node2.production.example.com and node3.production.example.com.

    admin-password

    The password to assign to the built-in admin account.

    The password must be 12-128 characters, and must include:

    • 2 uppercase letters
    • 2 lowercase letters
    • 1 digit
    • 1 special character

    For example: MySecretP@ss1

    account-name

    The organizational account name that identifies this Conjur Enterprise deployment. This is typically the name of your organization, without punctuation, for example: default.

    All database records will include this organizational account label.

     
    • Between the --hostname flag and the --master-altnames flag, all FQDNs of each need node in the Conjur Enterprise cluster must be included at setup. Including all possible node FQDNs ensures that certificates remain valid following a failover event.

    • It is possible to add additional Standby names after initial deployment, but waiting requires redeploying each Standby. This is necessary because each Standby's certificate is a copy of the Master's certificate. This may lead to problems if a Standby needs to be promoted.

    Example

Step 2: Optional - Import 3rd-party certificates

If you are using self-signed certificates, you can skip this step.

This step is optional and describes how to import 3rd-party certificates for the HA cluster. For more information, see Certificate Requirements.

Conjur Enterprise requires certificates for communication between the Master and Standbys in the cluster. It is important to gather all of the HA cluster certificates onto the Master.

 

A self-signed certificate for the Master and all the Standbys in the cluster was already generated by the evoke configure master command during the step, Configure the Master. It contains the name of the Master, and all of the DNS names provided in the -master-altnames parameter.

Assumptions

  • Certificates are placed in a folder /opt/cyberark/dap/certificates on the host machine.

  • Certificate files are named:

    • ca-chain.pem

    • master-key.pem

    • master-cert.pem

    • follower-key.pem

    • follower-cert.pem

  • The certificate destination is /opt/cyberark/dap/certificates inside the running container.

To import 3rd-party certificates from a 3rd-party issuer into the Master:

  1. Copy all of the certificate files received from the 3rd-party issuer into a directory on the Master.

    Be sure to copy all of the following files: 

    Purpose

    Required files

    Root/CA certificate

    Root certificate or a combined certificate chain (ca-chain.cert.pem) file.

    Server certificate

    Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the Master.

    Common certificate for all Standbys

    Server key file (key.pem) and certificate (cert.pem) containing the DNS name for the load balancer that sits in front of the HA cluster.

    Follower load balancer certificates

    A server key file (key.pem) and certificate (cert.pem) for each Follower load balancer. Each group of Followers shares the certificate for its respective load balancer.

  2. Copy each certificate into the running Conjur Enterprise container. On the host machine, run:

     
    # to ensure the folder exists in the container   
    $ docker exec <container-name> mkdir -p /opt/cyberark/dap/certificates
    $ docker cp /opt/cyberark/dap/certificates/ca-chain.pem <container-name>:/opt/cyberark/dap/certificates/ca-chain.pem
    $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/master-key.pem
    $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/master-cert.pem
    $ docker cp /opt/cyberark/dap/certificates/master-key.pem <container-name>:/opt/cyberark/dap/certificates/follower-key.pem
    $ docker cp /opt/cyberark/dap/certificates/master-cert.pem <container-name>:/opt/cyberark/dap/certificates/follower-cert.pem
  3. On the Master, import the Root/CA certificate:

     
    $ docker exec <container-name> evoke ca import --root <root-cert-name>

    where:

    • <root-cert-name> is the Root/CA certificate file (.pem)

    For example:

     
    $ docker exec mycontainer evoke ca import --root /opt/cyberark/dap/certificates/ca-chain.pem
     

    This stops all Conjur Enterprise services on the Master, removes all existing certificates, and installs the new Root/CA certificate.

  4. On the Master, import the key file and server certificate pair for the HA cluster certificate received from the 3rd party.

     
    $ docker exec <container-name> evoke ca import --key <master-key-file> --set <master-server-cert>

    For example:

     
    $ docker exec mycontainer evoke ca import \
        --key /opt/cyberark/dap/certificates/master-key.pem \
        --set /opt/cyberark/dap/certificates/master-cert.pem
  5. On the Master, import the key file and server certificate pairs of the Follower load balancers:

     
    $ docker exec <container-name> evoke ca import --key <key-file-name> <server-cert-name>

    For example:

     
    $ docker exec mycontainer evoke ca import \
        --key /opt/cyberark/dap/certificates/follower-key.pem \
        /opt/cyberark/dap/certificates/follower-cert.pem 

Step 3: Optional - Encrypt the server keys on the Master

This step is optional and describes how to encrypt the Conjur Enterprise server keys (data key, Conjur Enterprise UI key, and SSL keys) using a master key.

 

It is strongly recommended to encrypt your server keys.

 

Later in this section, you will generate seed files for Standbys. A seed file is an archive file, generated from the Master, that contains all the files necessary for configuring a Standby. These seed files contain secrets, such as the data key, Conjur Enterprise UI key, and SSL keys.

If you choose not to encrypt these keys, the keys are stored on the Master in plain text. It is very important to protect access to the seed files.

  1. Generate a master key using one of the following methods:

    Method

    Description

    Native key encryption This method uses a command line command to generate a master key. You manually manage access to the master key file or preferably store it offline in a safe and retrievable place.
    HSM Conjur Enterprise supports the use of a PKCS#11-compliant Hardware Security Module (HSM) to securely manage a master key.
    AWS KMS If you are running Conjur Enterprise in an AWS environment, Conjur Enterprise supports using the AWS Key Management Service (KMS) for generating and storing a master key.

    For details on encrypting server keys using these methods, see Server Key Encryption Methods.

  2. Copy the file containing the generated master key (the master key file) to the Master container.

  3. On the Master, apply the master key:

     
    $ docker exec <container-name> evoke keys encrypt <path to master key file>
  4. Immediately unlock the master key on the Master. This step allows the Conjur services to access the server keys.

     
    $ docker exec <container-name> evoke keys unlock <path to master key file>

    The following example shows the command sequence using a simple native key encryption:

     
    # Generate a master key named master.key
    $ openssl rand 32 > /etc/dap-secrets/master.key
    
    # Copy master.key to Master container
    $ docker cp /etc/dap-secrets/master.key dap:/etc/dap-secrets/master.key
    
    # Encrypt server keys using master.key
    $ docker exec mycontainer evoke keys encrypt /etc/dap-secrets/master.key
    
    # Unlock server keys to allow access to Conjur Enterprise
    $ docker exec mycontainer evoke keys unlock /etc/dap-secrets/master.key
    

Step 4: Verify the Conjur Enterprise Master configuration

This step describes how to verify the Conjur Enterprise Master configuration.

  1. Verify health

    Successful setup can be verified by checking the Master endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

     
    {
      "services": {
        "ldap-sync": "disabled",
        "ui": "ok",
        "possum": "ok",
        "ok": true
      },
      "database": {
        "ok": true,
        "connect": {
          "main": "ok"
        },
        "free_space": {
          "main": {
            "kbytes": 40749192,
            "inodes": 3245299
          }
        },
        "replication_status": {
          "pg_current_xlog_location": "0/1803F18",
          "pg_current_xlog_location_bytes": 25181976
        }
      },
      "ok": true
    }

    Check for a value of "ok": true at the bottom.

  2. Validate UI

    Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

After you have configured the Master, continue the setup of the Conjur cluster by configuring the Standbys. For more information, see Configure Standbys.

Configure Standbys

After you have configured the Conjur Enterprise Master, continue the setup of the Conjur cluster by configuring the Standbys.

Step 1: Create and unpack the Standby seed files

This step describes how to create a seed file for Standbys, and unpack the seed file on the corresponding Standby container.

A seed file is an archive file, generated from the Master, that contains all the files necessary for configuring a Standby. These seed files contain secrets, such as the data key, Conjur Enterprise UI key, and SSL keys.

 
  • The steps in this section should be run on each of the Standby nodes. For example, node2.production.example.com and node3.production.example.com

  • The following step copies the seed file by streaming it via SSH. If you prefer to transport the file manually, see Seeding a node.

For each Standby, run the following command from your local terminal (or a terminal with access to both nodes):

 
$ ssh -i "<path to Master .pem file>" <user@master-dns> "sudo docker exec <master-container-name> evoke seed standby <standby-dns><master-dns>" | ssh -i "<path to Standby .pem file>" <user@standby-dns> "sudo docker exec -i <standby-container-name> evoke unpack seed -"

For example:

 
$ ssh -i "dap.pem" core@node1.production.example.com "sudo docker exec mycontainer evoke seed standby node2.production.example.com node1.production.example.com" | ssh -i "standby.pem" core@node2.production.example.com "sudo docker exec -i mycontainer evoke unpack seed -"

Step 2: Configure the Standbys

This step configures the Standby. The command requirements are different depending on whether or not your server keys are encrypted (For more information about encryption, see Optional - Encrypt the server keys on the Master).

Step 3: Enable synchronous replication

Synchronous replication ensures all writes are written to the Master and one Standby successfully. This protects against potential data loss in the event the Master fails.

 

At least two Standbys are required for synchronous replication. This is a Conjur Enterprise system requirement.

SSH onto the Conjur Enterprise Master and enable Synchronous replication:

 
$ docker exec <container-name> sh -c "
  evoke replication sync start
"

Step 4: Verify the Conjur cluster configuration

This step describes how to verify the Conjur Enterprise cluster configuration.

  1. Verify health

    Successful setup can be verified by checking the Master endpoint. Navigate to <load balancer-dns>/health, for example https://lb.production.example.com/health. Assuming the Conjur cluster load balancer has been configured correctly, you should see an output similar to the following:

     
    {
      ...
      "database": {
      ...
        "replication_status": {
          "pg_stat_replication": [
            {
              "usename": "lb.production.example.com",
              "application_name": "node2.production.example.com",
              "client_addr": "10.10.0.2",
              ...
              "sync_state": "sync",
              ...
            },
            {
              "usename": "lb.production.example.com",
              "application_name": "node3.production.example.com",
              "client_addr": "10.10.0.3",
              ...
              "sync_state": "potential",
              ...
            },
            {
              "usename": "lb-follower.production.example.com",
              "application_name": "node4.production.example.com",
              "client_addr": "10.10.1.4",
              ...
              "sync_state": "async",
              ...
            }
          ],
          ...
        }
      },
      "ok": true
    }

    Check for a value of "ok": true at the bottom.

    This page provides some additional information about the state of the cluster and includes the following keys:

    Key

    Explanation

    "sync_state": "sync"

    This node handles synchronous replication.

    "sync_state": "potential"

    This node can take over synchronous replication if the current synchronous node fails.

    "sync_state": "async"

    This node is either a Follower, or is a disaster recovery Standby (which cannot handle synchronous replication).

  2. Validate UI

    Navigate to the load balancer URL, for example, https://lb.production.example.com/ui, and log in with the username (admin) and the password that you configured above.

Step 5: Optional - Configure auto-failover

In an auto-failover event, if the Conjur Enterprise Master becomes unavailable, auto-failover enables the automatic promotion of a Standby to Master.

To add auto-failover onto the Conjur cluster, see Configure Auto-Failover.

Troubleshoot the setup

To troubleshoot the Master or Standby setup, add the --debug flag to the evoke configure command. For more information, see Troubleshoot Conjur Enterprise Configuration (Docker).