Upgrade an Auto-Failover Cluster

This section describes how to upgrade a DAP auto-failover cluster.

The basic approach is to upgrade each Standby and then destroy the current Master. Auto-failover takes over, promoting one of the Standbys to be the new Master. Then you can reprovision the old Master as a Standby and add it back into the auto-failover cluster.

Example configuration

The examples in the upgrade steps assume a basic DAP cluster with four nodes, implemented on four different servers. (Server in this case is defined as a Docker host, such as a virtual machine.) The four nodes are a Master, two Standbys, and one Follower. For reference, here are the configurations for the four nodes.

Original Master: server-1

Property

Value

Server name

server-1

Container name

conjur

Domain name of DAP Server

conjur-master-1.mycompany.com

Configured as

Originally: Master

After upgrading: Standby

 

Original Standby: server-2

Property

Value

Server name

server-2

Container name

conjur

Domain name of DAP Server

conjur-master-2.mycompany.com

Configured as

Originally: Synchronous-Standby

After upgrading: Standby

 

Original Standby: server-3

Property

Value

Server name

server-3

Container name

conjur

Domain name of DAP Server

conjur-master-3.mycompany.com

Configured as

Originally: Asynchronous-Standby

After upgrading: Master

 

Original Follower: server-4

Property

Value

Server name

server-4

Container name

conjur

Domain name of DAP Server

conjur-follower-1.mycompany.com

Configured as

Follower

Upgrade steps

All steps in this procedure use the container name conjur, with the assumption that each container is running on a different server. Each example identifies the server name where the command is issued.

Use the following steps to upgrade an auto-failover cluster.

  1. Identify a Standby that you want to upgrade first. It will eventually become the new Master in your cluster. You need to know the server where the Standby is running and the FQDN of that Standby.

    We recommend choosing a Standby that is replicating asynchronously. This allows the synchronously replicated Standby to continue to run for a while during the beginning of the upgrade process. From our example configuration, we chose server-3.

  2. On all Standbys and Followers, use evoke to stop replication:

     
    evoke replication stop

    For example:

     
    #On server-2/server-3/server-4:
    $ docker exec conjur evoke replication stop
  3. On the Master, create Standby seed files. A seed file contains encryption keys and certificates along with their associated private keys. If your organization saved previously generated Standby seed files in a secure location and certificates have not changed, you may reuse the old Standby seed file.

     

    It is important to create seed files using the same FQDN as the existing Standbys. Otherwise, you will need to update the cluster policy and your DNS server.

     
    evoke seed standby <standby-FQDN> <master-FQDN> > standby-seed-file 

    For example:

     
    #On server-1
    $ docker exec conjur evoke seed standby conjur-master-3.mycompany.com conjur-master-1.mycompany.com > standby-seed.tar
  4. Copy the seed file (or otherwise make it available) to the Standby's server.

  5. On the Master, communicate with etcd to remove the Standby from the etcd cluster.

     
    $ evoke cluster member remove <standby-FQDN>

    For example: 

     
    $ docker exec conjur evoke cluster member remove conjur-master-3.mycompany.com
  6. On the Standby server, stop the existing Standby and create a new one. Here are the steps: 

    1. Stop the existing Standby container.

       
      #On server-3:
      $ docker stop conjur
    2. Create a new DAP container that runs the new DAP version. See Install & start a Conjur appliancefor options and descriptions. The following example pulls an image for DAP v5.2.3 from a local docker registry.

       
      #On server-3
      $ docker run --name conjur-new -d --restart=always --security-opt seccomp:<profile> -p "443:443" -p "5432:5432" -p "1999:1999" registry2.itci.conjur.net/conjur-appliance:5.2.3
    3. Unpack the seed file on this node.

       
      #On server-3
      $ docker exec conjur-new evoke unpack seed standby-seed.tar
    4. Configure the new container as a Standby.

       
      #On server-3
      $ docker exec conjur-new evoke configure standby
  7. On the Master, communicate with etcd, notifying it about the new node about to join the current cluster.

     
    $ docker exec conjur evoke cluster member add <standby-node-FQDN>

    For example:

     
    $ docker exec conjur evoke cluster member add conjur-master-3.mycompany.com
  8. On the Standby, communicate with etcd, re-enrolling the Standby into the cluster. This command requires all of the details about the cluster.

     
    evoke cluster enroll --reenroll -n <node-FQDN> -m <master-node-FQDN> <cluster-name>

    where <cluster-name> is the last component in the cluster policy resource id. The resource id is conjur/cluster/my-cluster-policy-name. For example:

     
    $ evoke cluster enroll --reenroll -n conjur-master-3.mycompany.com -m conjur-master-1.mycompany.com conjur-cluster
  9. Repeat steps 4 through 8, above, for each Standby in the cluster.

    Here are the steps for the second server in our example.

     
    # Step 4. Copy the appropriate seed file to server-2.
    # Step 5. On the Master, communicate with etcd to remove the Standbyfrom the etcd cluster. On server-1: 
    $ docker exec conjur evoke cluster member remove conjur-master-2.mycompany.com
    # Step 6. Stop the Standby container, create a new container with the new software image, unpack the seed file, and configure the node as a Standby. On server-2:
    $ docker stop conjur
    $ docker run --name conjur-new -d --restart=always --security-opt seccomp:<profile> -p "443:443" -p "5432:5432" -p "1999:1999" registry2.itci.conjur.net/conjur-appliance:5.2.3
    $ docker exec conjur-new evoke unpack seed standby-seed-2.tar
    $ docker exec conjur-new evoke configure standby
    # Step 7. On the Master, communicate with etcd to notify about the new node that will join the cluster. On server-1: 
    $ docker exec conjur evoke cluster member add conjur-master-2.mycompany.com
    # Step 8. On the Standby, communicate with etcd, re-enrolling the Standby into the cluster.
    $ evoke cluster enroll --reenroll -n conjur-master-2.mycompany.com -m conjur-master-1.mycompany.com conjur-cluster

     

     

    At this point, all of the Standbys are upgraded and running the new DAP version. The auto-failover cluster is healthy.

  10. On the server hosting the Master, stop the Master container. This triggers auto-failover.

     
    #On server-1
    $ docker stop conjur

    When auto-failover is complete, you have an upgraded cluster minus one Standby.

  11. Reprovision the original Master as a Standby. Be sure to reuse the original FQDN for this node.

     

    The following steps are almost the same as those used to reprovision all of the other Standbys, with one exception:

    The original Master was removed from the etcd cluster as part of auto-failover processing, so the evoke cluster remove command is not needed here.

    The steps to reprovision the old Master into an upgraded Standby are: 

    1. Start a new container running the new DAP image.

    2. Unpack the appropriate seed file in the new container.

    3. Configure the node as a Standby.

    4. On the new Master, communicate with etcd to add the node into the cluster.

    5. Switch context back to the new Standby. Re-enroll the node into the cluster.

    Here is an example command sequence that accomplishes all of the above steps.

     
    #On server-1
    $ docker run --name conjur-new -d --restart=always --security-opt seccomp:<profile> -p "443:443" -p "5432:5432" -p "1999:1999" registry2.itci.conjur.net/conjur-appliance:5.2.3
    $ docker exec conjur-new evoke unpack seed standby-seed-1.tar
    $ docker exec conjur-new evoke configure standby
    #On server-3 (the new Master)
    $ docker exec conjur evoke cluster member add conjur-master-1.mycompany.com
    #On server-1
    $ evoke cluster enroll --reenroll -n conjur-master-1.mycompany.com -m conjur-master-3.mycompany.com conjur-cluster
  12. Verify the health of all nodes. See Health Check.

  13. Remove the old DAP containers.

     
    #On server-1/server-2/server-3:
    docker rm conjur
  14. Redeploy each Follower to use the new DAP image. On each Follower, the steps are: 

    1. Switch context (log on) to the Follower server.

    2. Stop the Follower container.

    3. Create a new container using the new DAP image.

    4. Unpack the Follower seed file onto the new container.

    5. Configure the container as a Follower.

    6. Remove the old container.

    Here is an example command sequence that accomplishes all of the above steps.

     
    #On server-4
     
    $ docker stop conjur
    $ docker run --name conjur-new -d --restart=always --security-opt seccomp:<profile> -p "443:443" registry2.itci.conjur.net/conjur-appliance:5.2.3
    $ docker exec conjur-new evoke unpack seed follower-seed-1.tar
    $ docker exec conjur-new evoke configure follower
    $ docker rm conjur
  15. Delete seed files after use. Alternatively, you may choose to store seed files in a secure way.

     

    Seed files contain sensitive information. We recommend deleting these files. You can regenerate seed files on the Master at any time.

  16. LDAP Sync: If you are using the LDAP sync service, re-enabled and start the service.

The upgrade process is complete. All nodes in the cluster are running a new DAP version. The Master is running on server-3. The Standbys are running on server-1 and server-2. The cluster chooses which one is synchronous.

 
TrueApplication Access ManagerDynamic Access Provider10.10