Components of Jenkins Enterprise HA

This section describes 3 building blocks of Jenkins Enterprise by CloudBees that you will combine into a deployment.

Jenkins Enterprise by CloudBees WAR file

As you saw in the tutorial, Jenkins Enterprise WAR file contains the necessary code to run Jenkins in a highly-available setup. The extra control logic in this WAR file cooperatively elect the "primary" JVM, and depending on the outcome of this election, members of a cluster starts/stops the Jenkins master in the same JVM. From the view point of the code inside Jenkins, this is as if Jenkins is started/stopped programmatically.

Jenkins Enterprise relies on JGroups for the underlying group membership service.

Network configuration

By default, Jenkins Enterprise uses TCP to communicate between members, with IP addresses and ports registered in a directory $JENKINS_HOME/jgroups (which all members must be able to write to). This can be changed by creating $JENKINS_HOME/jgroups.xml that describes the JGroups protocol stack configuration XML. See the JBoss Clustering documentation for the format of this file, as well as typical configuration tips and trouble-shooting.

Jenkins Enterprise HA proxy WAR

To allow you to run OSS Jenkins WAR in the HA configuration, a separate "proxy WAR" is provided. This optional file can be downloaded from the ha-proxy section of the download site. The proxy WAR is a proxy to real Jenkins WAR file; you drop it in place of jenkins.war, and instead of acting as Jenkins by itself, it expects you to have the real thing in $JENKINS_HOME/jenkins.war. The proxy WAR only controls the cluster control logic, and when it decides that it needs to act as the primary JVM, it’ll load $JENKINS_HOME/jenkins.war into memory and runs that program.

This setup allows you to run newer versions of mainline Jenkins releases (for example to benefit from the latest and greatest features), or your custom build of Jenkins.

With the rpm/deb packages of Jenkins or Jenkins Enterprise, the easiest way to use this feature is to modify /etc/init.d/jenkins (for RPM) and /etc/default/jenkins (for DEB) and update JENKINS_WAR variable to point to the proxy war. For other deployments, move the real jenkins.war into $JENKINS_HOME, then put ha-proxy.war as jenkins.war.

The proxy WAR uses $JENKINS_HOME in the same ways as Jenkins Enterprise WAR to pick up the network configuration and other parameters.

Jenkins Enterprise HA monitor tool

Depending on the way you deploy the high-availability setup, some system level setup/teardown activities need to take place when the primary JVM changes. In the tutorial, we needed to run ifconfig to move the floating IP address. Other examples include DRBD mode change, updates to the front-end reverse proxy, or sending out an e-mail.

Because this needs to be executed when the primary Jenkins JVM becomes unresponsive, such setup/teardown scripts can only be reliably triggered from outside Jenkins. Those scripts also normally require root privileges to run. The Jenkins Enterprise HA monitor tool is a small background application that provides this role. It is an optional component that should be installed only when necessary. It can be downloaded from the jenkins-ha-monitor section of the download site.

The monitor tool is packaged into a single jar file that can be executed as java -jar jenkins-ha-monitor.jar. It is also packaged as the jenkins-ha-monitor RPM/DEB packages for an easier installation. Its 3 defining options are as follows:

  • The -home option that specifies the location of $JENKINS_HOME. The monitor tool picks up network configuration and other important parameters from here, so it needs to know this location.
  • The -host-promotion option that specifies the location of the promotion script, which gets executed when the primary Jenkins JVM moves in from another system into this system as a result of election. In native packages, this file is placed at /etc/jenkins-ha-monitor/promotion.sh.
  • The -host-demotion option that specifies the demotion script, which is the opposite of the promotion script and gets executed when the primary Jenkins JVM moves from this system into another system. In native packages, this file is placed at /etc/jenkins-ha-monitor/demotion.sh.

Promotion and demotion scripts need to be idempotent, in the sense that the monitor tool may run the promotion script on an already promoted node, and the demotion script on an already demoted node. This can happen, for example, when a power outage hits a stand-by node and when it comes back up. The monitor tool runs the demotion script again on this node, since it cannot be certain about the state of the node before the power outage.

Run the tool with the -help option to see the complete list of available options.