Tutorial

In this tutorial, we’ll describe the simplest HA Jenkins setup, which creates a baseline when we later discuss other modes of deployment. Here, we deploy HA Jenkins in the following configuration:

  • An NFS server that hosts $JENKINS_HOME. We assume this is available already, and we will not discuss how one would set this up. (We’ll call this machine sierra.)
  • Two linux systems that form a Jenkins HA cluster, by running one JVM each (we’ll call them 'alpha' and 'bravo'). In this tutorial, those two machines need to be on the same local network.
  • One floating IP address (we’ll call this 1.2.3.4). Alpha and bravo take this IP address while each is acting as the primary, thereby ensuring that users can always access Jenkins through the DNS name that’s associated with this IP address.

First, install Jenkins Enterprise packages to alpha and bravo. You need to install both the jenkins package (for the Jenkins Enterprise by CloudBees itself), and the jenkins-ha-monitor package, which is the Jenkins Enterprise by CloudBees HA monitor tool.

Tip

Linux packages for jenkins-ha-monitor are not included in all versions of the Jenkins Enterprise repository. If yours is missing this package, or you are using a non-Linux system, see the section called “Jenkins Enterprise HA monitor tool” for information on direct download.

Choose the appropriate debian/redhat/openSUSE package format depending on the type of your distribution. Upon installation, both instances of Jenkins will start running. Stop them by issuing /etc/init.d/jenkins stop, while we work on the HA setup.

Let’s say sierra exports the /jenkins directory that hosts $JENKINS_HOME. Log on to alpha and mount this directory.

$ mount -t nfs -o rw,hard,intr sierra:/jenkins /var/lib/jenkins

/var/lib/jenkins is chosen to match what the Jenkins Enterprise packages use as $JENKINS_HOME. If you change them, update /etc/default/jenkins (on Debian) or /etc/sysconfig/jenkins (on RedHat and SUSE) to have $JENKINS_HOME point to the correct directory.

To make this mount automatically happen, update your /etc/fstab by adding the following entry:

sierra:/jenkins nfs rw,hard,intr 0 2

Repeat this mount setup on bravo, and ensure that both alpha and bravo see the same data. (For example, touch a from alpha, and make sure ls from bravo will see it. Make sure the uid and the gid appear the same on alpha and bravo.)

Boot Jenkins on alpha and bravo by issuing /etc/init.d/jenkins start. Now Jenkins Enterprise boots up in a two-node HA cluster. Access http://alpha:8080/ and http://bravo:8080/. One will serve the familiar Jenkins UI, and the other will tell you that it’s acting as a stand-by node. In the Jenkins UI, go to "Manage Jenkins" then click "High Availability Status" and make sure two nodes are listed as members. You can kill the primary JVM (for example by kill -9 PID) while watching the log file via tail -f /var/log/jenkins/jenkins.log, and you’ll see the stand-by node take over the primary role.

Finally, we set up a monitoring service to ensure that the floating IP address gets assigned to the system that’s hosting the primary. To do this, log on to alpha and install the jenkins-ha-monitor package.

This monitoring program watches Jenkins as root, and when the role transition occurs, it’ll execute the promotion script or the demotion script. In this tutorial, we’ll make these scripts assign/release the floating IP address.

In /etc/jenkins-ha-monitor/promotion.sh, write the following script:

#!/bin/sh
# assign the floating IP address 1.2.3.4 as an alias of eth1
ifconfig eth1:100 1.2.3.4

Similarly, in /etc/jenkins-ha-monitor/demotion.sh, write the following script:

#!/bin/sh
# release the floating IP address
ifconfig eth1:100 down

eth1:100 needs to be your network interface name followed by an unique alias ID (see "Linux IP aliasing" for more details.) Now that the configuration file is updated, restart the JA monitoring service by running /etc/init.d/jenkins-ha-monitor restart. Access the "High Availability Status" in the "Manage Jenkins" section from the web UI to verify that the monitoring service is recognized as a part of the cluster. Run ifconfig to verify that the virtual IP address is assigned to the system that’s hosting the primary JVM.

Congratulations, now you have a highly-available Jenkins!

Using haproxy as a reverse proxy

Let’s expand on this setup further by introducing an external load balancer / reverse proxy that receives traffic from users, then direct them to the active primary JVM. Compared to IP aliasing, this is more complex, but it allows two nodes that aren’t in the same subnet to form a cluster, and you can set this up without having a root access.

haproxy can be installed on most Linux systems via native packages, such as apt-get install haproxy or yum install haproxy. For Jenkins Enterprise HA, the configuration file (normally /etc/haproxy/haproxy.cfg) should look like the following:

# this section is a stock setting
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        maxconn 4096
        user haproxy
        group haproxy

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        redispatch
        maxconn 2000
        contimeout      1500
        clitimeout      500000
        srvtimeout      5000

# this block specifies a Jenkins Enterprise HA cluster
listen application 0.0.0.0:80
  balance roundrobin
  option httpchk /ha/health-check
  server alpha alpha:8080 check
  server bravo bravo:8080 check

# monitor port
listen status 0.0.0.0:8081
  stats enable
  stats uri /
>

The global and the defaults section are stock settings. The part that you need to add is the "listen application" block, which tells haproxy to forward traffic to two servers alpha and bravo, and periodically check their health by sending a GET request to /ha/health-check. Stand-by nodes do not respond positively to this health check, while real Jenkins will, and that’s how haproxy will know which of the two machines to send the traffic to.

The second "listen status" section allows you to monitor haproxy by accessing port 8081. This is handy when you want to understand how haproxy is behaving.