In this tutorial, we’ll describe the simplest HA Jenkins setup, which creates a baseline when we later discuss other modes of deployment. Here, we deploy HA Jenkins in the following configuration:
- An NFS server that hosts
$JENKINS_HOME
. We assume this is available already, and we will not discuss how one would set this up. (We’ll call this machine sierra.) - Two linux systems that form a Jenkins HA cluster, by running one JVM each (we’ll call them 'alpha' and 'bravo'). In this tutorial, those two machines need to be on the same local network.
- One floating IP address (we’ll call this
1.2.3.4
). Alpha and bravo take this IP address while each is acting as the primary, thereby ensuring that users can always access Jenkins through the DNS name that’s associated with this IP address.
First, install Jenkins Enterprise packages to alpha and bravo. You need to install both the jenkins
package (for the Jenkins Enterprise by CloudBees itself), and the jenkins-ha-monitor
package, which is the Jenkins Enterprise by CloudBees HA monitor tool.
Tip
Linux packages for jenkins-ha-monitor
are not included in all versions of the Jenkins Enterprise repository. If yours is missing this package, or you are using a non-Linux system, see the section called “Jenkins Enterprise HA monitor tool” for information on direct download.
Choose the appropriate debian/redhat/openSUSE package format depending on the type of your distribution. Upon installation, both instances of Jenkins will start running. Stop them by issuing /etc/init.d/jenkins stop
, while we work on the HA setup.
Let’s say sierra exports the /jenkins
directory that hosts $JENKINS_HOME
. Log on to alpha and mount this directory.
$ mount -t nfs -o rw,hard,intr sierra:/jenkins /var/lib/jenkins
/var/lib/jenkins
is chosen to match what the Jenkins Enterprise packages use as $JENKINS_HOME
. If you change them, update /etc/default/jenkins
(on Debian) or /etc/sysconfig/jenkins
(on RedHat and SUSE) to have $JENKINS_HOME
point to the correct directory.
To make this mount automatically happen, update your /etc/fstab
by adding the following entry:
sierra:/jenkins nfs rw,hard,intr 0 2
Repeat this mount setup on bravo, and ensure that both alpha and bravo see the same data. (For example, touch a
from alpha, and make sure ls
from bravo will see it. Make sure the uid and the gid appear the same on alpha and bravo.)
Boot Jenkins on alpha and bravo by issuing /etc/init.d/jenkins start
. Now Jenkins Enterprise boots up in a two-node HA cluster. Access http://alpha:8080/
and http://bravo:8080/
. One will serve the familiar Jenkins UI, and the other will tell you that it’s acting as a stand-by node. In the Jenkins UI, go to "Manage Jenkins" then click "High Availability Status" and make sure two nodes are listed as members. You can kill the primary JVM (for example by kill -9 PID
) while watching the log file via tail -f /var/log/jenkins/jenkins.log
, and you’ll see the stand-by node take over the primary role.
Finally, we set up a monitoring service to ensure that the floating IP address gets assigned to the system that’s hosting the primary. To do this, log on to alpha and install the jenkins-ha-monitor
package.
This monitoring program watches Jenkins as root
, and when the role transition occurs, it’ll execute the promotion script or the demotion script. In this tutorial, we’ll make these scripts assign/release the floating IP address.
In /etc/jenkins-ha-monitor/promotion.sh
, write the following script:
#!/bin/sh # assign the floating IP address 1.2.3.4 as an alias of eth1 ifconfig eth1:100 1.2.3.4
Similarly, in /etc/jenkins-ha-monitor/demotion.sh
, write the following script:
#!/bin/sh # release the floating IP address ifconfig eth1:100 down
eth1:100
needs to be your network interface name followed by an unique alias ID (see "Linux IP aliasing" for more details.) Now that the configuration file is updated, restart the JA monitoring service by running /etc/init.d/jenkins-ha-monitor restart
. Access the "High Availability Status" in the "Manage Jenkins" section from the web UI to verify that the monitoring service is recognized as a part of the cluster. Run ifconfig
to verify that the virtual IP address is assigned to the system that’s hosting the primary JVM.
Congratulations, now you have a highly-available Jenkins!
Let’s expand on this setup further by introducing an external load balancer / reverse proxy that receives traffic from users, then direct them to the active primary JVM. Compared to IP aliasing, this is more complex, but it allows two nodes that aren’t in the same subnet to form a cluster, and you can set this up without having a root access.
haproxy
can be installed on most Linux systems via native packages, such as apt-get install haproxy
or yum install haproxy
. For Jenkins Enterprise HA, the configuration file (normally /etc/haproxy/haproxy.cfg
) should look like the following:
# this section is a stock setting global log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 user haproxy group haproxy defaults log global mode http option httplog option dontlognull retries 3 redispatch maxconn 2000 contimeout 1500 clitimeout 500000 srvtimeout 5000 # this block specifies a Jenkins Enterprise HA cluster listen application 0.0.0.0:80 balance roundrobin option httpchk /ha/health-check server alpha alpha:8080 check server bravo bravo:8080 check # monitor port listen status 0.0.0.0:8081 stats enable stats uri / >
The global and the defaults section are stock settings. The part that you need to add is the "listen application" block, which tells haproxy to forward traffic to two servers alpha
and bravo
, and periodically check their health by sending a GET request to /ha/health-check
. Stand-by nodes do not respond positively to this health check, while real Jenkins will, and that’s how haproxy will know which of the two machines to send the traffic to.
The second "listen status" section allows you to monitor haproxy by accessing port 8081. This is handy when you want to understand how haproxy is behaving.