Scaling Your Jenkins Jobs
Jenkins Job Builder

Khai Do <[email protected]>

Team

Openstack Infrastructure Team
IRC: #openstack-infra on freenode.net
Email: [email protected]

OpenStack

Openstack is open source software for building private and public clouds

CI System

Storyboard
Git
Gerrit
Zuul*
Gearman
Jenkins (with jenkins-job-builder* and gearman-plugin*)
Devstack*
Nodepool*

* Started by the OpenStack Infrastructure team

CI Workflow

CI Workflow

OpenStack
Documentation
Translations
Operations
Utilities and tools

CI Scale

~500 Git Projects
~5,000 Jenkins Jobs
~10 Jenkins Masters
~1,000 Slaves
~24,000 Jobs per day

CI Challenges

Git Projects (nova, swift, cinder, etc..)
Project Branches (master, stable, feature, etc..)
Jobs (pep8, doc, test, deploy, etc..)
Job Configurations (multiple backends, multiple clouds, multiple deploy sites, etc..)
CI Pipelines (check, gate, release, periodic, etc..)

CI Challenges

Many, Many Jobs...
nova-py27-unit-master
nova-py27-perf-master
nova-py26-unit-stable-1.0
nova-py26-perf-stable-1.0
nova-py34-..

Openstack Jobs Over Time

Jobs Growth

Problems

XML difficult to maintain
Copy and pasting of jobs
Divergent jobs
Job definitions not under source control
No way to mimic jobs with similar configurations
1 Job : 1 XML file
Difficult to create, maintain and automate

Jenkins Job Builder

Command line tool to create and manage jenkins jobs
Job definitions in yaml
Templates to define multiple jobs in one file
Supported on Python 2.7-3.4
Open source
Active development

Openstack Job Definitions

Jobs Growth

How we use it

All jobs are managed by JJB (Jenkins is view only)
Jobs definitions created in yaml files
Job creation/changes go thru Gerrit code review
JJB creates multiple jobs from one job definition
Puppet instructs JJB to deploy and update jobs on Jenkins

Demo

Install

pip install jenkins-job-builder

Run

jenkins-jobs --conf my_jenkins.ini update simple.yaml

Simple Job

Simple Job

Templates

Run JJB

Templates

Benefits

One common definition for multiple jobs (efficient job creation)
Define common parameters for all jobs in one place (easier to maintain jobs)
Updates to multiple jobs can be done with one change (update scm polling, update node targets)
Less snowflakes and less time spent debugging jobs
Job definitions are in SCM
Job definitions are easy to read and update
Easier and more efficient to create and maintain jobs
Developers empowered to own their Jenkins jobs

Features

Works with Jenkins REST Api
Default definitions to reuse parameters
Macro definitions to reuse plugins
Template definitions to reuse jobs
Projects to group related jobs
Job groups to organize related job templates
Test jobs before deploying
Automated job removal
Cache for idempotent job updates
Unit tests (84% coverage)
Documented

Future Improvements

Jenkins folders support
Validation of yaml metadata
Improve performance of yaml to xml parsing
Improve performance of Job uploads
Refactor to be usuable API
Additional Jenkins plugin support
Contributions Welcome!

References

Lingo

Jenkins Jobs: aka Jenkins Projects
JJB Definitions: yaml metadata for the jobs
JJB parsing: Transform JJB yaml to Jenkins xml file
JJB modules: JJB python code to represent Jenkins plugins
JJB job: definition of a specific Jenkins job
JJB templates: a generic definition of a jenkins job
JJB job groups: a group of related JJB templates
JJB projects: a group of related JJB job groups

Logistics

GIT: http://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/
Pypi: https://pypi.python.org/pypi/jenkins-job-builder
Documentation (with examples): http://ci.openstack.org/jenkins-job-builder/
More Examples: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/