About this Presentation

• The OpenStack development process
• Project Gating: Preemptive CI
• Zuul: a system to implement gating with Jenkins
• Using the Jenkins API to integrate Jenkins into any system

OpenStack

Is open source software for building private and public clouds.

• http://openstack.org

Projects

Contributors

• Individual Contributors
• Commercial Entities
• Number, quality, and area of contributions can change daily

Vision

• Consistent Tooling
• Consistent Process
• Consistent Product
• Multiplier Effect

Consistent Tooling

• Minimize meta-development
• Process divergence == wasted developer time
• Lowers onboarding time
• Consolidate tool development
• Minimize project-specific weird build crud

Gated Trunk

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Everything Is Automated

Process Flow

• Code is written and locally tested in a virtualenv
• Code is submitted for code review to gerrit
• Code is run through patch-uploaded automated checks
• Code is peer-reviewed
• Code is accepted or rejected by core team
• Code is run through pre-merge automated checks
• Code is merged or rejected
• Code is run through post-merge analysis

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review

States of a Patch

• Code Submitted
• Code Verified
• Code Reviewed
• Code Accepted
• Code Merged

Types of Gerrit Triggers

• Patchset uploaded
• Change merged
• Comment added (review state)

Jenkins Feedback

Jenkins

Types of jobs:

• Check / Gate
  • unit tests
  • integraton tests
  • code style
• Post
  • docs
  • tarballs
  • pypi

Zuul

• A general purpose trunk gating system
• Interfaces with Gerrit and Jenkins
• Flexible configuration allows for many kinds of project automation
• Allows parallel testing of serialized changes

Speculative Execution

• Serialize changes across all projects
• Speculative execution of tests
• Run in parallel in order triggered
• Assume success
• Start over on failure

Zuul Simulation

Zuul Check Pipeline

pipelines:
  - name: check
    manager: IndependentPipelineManager
    trigger:
      - event: patchset-created
    success:
      verified: 1
    failure:
      verified: -1

Zuul Gate Pipeline

pipelines:
  - name: gate
    manager: DependentPipelineManager
    trigger:
      - event: comment-added
        approval:
          - approved: 1
    start:
      verified: 0
    success:
      verified: 2
      submit: true
    failure:
      verified: -2

Zuul Post-Merge Pipeline

pipelines:
  - name: post
    manager: IndependentPipelineManager
    trigger:
      - event: ref-updated
        ref: ^(?!refs/).*$

Other Pipelines We've Created

pipelines:
  - name: silent
    manager: IndependentPipelineManager
    trigger:
      - event: patchset-created

  - name: publish
    manager: IndependentPipelineManager
    trigger:
      - event: ref-updated
        ref: ^refs/tags/.*$

Zuul Project Configuration

projects:
  - name: openstack/nova
    check:
      - gate-nova-merge:
        - gate-nova-pep8
        - gate-nova-python26
        - gate-nova-python27
        - gate-tempest-devstack-vm
    gate:
      - gate-nova-merge:
        - gate-nova-pep8
        - gate-nova-python26
        - gate-nova-python27
        - gate-tempest-devstack-vm
    post:
      - nova-tarball
      - nova-docs

Zuul Change Queues

Projects with shared jobs automatically get a shared queue:

projects:
  - name: openstack/nova
    gate:
      - gate-nova-merge:
        - gate-nova-python27
        - gate-tempest-devstack-vm

  - name: openstack/glance
    gate:
      - gate-glance-merge:
        - gate-glance-python27
        - gate-tempest-devstack-vm

Status View

Jenkins API

• Job and view lists
• Job configuration
• Build jobs
• Build descriptions
• Cancel builds
• Inspect and cancel queue
• Node configuration

Jenkins API: Launching

The python library jenkins allows for easy interaction:

from jenkins import Jenkins
jenkins = Jenkins('servername', 'username', 'apikey')
jenkins.build_job(job.name, parameters=params)

Abstracts simple GET/POST transactions.

Jenkins API: Notifications

Notification plugin provides for HTTP endpoints.

class JenkinsCallback(threading.Thread):
    def run(self):
        httpserver.serve(self.app, host='0.0.0.0', port='8001')

    def app(self, environ, start_response):
        request = Request(environ)
        start_response('200 OK', [('content-type', 'text/html')])
        if request.path == '/jenkins_endpoint':
            data = json.loads(request.body)
            build = data.get('build')
            phase = build.get('phase')
            status = build.get('status')
            url = build.get('full_url')
            number = build.get('number')
            params = build.get('parameters')
        return ['OK']

Development / Contributing

• Launchpad: https://launchpad.net/zuul
• GitHub: https://github.com/openstack-ci/zuul
• Docs: http://ci.openstack.org/zuul/
• Freenode: #openstack-infra
• Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Thanks!

These slides available at: https://github.com/openstack-ci/publications