About this Presentation

most insane CI infrastructure I've ever been a part of

— Alex Gaynor

About this Presentation

• The OpenStack development process
• Project Gating: Preemptive CI
• Zuul: a system to implement gating
• Using Gearman to distribute work for Zuul

OpenStack

Is open source software for building private and public clouds.

• http://openstack.org

Projects

Contributors

• Unaffiliated individuals
• Commercial entities
• Nonprofit organizations
• National and local governments
• Number, quality, and area of contributions can change daily

Consistent Tooling

• Minimize meta-development
• Process divergence == wasted developer time
• Lowers onboarding time
• Consolidate tool development
• Minimize project-specific weird build crud

Project Gating

• Ensures Code Quality
• Protects developers
  • Devs always start from working code
• Protects tree
  • Bad code doesn't land
• Egalitarian
  • Process is the same for everyone
  • Process is transparent
  • Process is automated

Everything Is Automated

Process Flow

Gerrit

• Developed by Google for Android
• Stand-alone patch review system
• Integration points: hooks, JSON queries, event-stream
• Extensible review categories, default: Verified, Code-Review

States of a Patch

• Code Submitted
• Code Verified
• Code Reviewed
• Code Accepted
• Code Merged

Types of Gerrit Triggers

• Patchset uploaded
• Change merged
• Comment added (review state)
• Ref updated (branches, tags, etc)

Job Feedback

Jenkins

Types of jobs:

• Gate tests
  • unit tests
  • integraton tests
  • code style
• Post-merge automation
  • docs
  • tarballs
  • pypi
• Jobs under development
  • experimental
  • silent
  • non-voting
• Periodic jobs
  • bitrot checks
  • image updates
  • very long tests

Zuul

• A general purpose trunk gating system
• Interfaces with Gerrit and Jenkins, or other systems
• Flexible configuration allows for many kinds of project automation
• Allows parallel testing of serialized changes

Speculative Execution

• Serialize changes across all projects
• Speculative execution of tests
• Run in parallel in order triggered
• Assume success
• Start over on failure

Zuul Simulation

Check Pipeline

pipelines:
  - name: check
    manager: IndependentPipelineManager
    precedence: low
    trigger:
      gerrit:
        - event: patchset-created
    success:
      gerrit:
        verified: 1
    failure:
      gerrit:
        verified: -1

Gate Pipeline

pipelines:
  - name: gate
    manager: DependentPipelineManager
    precedence: high
    trigger:
      gerrit:
        - event: comment-added
          approval:
            - approved: 1
    start:
      gerrit:
        verified: 0
    success:
      gerrit:
        verified: 2
        submit: true
    failure:
      gerrit:
        verified: -2

Post-Merge Pipeline

pipelines:
  - name: post
    manager: IndependentPipelineManager
    trigger:
      gerrit:
        - event: ref-updated
          ref: ^(?!refs/).*$

Experimental Pipeline

pipelines:
  - name: experimental
    precedence: low
    trigger:
      gerrit:
        - event: comment-added
          comment_filter: (?i)^\s*check experimental\s*$
    success:
      gerrit:
        force-message: true
    failure:
      gerrit:
        force-message: true

Silent Pipeline

pipelines:
  - name: silent
    manager: IndependentPipelineManager
    trigger:
      gerrit:
        - event: patchset-created

Release Pipelines

pipelines:
  - name: release
    manager: IndependentPipelineManager
    precedence: high
    trigger:
      gerrit:
        - event: ref-updated
          ref: ^refs/tags/([0-9]+\.)+[0-9]+$

  - name: pre-release
    manager: IndependentPipelineManager
    precedence: high
    trigger:
      gerrit:
        - event: ref-updated
          ref: ^refs/tags/([0-9]+\.)+[0-9]*(alpha|beta|candidate|rc|a|b|c|r|g)[0-9]*$

Periodic Pipeline

  - name: periodic-stable
    description: Periodic checks of the stable branches.
    manager: IndependentPipelineManager
    precedence: low
    trigger:
      timer:
        - time: '1 6 * * *'
    failure:
      smtp:
        from: [email protected]
        to: [email protected]
        subject: 'Stable check of {change.project} failed'

Zuul Project Configuration

projects:
  - name: openstack/nova
    check:
      - gate-nova-pep8
      - gate-nova-docs
      - gate-nova-python27
      - gate-tempest-devstack-vm-full
    gate:
      - gate-nova-pep8
      - gate-nova-docs
      - gate-nova-python27
      - gate-tempest-devstack-vm-full
    experimental:
      - gate-devstack-vm-cells
    silent:
      - gate-tempest-devstack-vm-large-ops
    post:
      - nova-branch-tarball
      - nova-coverage
      - nova-docs
      - nova-upstream-translation-update
    pre-release:
      - nova-tarball
    release:
      - nova-tarball
      - nova-docs
    periodic:
      - nova-propose-translation-update
      - periodic-nova-python27-stable-folsom
      - periodic-nova-python27-stable-grizzly

Zuul Change Queues

Projects with shared jobs automatically get a shared queue:

projects:
  - name: openstack/nova
    gate:
      - gate-nova-python27
      - gate-tempest-devstack-vm

  - name: openstack/glance
    gate:
      - gate-glance-python27
      - gate-tempest-devstack-vm

Status View

Launcher: Gearman

• Workers register with Gearman
• Zuul submits build requests to Gearman
• Gearman distributes jobs to workers
• Meta-jobs:
  • stop job
  • set description

Triggers / Reporters

• Gerrit trigger/reporter
• Periodic trigger
• SMTP reporter
• Others in future

Development / Contributing

• Launchpad: https://launchpad.net/zuul
• Git: https://git.openstack.org/cgit/openstack-infra/zuul
• Docs: http://ci.openstack.org/zuul/
• Freenode: #openstack-infra
• Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/zuul/