OpenStack

Is open source software for building private and public clouds.

Openstack Projects

Programs/Horizontal Efforts

Contributors

• Unaffiliated individuals
• Commercial entities
• Nonprofit organizations
• National and local governments
• Number, quality, and area of contributions can change daily
• 1000+ active contributors

Developer Infrastructure

Openstack CI Preface

• Environment
  • Open Source
  • Python
  • High volume of changes
  • Not resource contrained
• Goals
  • Keep mainline development clean
  • Validate correctness of changes
  • Reduce regressions
  • Reduce feedback loop
  • Automate everything

System Administration is in Gerrit

• All system administration performed via code review
• Anyone on the internet can look at our changes
• Anyone can sign up for a code review account
• Anyone can write and propose changes
• Any changes to Gerrit are reviewable
• source
• reviews

CI Workflow

Gerrit

• Currently on version 2.4.x transitioning to 2.8.x
• Votes: verified, code-review, approved
• Hooks: integration with issue tracker
• Event-stream: integration with Zuul and Chat
• Command Line: auto create projects
• *Custom Work In Progress state

Process Flow

Git Review

• External Git subcommand
• Developers can easily incorporate code review into git workflow
• Zero-configuration
• Can be used for any project, being adopted by other projects

corvus@shiprock:~/git/neutron$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
 3 files changed, 28 insertions(+), 107 deletions(-)
 delete mode 100644 version.py

corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas:   0% (0/3)
remote:
remote: New Changes:
remote:   https://review.openstack.org/3072
remote:
To ssh://[email protected]:29418/openstack/neutron.git
 * [new branch]      HEAD -> refs/for/master/bug/916018

Gerrit Events

Feature Request

• Cross project dependencies
• Single page diffs
• Reviewable tagging
• Configurable columns
• Sortable columns
• Filtering reviews
• Grouping reviews
• Reduce the noise
• Multi-master

Challenges

• Testing effect of pre-merge change
• High volume of changes
• Keep mainline clean
• Keep feedback loop short
• Disparate hardware configurations

Zuul

• A general purpose trunk gating system
• Interfaces with Gerrit and Gearman/Jenkins
• Flexible configuration allows for many kinds of project automation
• Allows parallel testing of serialized changes

Bottlenecking

• Serialize changes across all projects
• Speculative execution of tests
• Run in parallel in order triggered
• Assume success
• Start over on failure

Zuul Simulation

Zuul Status

Zuul Reports

Zuul Jobs

Scaling Hardware Needs

• Different vendors have different combinations of configurations
• http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way
• Run their own Jenkins
• Consume Gerrit event stream
• Their Jenkins votes VRFY +1/-1
• Our Zuul votes VRFY +2/-2
• They can run their lab - we don't have to know anything about it

Third Party Reports

Multi-Master Jenkins

• Horizontal Scaling
• Redundancy
• Uses the Jenkins Gearman Plugin
• Masters: jenkins01.openstack.org, jenkins02.o.o, jenkins03.o.o, etc..
• 600+ test nodes

Jenkins Slaves

Templated Jobs

• TONS of consistent projects
• Manage everything through Git and code review
• Jenkins Job Builder
• Manage jobs as YAML files in Git
• Jobs updated via Jenkins API
• groupings of jobs supported

Elastic test nodes

• Nodepool
• Automate management of test nodes
• Registers nodes with Jenkins masters
• Uses Jenkins ZeroMQ events for build status
• Manages multiple openstack clouds

Elastic test nodes

Logs

• Massive amount of logs generated
  • Terabytes of compressed logs per development cycle
• Filtered and normalized by LogStash
• Recent logs indexed by ElasticSearch
  • Searchable through Kibana and ElasticSearch API
• Driving automatic failure classification

Query Logs

Elastic Recheck

Elastic Recheck

Elastic Recheck

Everything Is Automated

Thanks!

These slides available at: http://docs.openstack.org/infra/publications/