What's OpenStack?

Projects

Contributors

Top contributing companies in number of reviews for OpenStack Juno release

• Unaffiliated individuals, commercial entities,
  nonprofit organizations, national and local governments
• Number, quality, and area of contributions can change daily

CI Workflow

Gerrit Code review

• Code review system used in OpenStack
• Configuration points: projects, groups, users, acl
• Integration points: hooks, event-stream, REST API
• Help on configuration: jeepyb

Gerrit Code review

Jeepyb

• Collection of tools to manage Gerrit easier
• Easily manage projects and Launchpad/Github integrations (manage-projects call)
• Projects to be defined in a YAML file
  

  - project: example/gerrit
    description: Fork of Gerrit used by Example
  - project: openstack/project-name
    acl-config: /home/gerrit2/acls/project-name.config
    upstream: git://github.com/awesumsauce/project-name.git

Jeepyb ACLs

• To define roles for each project, have a config file with ACL for each project
• Based on the concept of Gerrit groups: Administrators, Registered Users, Anonymous users
• Using git refs syntax to describe permissions on projects and branches
• Follow syntax of Gerrit project config: https://gerrit-review.googlesource.com/Documentation/config-project-config.html

[access "refs/heads/*"]
label-Code-Review = -2..+2 group project-name-core
label-Workflow = -1..+1 group project-name-core

[access "refs/heads/proposed/*"]
label-Code-Review = -2..+2 group project-name-milestone
label-Workflow = -1..+1 group project-name-milestone

[receive]
requireChangeId = true
requireContributorAgreement = true

[submit]
mergeContent = true

Zuul

• General purpose trunk gating system
• Interfaces with Gerrit and Jenkins (Gearman)
• Two main components: scheduler and merger
• Allows parallel testing of serialized changes: speculative execution
• Pipeline oriented
• Flexible configuration: layout.yaml

Zuul pipelines

• Different types: check, gate, release, silent, experimental, periodic...
• Zuul watches events on Gerrit matching to the right pipeline
• When a match is found, runs the related jobs
• Pipeline syntax:
  

  pipelines:
    - name: check
        manager: IndependentPipelineManager
        trigger:
        gerrit:
          - event: patchset-created
        success:
          verified: 1
        failure:
          verified: -1

Zuul projects

• Project syntax:
  

  projects:
    - name: example/project
        check:
          - project-merge:
            - project-unittest
            - project-pep8
            - project-pyflakes
        gate:
          - project-merge:
            - project-unittest
            - project-pep8
            - project-pyflakes
        post:
          - project-publish

Jenkins

• CI system that runs tests and automated tasks
• Several types of jobs launched: test, docs, builds
• Controlled by Zuul, using Gearman plugin
• Gearman: protocol to send jobs between client, job server and job worker/s
• Configuration of jobs is managed by Jenkins Job Builder (jjb)
• Single-use and reusable slaves are used. Single-use nodes spinned by nodepool

Jenkins Job Builder

• Tool for configuring Jenkins jobs using YAML files stored in our repos
• jjb communicates with Jenkins using its API: Jenkins user needed
• Using templated jobs, to apply same jobs to tons of different projects:
  

  - job-template:
      name: 'gate-{name}-docs'
      builders:
        - shell: 'git checkout {branch_name}'
  

• jjb projects definition:
  

  - project:
     name: project-name
     branch_name: new_branch
     jobs:
       - gate-{name}-docs

• Grouping of jobs is supported:
  

  - job-group:
      name: '{name}-tests'
      jobs:
      - '{name}-unit-tests'
      - '{name}-perf-tests'
  

Jenkins Job Builder jobs

• Simply defining a job:
  

- job:
    name: foo-test
    project-type: freestyle
    builders:
      - make-test
    publishers:
      - archive

• Modules can be added to jobs: builders, publishers, etc...
• Using macros to define actions in Jenkins:
  

  - builder:
    name: make-test
    builders:
      - shell: 'make test'

Nodepool

• Service used to deploy and manage a pool of devstack images on the cloud
• Works with any OpenStack provider
• Once per day a new image is generated with cached devstack content
• Spins instances based on desired image on demand
• Can use disk-image-builder to build images
• Communicates with Zuul using gearman-servers for getting realtime demand
• Communicates with Jenkins to attach nodes using Jenkins API

Nodepool configuration

• Labels: types of nodes that should be created. Maps node types to images:
  

  labels:
    - name: my-trusty
      image: trusty
      min-ready: 2
      providers:
        - name: provider1
        - name: provider2

• Targets: list the Jenkins masters to which Nodepool should attach nodes:
  

  targets:
    - name: jenkins1
      jenkins:
        url: https://jenkins1.example.org/
        user: username
        apikey: key
        credentials-id: id

Nodepool providers

• List cloud provider to be used, and define image types for each one.
  Can use multiple providers to spin up the total instances needed:
  

  providers:
    - name: provider1
      username: 'username'
      password: 'password'
      auth-url: 'http://auth.provider1.example.com/'
      project-id: 'project'
      service-type: 'compute'
      service-name: 'compute'
      region-name: 'region1'
      max-servers: 96
      images:
        - name: trusty
          base-image: 'Trusty'
          min-ram: 8192
          setup: prepare_node.sh
          username: jenkins
          private-key: /var/lib/jenkins/.ssh/id_rsa
      

Automated management

• Strong need to use a config management system: Puppet (and Ansible) used, could be using others
• Puppet agent pulls puppet manifests and configuration from a centralized master
• Only one central and secure place to hold secure information: neded to store certs, keys and passwods
• Using hiera for systemwide configuration, allowing multiple environments to be defined (development, production), as long as configuration depending on facts
• Hiera values are simple key-value pairs stored on yaml in the defined path. Keys need to be referenced in every site.pp:
  

  node default {
    class { 'openstack_project::server':
      sysadmins => hiera('sysadmins', []),
    }
  }

Questions

Contact

• Yolanda Robla Mota <[email protected]>

OpenStack CI Resources

• http://ci.openstack.org

These slides available at: http://docs.openstack.org/infra/publications/