What's OpenStack?

CI Workflow

Why use it?

• Reliable: production ready, handling thousand of tests daily
• Flexible: can be easily configurated to follow your workflow
• Easily scalable: design allows to add servers depending on load
• Uses known tools: Gerrit, Jenkins
• Supported by the OpenStack community

Requirements to start

• You need a cloud: our tool is built in top of OpenStack
• A service account for your CI systems into that cloud(s)
• Optional: service account for Jenkins nodes
• Puppet is hostname based: a domain for your servers
• A git repository to store your code

CI system overview

• Full code on: https://git.openstack.org/openstack-infra/system-config
• Based on puppet
• Need a puppet master and puppet nodes for each service
• Minimal services needed: ci-puppetmaster, gerrit, jenkins,
  nodepool, zuul, and specific slaves
• Using hiera puppet module to store configuration values: keys, certificates...

How to adapt system to your project

• manifests/site.pp: update domains for your servers there
  

  node 'review.openstack.org' {
    class { 'openstack_project::review':
      github_oauth_token      => hiera('gerrit_github_token', 'XXX'),
      github_project_username => hiera('github_project_username', 'username'),
      github_project_password => hiera('github_project_password', 'XXX'),
  ...

• https://git.openstack.org/cgit/openstack-infra/project-config: defines the shape of servers, rename it and adapt to your needs

Bootstrapping

• Install a puppetmaster, replace openstack_project with your own name:

  sudo su -
  git clone https://git.openstack.org/openstack-infra/system-config /opt/system-config/production
  /opt/system-config/production/install_puppet.sh
  apt-get install puppetmaster-passenger hiera hiera-puppet

  bash /opt/system-config/production/install_modules.sh
  echo $REAL_HOSTNAME > /etc/hostname
  service hostname restart
  puppet apply --modulepath='/opt/system-config/production/modules:/etc/puppet/modules'
  -e 'include openstack_project::puppetmaster'
  

• Bring up puppetboard
• Update hiera files according to your needs
• Add nodes for each of the services you are going to use

Configure gerrit

• Is our code review system
• List of projects and ACL for each project needs to be configured
• Can choose authentication type (LDAP, SSO)...
• Needs custom keys and certificates
• Can customize look & feel: logo, css
• CLA files needs to be adapted: are OpenStack specific
• Node needs at least 10gb to run

Configure zuul

• Is the scheduler in the OpenStack CI system
• Configuration relies on zuul project-config:zuul/layout.yaml file:
  
  • pipelines:
    

    - name: check
      manager: IndependentPipelineManager
      source: gerrit
      trigger:
        gerrit:
          - event: patchset-created
      success:
        verified: 1
      failure:
        verified: -1

  • projects:
    

    projects:
      - name: example/project
        check:
          - project-merge:
            - project-unittest
          

• Need 1GB node to run

Configure Jenkins Master(s)

• Only one master needed, but multi-master is recommended
• Configuration of jobs done using jenkins job builder, relying on project-config:jenkins/jobs:
  
  • macros:
    

    - builder:
        name: make-test
        builders:
          - shell: 'make test'
    

  • job templates:
    

    - job-template:
        name: '{name}-unit-tests'
        builders:
          - make-test
    

  • projects:
    

    - project:
        name: project-name
        jobs:
          - '{name}-unit-tests'
    

• Launch the node with a size larger than specified in jenkins.default

Configure Static slaves

• Two types of slaves: statically managed or dynamically managed by nodepool
• Reusable or single used defined per job in zuul layout.yaml:

  jobs:
    - name: ^.*$
      parameter-function: single_use_node
    - name: ^gate-.*$
      parameter-function: reusable_node
      

• Type of node defined in jenkins job builder:
  

  - project:
      name: test_project
      node: bare-trusty
      jobs:
        - python-jobs
  

• Static slaves can be added manually into Jenkins, dynamic slaves need nodepool

Main problems, improvements

• System not automated 100%, needs manual configuration
• Initial deployment and learning curve can be complicated
• Main engine is still related with Openstack configuration: gradual steps to separate it
• Trying to isolate all project-depending settings into openstack-project folder
• Some configurations hardcoded, moving to make every setting configurable

Development / Contributing

• Git: https://git.openstack.org/cgit/openstack-infra/system-config
  https://git.openstack.org/cgit/openstack-infra/project-config
• Docs: http://ci.openstack.org
• Freenode: #openstack-infra
• Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Questions

Contact

• Yolanda Robla Mota <[email protected]>

OpenStack CI Resources

• http://ci.openstack.org

These slides available at: http://docs.openstack.org/infra/publications/