This page explains the workflows a developer is expected to follow to implement the goals that are part of the Ceph lifecycle. It does not go into technical details and is designed to provide a high level view instead. Each chapter is about a given goal such as Merging bug fixes or features or Publishing point releases and backporting.
A key aspect of all workflows is that none of them blocks another. For instance, a bug fix can be backported and merged to a stable branch while the next point release is being published. For that specific example to work, a branch should be created to avoid any interference. In practice it is not necessary for Ceph because:
This ad-hoc approach implies the workflows are changed on a regular basis to adapt. For instance, quality engineers were not involved in the workflow to publish dumpling point releases. The number of commits being backported to firefly made it impractical for developers tasked to write code or fix bugs to also run and verify the full suite of integration tests. Inserting quality engineers makes it possible for someone to participate in the workflow by analyzing test results.
The workflows are not enforced when they impose an overhead that does not make sense. For instance, if the release notes for a point release were not written prior to checking all integration tests, they can be commited to the stable branch and the result sent for publication without going through another run of integraiton tests.
Ceph hammer infernalis
Developer CDS CDS
Summit | |
| |
development | |
release | v0.88 v0.89 v0.90 ... | v0.95
--v--^----^--v---^------^--v- ---v----^----^--- 2015
| | | |
stable giant | | hammer
release v0.87 | | v0.94
| |
point firefly dumpling
release v0.80.8 v0.67.12
Four times a year, the development roadmap is discussed online during the Ceph Developer Summit. A new stable release (argonaut, cuttlefish, dumpling, emperor, firefly, giant, hammer ...) is published at the same frequency. Every other release is a long time support (dumpling, firefly, hammer, ...) which means point releases are published more often. In 2014 point releases (i.e. dumpling 0.67.11, dumpling 0.67.12 ...) are published up to eighteen months after the publication of a stable release. Once or twice a month, a development release is published.
The development branch is master and the workflow followed by all developers can be summarized as follows:
The developer is the author of a series of commits. The reviewer is responsible for providing feedback to the developer on a regular basis and the developer is invited to ping the reviewer if nothing happened after a week. After the reviewer is satisfied with the pull request, (s)he passes it to the tester. The tester is responsible for running teuthology integration tests on the pull request. If nothing happens within a month the reviewer is invited to ping the tester.
All bug reports and feature requests are in the issue tracker and the workflow can be summarized as follows:
Each team is responsible for a project:
The developer assigned to an issue is responsible for it. The status of an open issue can be:
The Sepia community test lab runs teuthology integration tests and the results are posted on pulpito and the ceph-qa mailing list.
The quality engineer is either a developer or a member of the QE team. There is at least one integration test suite per project:
and a many others such as
A release is prepared in a dedicated branch, different from the master branch.
The workflow expected of all developers to stabilize the release candidate is the same as the normal development workflow with the following differences:
A new stable release can be cut when:
Publishing a new stable release implies a risk of regression or discovering new bugs during the upgrade, no matter how carefully it is tested. The decision to cut a release must take this into account: it may not be wise to publish a stable release that only fixes a few minor bugs. For instance if only one commit has been backported to a stable release that is not a LTS, it is better to wait until there are more.
When a stable release reaches its end of life, it may be safer to recommend an upgrade to the next long term support release instead of proposing a new point release to fix a problem. For instance, the Dumpling v0.67.11 release has bugs related to backfilling which have been fixed in Firefly v0.80.x. A backport fixing these backfilling bugs has been tested in the draft point release Dumpling v0.67.12 but they are large enough to introduce a risk of regression. As Dumpling is approaching its end of life, users suffering from this bug can upgrade to Firefly to fix it. Unless users manifest themselves and ask for Dumpling v0.67.12, this draft release may never be published.
The person responsible for each role is:
The publication workflow of a development release is the same as preparing a new release and cutting it, with the following differences:
The publication workflow of the point releases is the same as preparing a new release and cutting it, with the following differences: