Tempest Field Guide to CLI tests

What are these tests?

The cli tests test the various OpenStack command line interface tools to ensure that they minimally function. The current scope is read only operations on a cloud that are hard to test via unit tests.

Why are these tests in tempest?

These tests exist here because it is extremely difficult to build a functional enough environment in the python-*client unit tests to provide this kind of testing. Because we already put up a cloud in the gate with devstack + tempest it was decided it was better to have these as a side tree in tempest instead of another QA effort which would split review time.

Scope of these tests

This should stay limited to the scope of testing the cli. Functional testing of the cloud should be elsewhere, this is about exercising the cli code.

Example of a good test

Tests should be isolated to a single command in one of the python clients.

Tests should not modify the cloud.

If a test is validating the cli for bad data, it should do it with assertRaises.

A reasonable example of an existing test is as follows:

def test_admin_list(self):
    self.nova('list')
    self.nova('list', params='--all-tenants 1')
    self.nova('list', params='--all-tenants 0')
    self.assertRaises(subprocess.CalledProcessError,
                      self.nova,
                      'list',
                      params='--all-tenants bad')