System Administration

Our infrastructure is code and contributions to it are handled just like the rest of OpenStack. This means that anyone can contribute to the installation and long-running maintenance of systems without shell access, and anyone who is interested can provide feedback and collaborate on code reviews.

The configuration of every system operated by the infrastructure team is managed by Puppet in a single Git repository:

https://github.com/openstack-infra/config

All system configuration should be encoded in that repository so that anyone may propose a change in the running configuration to Gerrit.

Making a Change in Puppet

Many changes to the Puppet configuration can safely be made while only performing syntax checks. Some more complicated changes merit local testing and an interactive development cycle. The config repo is structured to facilitate local testing before proposing a change for review. This is accomplished by separating the puppet configuration into several layers with increasing specificity about site configuration higher in the stack.

The modules/ directory holds puppet modules that abstractly describe the configuration of a service. Ideally, these should have no OpenStack-specific information in them, and eventually they should all become modules that are directly consumed from PuppetForge, only existing in the config repo during an initial incubation period. This is not yet the case, so you may find OpenStack-specific configuration in these modules, though we are working to reduce it.

The modules/openstack_project/manifests/ directory holds configuration for each of the servers that the OpenStack project runs. Think of these manifests as describing how OpenStack runs a particular service. However, no site-specific configuration such as hostnames or credentials should be included in these files. This is what lets you easily test an OpenStack project manifest on your own server.

Finally, the manifests/site.pp file contains the information that is specific to the actual servers that OpenStack runs. These should be very simple node definitions that largely exist simply to provide private date from hiera to the more robust manifests in the openstack_project modules.

This means that you can run the same configuration on your own server simply by providing a different manifest file instead of site.pp.

As an example, to run the etherpad configuration on your own server, start by cloning the config Git repo:

git clone https://github.com/openstack-infra/config

Then copy the etherpad node definition from manifests/site.pp to a new file (be sure to specify the FQDN of the host you are working with in the node specifier). It might look something like this:

# local.pp
node 'etherpad.example.org' {
  class { 'openstack_project::etherpad':
    database_password       => 'badpassword',
    sysadmins               => '[email protected]',
  }
}

Then to apply that configuration, run the following:

cd config
bash install_puppet.sh
bash install_modules.sh
puppet apply -l manifest.log --modulepath=modules:/etc/puppet/modules local.pp

That should turn the system you are logged into into an etherpad server with the same configuration as that used by the OpenStack project. You can edit the contents of the config repo and iterate as needed. When you’re ready to propose the change for review, you can propose the change with git-review. See the Gerrit Workflow wiki article for more information.

Adding a New Server

To create a new server, do the following:

• Add a file in modules/openstack_project/manifests/ that defines a class which specifies the configuration of the server.

• Add a node entry in manifests/site.pp for the server that uses that class.

• If your server needs private information such as password,s use hiera calls in the site manifest, and ask an infra-core team member to manually add the private information to hiera.

• You should be able to install and configure most software only with puppet. Nonetheless, if you need SSH access to the host, add your public key to modules/openstack_project/manifests/users.pp and include a stanza like this in your server class:

  realize (
     User::Virtual::Localuser['USERNAME'],
  )

• Add an RST file with documentation about the server in doc/source and add it to the index in that directory.

SSH Access

For any of the systems managed by the OpenStack Infrastructure team, the following practices must be observed for SSH access:

• SSH access is only permitted with SSH public/private key authentication.
• Users must use a strong passphrase to protect their private key. A passphrase of several words, at least one of which is not in a dictionary is advised, or a random string of at least 16 characters.
• To mitigate the inconvenience of using a long passphrase, users may want to use an SSH agent so that the passphrase is only requested once per desktop session.
• Users private keys must never be stored anywhere except their own workstation(s). In particular, they must never be stored on any remote server.
• If users need to ‘hop’ from a server or bastion host to another machine, they must not copy a private key to the intermediate machine (see above). Instead SSH agent forwarding may be used. However due to the potential for a compromised intermediate machine to ask the agent to sign requests without the users knowledge, in this case only an SSH agent that interactively prompts the user each time a signing request (ie, ssh-agent, but not gnome-keyring) is received should be used, and the SSH keys should be added with the confirmation constraint (‘ssh-add -c’).
• The number of SSH keys that are configured to permit access to OpenStack machines should be kept to a minimum.
• OpenStack Infrastructure machines must use puppet to centrally manage and configure user accounts, and the SSH authorized_keys files from the openstack-infra/config repository.
• SSH keys should be periodically rotated (at least once per year). During rotation, a new key can be added to puppet for a time, and then the old one removed. Be sure to run puppet on the backup servers to make sure they are updated.

Backups

Off-site backups are made to two servers:

• ci-backup-rs-ord.openstack.org
• ci-backup-hp-az1.openstack.org

Puppet is used to perform the initial configuration of those machines, but to protect them from unauthorized access in case access to the puppet git repo is compromised, it is not run in agent or in cron mode on them. Instead, it should be manually run when changes are made that should be applied to the backup servers.

To start backing up a server, some commands need to be run manually on both the backup server, and the server to be backed up. On the server to be backed up:

ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ""

And then ‘’cat /root/.ssh/id_rsa.pub’’ for use later.

On the backup servers:

sudo su -
BUPUSER=bup-<short-servername>  # eg, bup-jenkins-dev
useradd -r $BUPUSER -s /bin/bash -m
cd /home/$BUPUSER
mkdir .ssh
cat >.ssh/authorized_keys

and add this to the authorized_keys file:

command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty <ssh key from earlier>

Switching back to the server to be backed up, run:

ssh [email protected]
ssh [email protected]

And verify the host key. Add the “backup” class in puppet to the server to be backed up.

GitHub Access

To ensure that code review and testing are not bypassed in the public Git repositories, only Gerrit will be permitted to commit code to OpenStack repositories. Because GitHub always allows project administrators to commit code, accounts that have access to manage the GitHub projects necessarily will have commit access to the repositories. Therefore, to avoid inadvertent commits to the public repositories, unique administrative-only accounts must be used to manage the OpenStack GitHub organization and projects. These accounts will not be used to check out or commit code for any project.