Probably the most important step of all is the pre-upgrade testing. Especially if you are upgrading immediately after release of a new version, undiscovered bugs might hinder your progress. Some deployers prefer to wait until the first point release is announced. However, if you have a significant deployment, you might follow the development and testing of the release, thereby ensuring that bugs for your use cases are fixed.
Each OpenStack cloud is different, and as a result, even with what may seem a near-identical architecture to this guide, you must still test upgrades between versions in your environment. For this, you need an approximate clone of your environment.
However, that is not to say that it needs to be the same size or use identical hardware as the production environment—few of us have that luxury. It is important to consider the hardware and scale of the cloud you are upgrading, but here are some tips to avoid that incredible cost:
Use your own cloud. The simplest place to start testing the next version of OpenStack is by setting up a new environment inside your own cloud. This may seem odd—especially the double virtualisation used in running compute nodes—but it's a sure way to very quickly test your configuration.
Use a public cloud. Especially because your own cloud is unlikely to have sufficient space to scale test to the level of the entire cloud, consider using a public cloud to test the scalability limits of your cloud controller configuration. Most public clouds bill by the hour, which means it can be inexpensive to perform even a test with many nodes.
Make another storage endpoint on the same system. If you use an external storage plug-in or shared file system with your cloud, in many cases it's possible to test that it works by creating a second share or endpoint. This will enable you to test the system before entrusting the new version onto your storage.
Watch the network. Even at smaller-scale testing, it should be possible to determine whether something is going horribly wrong in intercomponent communication if you look at the network packets and see too many.
To actually set up the test environment, there are several methods. Some prefer to do a full manual install using the OpenStack Installation Guides, and then see what the final configuration files look like and which packages were installed. Others prefer to create a clone of their automated configuration infrastructure with changed package repository URLs and then alter the configuration until it starts working. Either approach is valid, and which you use depends on experience.
An upgrade pre-testing system is excellent for getting the configuration to work, however, it is important to note that the historical use of the system and differences in user interaction can affect the success of upgrades, too. We've seen experiences where database migrations encountered a bug (later fixed!) because of slight table differences between fresh Grizzly installs and those that migrated from Folsom to Grizzly.
Artificial scale testing can go only so far. Once your cloud is upgraded, you'll also need to pay careful attention to the performance aspects of your cloud.
