Rolling Back a Failed Upgrade

While we do not wish this fate upon anyone, upgrades involve complex operations and can fail. This section provides guidance for rolling back to a previous release of OpenStack. Although only tested on Ubuntu, other distributions follow a similar procedure.

In this section, we consider only the most immediate case: you have taken down production management services in preparation for an upgrade, completed part of the upgrade process, discovered one or more problems not encountered during testing, and need to roll back your environment to the original "known good" state. We specifically assume that you did not make any state changes after attempting the upgrade process: no new instances, networks, storage volumes, etc.

Within this scope, you need to accomplish three main steps to successfully roll back your environment:

Roll back configuration files
Roll back databases
Roll back packages

The upgrade instructions provided in earlier sections ensure that you have proper backups of your databases and configuration files. You should read through this section carefully and verify that you have the requisite backups to restore. Rolling back upgrades is a tricky process because distributions tend to put much more effort into testing upgrades than downgrades. Broken downgrades often take significantly more effort to troubleshoot and, hopefully, resolve than broken upgrades. Only you can weigh the risks of trying to push a failed upgrade forward versus rolling it back. Generally, we consider rolling back the very last option.

The steps described below for Ubuntu have worked on at least one production environment, but they may not work for all environments.

Perform the Roll Back from Havana to Grizzly

Stop all OpenStack services.
Copy contents of configuration backup directories /etc/<service>.grizzly that you created during the upgrade process back to /etc/<service>:
Restore databases from the backup file grizzly-db-backup.sql that you created with mysqldump during the upgrade process.
```
# mysql -u root -p < grizzly-db-backup.sql
```
If you created this backup using the --add-drop-database flag as instructed, you can proceed to the next step. If you omitted this flag, MySQL will revert all of the tables that existed in Grizzly, but not drop any tables created during the database migration for Havana. In this case, you need to manually determine which tables should not exist and drop them to prevent issues with your next upgrade attempt.

Downgrade OpenStack packages.

	Warning
	We consider downgrading packages by far the most complicated step and highly dependent on the distribution as well as overall administration of the system.

Determine the OpenStack packages installed on your system. This is done using dpkg --get-selections, filtering for OpenStack packages, filtering again to omit packages explicitly marked in the deinstall state, and saving the final output to a file. For example, the following command covers a controller node with keystone, glance, nova, neutron, and cinder:

# dpkg --get-selections | grep -e keystone -e glance -e nova -e neutron -e cinder \
  | grep -v deinstall | tee openstack-selections
cinder-api                                      install
cinder-common                                   install
cinder-scheduler                                install
cinder-volume                                   install
glance                                          install
glance-api                                      install
glance-common                                   install
glance-registry                                 install
neutron-common                                  install
neutron-dhcp-agent                              install
neutron-l3-agent                                install
neutron-lbaas-agent                             install
neutron-metadata-agent                          install
neutron-plugin-openvswitch                      install
neutron-plugin-openvswitch-agent                install
neutron-server                                  install
nova-api                                        install
nova-cert                                       install
nova-common                                     install
nova-conductor                                  install
nova-consoleauth                                install
nova-novncproxy                                 install
nova-objectstore                                install
nova-scheduler                                  install
python-cinder                                   install
python-cinderclient                             install
python-glance                                   install
python-glanceclient                             install
python-keystone                                 install
python-keystoneclient                           install
python-neutron                                  install
python-neutronclient                            install
python-nova                                     install
python-novaclient                               install

	Note
	Depending on the type of server, the contents and order of your package list may vary from this example.

You can determine the package versions available for reversion by using apt-cache policy. If you removed the Grizzly repositories, you must first reinstall them and run apt-get update.

# apt-cache policy nova-common
nova-common:
  Installed: 1:2013.2-0ubuntu1~cloud0
  Candidate: 1:2013.2-0ubuntu1~cloud0
  Version table:
 *** 1:2013.2-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/havana/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2013.1.4-0ubuntu1~cloud0 0
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/grizzly/main amd64 Packages
     2012.1.3+stable-20130423-e52e6912-0ubuntu1.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
     2012.1-0ubuntu2 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

This tells us the currently installed version of the package, newest candidate version, and all versions along with the repository that contains each version. Look for the appropriate Grizzly version, in this case 1:2013.1.4-0ubuntu1~cloud0. The process of manually picking through this list of packages is rather tedious and prone to errors. You should consider using the following script to help with this process:

# for i in `cut -f 1 openstack-selections | sed 's/neutron/quantum/;'`; \
  do echo -n $i ;apt-cache policy $i | grep -B 1 grizzly | grep -v Packages \
  | awk '{print "="$1}';done | tr '\n' ' ' | tee openstack-grizzly-versions
cinder-api=1:2013.1.4-0ubuntu1~cloud0 cinder-common=1:2013.1.4-0ubuntu1~cloud0 cinder-scheduler=1:2013.1.4-0ubuntu1~cloud0 cinder-volume=1:2013.1.4-0ubuntu1~cloud0 glance=1:2013.1.4-0ubuntu1~cloud0 glance-api=1:2013.1.4-0ubuntu1~cloud0 glance-common=1:2013.1.4-0ubuntu1~cloud0 glance-registry=1:2013.1.4-0ubuntu1~cloud0 quantum-common=1:2013.1.4-0ubuntu1~cloud0 quantum-dhcp-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-l3-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-lbaas-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-metadata-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-server=1:2013.1.4-0ubuntu1~cloud0 nova-api=1:2013.1.4-0ubuntu1~cloud0 nova-cert=1:2013.1.4-0ubuntu1~cloud0 nova-common=1:2013.1.4-0ubuntu1~cloud0 nova-conductor=1:2013.1.4-0ubuntu1~cloud0 nova-consoleauth=1:2013.1.4-0ubuntu1~cloud0 nova-novncproxy=1:2013.1.4-0ubuntu1~cloud0 nova-objectstore=1:2013.1.4-0ubuntu1~cloud0 nova-scheduler=1:2013.1.4-0ubuntu1~cloud0 python-cinder=1:2013.1.4-0ubuntu1~cloud0 python-cinderclient=1:1.0.3-0ubuntu1~cloud0 python-glance=1:2013.1.4-0ubuntu1~cloud0 python-glanceclient=1:0.9.0-0ubuntu1.2~cloud0 python-quantum=1:2013.1.4-0ubuntu1~cloud0 python-quantumclient=1:2.2.0-0ubuntu1~cloud0 python-nova=1:2013.1.4-0ubuntu1~cloud0 python-novaclient=1:2.13.0-0ubuntu1~cloud0

	Note
	If you decide to continue this step manually, don't forget to change `neutron` to `quantum` where applicable.

Use apt-get install to install specific versions of each package by specifying <package-name>=<version>. The script in the previous step conveniently created a list of package=version pairs for you.
```
# apt-get install `cat openstack-grizzly-versions`
```
This completes the roll back procedure. You should remove the Havana repository and run apt-get update to prevent accidental upgrades until you solve whatever issue caused you to roll back your environment.

Questions? Discuss on ask.openstack.org
Found an error? Report a bug against this page

Legal notices