Rolling Back a Failed Upgrade

While we do not wish this fate upon anyone, upgrades involve complex operations and can fail. This section provides guidance for rolling back to a previous release of OpenStack. Although only tested on Ubuntu, other distributions follow a similar procedure.

In this section, we consider only the most immediate case: you have taken down production management services in preparation for an upgrade, completed part of the upgrade process, discovered one or more problems not encountered during testing, and need to roll back your environment to the original "known good" state. We specifically assume that you did not make any state changes after attempting the upgrade process: no new instances, networks, storage volumes, etc.

Within this scope, you need to accomplish three main steps to successfully roll back your environment:

  • Roll back configuration files

  • Roll back databases

  • Roll back packages

The upgrade instructions provided in earlier sections ensure that you have proper backups of your databases and configuration files. You should read through this section carefully and verify that you have the requisite backups to restore. Rolling back upgrades is a tricky process because distributions tend to put much more effort into testing upgrades than downgrades. Broken downgrades often take significantly more effort to troubleshoot and, hopefully, resolve than broken upgrades. Only you can weigh the risks of trying to push a failed upgrade forward versus rolling it back. Generally, we consider rolling back the very last option.

The steps described below for Ubuntu have worked on at least one production environment, but they may not work for all environments.

 

Perform the Roll Back from Havana to Grizzly

  1. Stop all OpenStack services.

  2. Copy contents of configuration backup directories /etc/<service>.grizzly that you created during the upgrade process back to /etc/<service>:

  3. Restore databases from the backup file grizzly-db-backup.sql that you created with mysqldump during the upgrade process.

    # mysql -u root -p < grizzly-db-backup.sql

    If you created this backup using the --add-drop-database flag as instructed, you can proceed to the next step. If you omitted this flag, MySQL will revert all of the tables that existed in Grizzly, but not drop any tables created during the database migration for Havana. In this case, you need to manually determine which tables should not exist and drop them to prevent issues with your next upgrade attempt.

  4. Downgrade OpenStack packages.

    [Warning]Warning

    We consider downgrading packages by far the most complicated step and highly dependent on the distribution as well as overall administration of the system.

    1. Determine the OpenStack packages installed on your system. This is done using dpkg --get-selections, filtering for OpenStack packages, filtering again to omit packages explicitly marked in the deinstall state, and saving the final output to a file. For example, the following command covers a controller node with keystone, glance, nova, neutron, and cinder:

      # dpkg --get-selections | grep -e keystone -e glance -e nova -e neutron -e cinder \
        | grep -v deinstall | tee openstack-selections
      cinder-api                                      install
      cinder-common                                   install
      cinder-scheduler                                install
      cinder-volume                                   install
      glance                                          install
      glance-api                                      install
      glance-common                                   install
      glance-registry                                 install
      neutron-common                                  install
      neutron-dhcp-agent                              install
      neutron-l3-agent                                install
      neutron-lbaas-agent                             install
      neutron-metadata-agent                          install
      neutron-plugin-openvswitch                      install
      neutron-plugin-openvswitch-agent                install
      neutron-server                                  install
      nova-api                                        install
      nova-cert                                       install
      nova-common                                     install
      nova-conductor                                  install
      nova-consoleauth                                install
      nova-novncproxy                                 install
      nova-objectstore                                install
      nova-scheduler                                  install
      python-cinder                                   install
      python-cinderclient                             install
      python-glance                                   install
      python-glanceclient                             install
      python-keystone                                 install
      python-keystoneclient                           install
      python-neutron                                  install
      python-neutronclient                            install
      python-nova                                     install
      python-novaclient                               install
      
      [Note]Note

      Depending on the type of server, the contents and order of your package list may vary from this example.

    2. You can determine the package versions available for reversion by using apt-cache policy. If you removed the Grizzly repositories, you must first reinstall them and run apt-get update.

      # apt-cache policy nova-common
      nova-common:
        Installed: 1:2013.2-0ubuntu1~cloud0
        Candidate: 1:2013.2-0ubuntu1~cloud0
        Version table:
       *** 1:2013.2-0ubuntu1~cloud0 0
              500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/havana/main amd64 Packages
              100 /var/lib/dpkg/status
           1:2013.1.4-0ubuntu1~cloud0 0
              500 http://ubuntu-cloud.archive.canonical.com/ubuntu/ precise-updates/grizzly/main amd64 Packages
           2012.1.3+stable-20130423-e52e6912-0ubuntu1.2 0
              500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
              500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
           2012.1-0ubuntu2 0
              500 http://us.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
      

      This tells us the currently installed version of the package, newest candidate version, and all versions along with the repository that contains each version. Look for the appropriate Grizzly version, in this case 1:2013.1.4-0ubuntu1~cloud0. The process of manually picking through this list of packages is rather tedious and prone to errors. You should consider using the following script to help with this process:

      # for i in `cut -f 1 openstack-selections | sed 's/neutron/quantum/;'`; \
        do echo -n $i ;apt-cache policy $i | grep -B 1 grizzly | grep -v Packages \
        | awk '{print "="$1}';done | tr '\n' ' ' | tee openstack-grizzly-versions
      cinder-api=1:2013.1.4-0ubuntu1~cloud0 cinder-common=1:2013.1.4-0ubuntu1~cloud0 cinder-scheduler=1:2013.1.4-0ubuntu1~cloud0 cinder-volume=1:2013.1.4-0ubuntu1~cloud0 glance=1:2013.1.4-0ubuntu1~cloud0 glance-api=1:2013.1.4-0ubuntu1~cloud0 glance-common=1:2013.1.4-0ubuntu1~cloud0 glance-registry=1:2013.1.4-0ubuntu1~cloud0 quantum-common=1:2013.1.4-0ubuntu1~cloud0 quantum-dhcp-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-l3-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-lbaas-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-metadata-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch=1:2013.1.4-0ubuntu1~cloud0 quantum-plugin-openvswitch-agent=1:2013.1.4-0ubuntu1~cloud0 quantum-server=1:2013.1.4-0ubuntu1~cloud0 nova-api=1:2013.1.4-0ubuntu1~cloud0 nova-cert=1:2013.1.4-0ubuntu1~cloud0 nova-common=1:2013.1.4-0ubuntu1~cloud0 nova-conductor=1:2013.1.4-0ubuntu1~cloud0 nova-consoleauth=1:2013.1.4-0ubuntu1~cloud0 nova-novncproxy=1:2013.1.4-0ubuntu1~cloud0 nova-objectstore=1:2013.1.4-0ubuntu1~cloud0 nova-scheduler=1:2013.1.4-0ubuntu1~cloud0 python-cinder=1:2013.1.4-0ubuntu1~cloud0 python-cinderclient=1:1.0.3-0ubuntu1~cloud0 python-glance=1:2013.1.4-0ubuntu1~cloud0 python-glanceclient=1:0.9.0-0ubuntu1.2~cloud0 python-quantum=1:2013.1.4-0ubuntu1~cloud0 python-quantumclient=1:2.2.0-0ubuntu1~cloud0 python-nova=1:2013.1.4-0ubuntu1~cloud0 python-novaclient=1:2.13.0-0ubuntu1~cloud0
      
      [Note]Note

      If you decide to continue this step manually, don't forget to change neutron to quantum where applicable.

    3. Use apt-get install to install specific versions of each package by specifying <package-name>=<version>. The script in the previous step conveniently created a list of package=version pairs for you.

      # apt-get install `cat openstack-grizzly-versions`

      This completes the roll back procedure. You should remove the Havana repository and run apt-get update to prevent accidental upgrades until you solve whatever issue caused you to roll back your environment.

Questions? Discuss on ask.openstack.org
Found an error? Report a bug against this page

loading table of contents...