openstack team mailing list archive

[Vishvananda Ishaya <vishvananda@xxxxxxxxx>]

On Jul 12, 2012, at 2:22 PM, Narayan Desai wrote:

> I think that the long term maintenance or removal of nova-volume in
> its existing form is orthogonal to the actual issue of continuity from
> one release to the next.

Agreed. Discussion the volume/cinder strategy is a separate topic. I've
taken the liberty of updating the subject to keep the discussions on point

> 
> At this point, we've now run cactus, diablo and are in testing with
> essex. Each of these has effectively been a flag day for us; we build
> the new system, migrate users, images, etc, and let users do a bunch
> of manual migration of volume data, etc, while running both systems in
> parallel. This hasn't been as painful as it sounds because our
> understanding of best practices for running openstack is moving pretty
> quickly and each system has been quite different from the previous.

Upgrading has been painful and we are striving to improve this process
as much as possible.

> 
> The lack of an effective process to move from one major release to the
> next is the major issue here in my mind. It would be fantastic if
> (some day, ha ha ha) you could apt-get upgrade from folsom to grizzly,
> but i think that is likely to be more trouble than it is worth. A
> reasonable compromise would be a well documented process as well as
> tools to aid in the process. Each real deployment will have a
> substantial set of local customizations, particularly if they are
> running at any sort of scale. While it won't be feasible to support
> any upgrade with these customizations, tools for the process (which
> may only be used a straw man in complex cases) would go a long way.

I would like to take this a bit further. Documentation is a great first step,
but I would actually like to have an actual Jenkins test that does the upgrade
from essex to Folsom with live resources created. I think this the only way
that we can be sure that the upgrade is working properly.

The first version of this doesn't even have to be on a full cluster. I'm thinking
something as simple as:

* configure devstack to checkout stable/essex from all of the projects
* run the system, launch some instances and volumes
* terminate the workers
* upgrade all of the code to folsom
* do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
* run all the workers
* make sure the existing data still works and new api commands run

The manual upgrade steps should be contained in a single script so that the
distress can use it to help make their package upgrades and deployers can
use it for reference when upgrading their clusters.

This is something we can start working on today and we can run after each
commit. Then we will immediately know if we do something that breaks
upgradability, and we will have a testable documented process for upgrading.

Vish