openstack team mailing list archive

[Monty Taylor <mordred@xxxxxxxxxxxx>]

On 07/12/2012 04:36 PM, Vishvananda Ishaya wrote:
> 
> On Jul 12, 2012, at 2:22 PM, Narayan Desai wrote:
> 
>> I think that the long term maintenance or removal of nova-volume in
>> its existing form is orthogonal to the actual issue of continuity from
>> one release to the next.
> 
> Agreed. Discussion the volume/cinder strategy is a separate topic. I've
> taken the liberty of updating the subject to keep the discussions on point
> 
>>
>> At this point, we've now run cactus, diablo and are in testing with
>> essex. Each of these has effectively been a flag day for us; we build
>> the new system, migrate users, images, etc, and let users do a bunch
>> of manual migration of volume data, etc, while running both systems in
>> parallel. This hasn't been as painful as it sounds because our
>> understanding of best practices for running openstack is moving pretty
>> quickly and each system has been quite different from the previous.
> 
> Upgrading has been painful and we are striving to improve this process
> as much as possible.
> 
>>
>> The lack of an effective process to move from one major release to the
>> next is the major issue here in my mind. It would be fantastic if
>> (some day, ha ha ha) you could apt-get upgrade from folsom to grizzly,
>> but i think that is likely to be more trouble than it is worth. A
>> reasonable compromise would be a well documented process as well as
>> tools to aid in the process. Each real deployment will have a
>> substantial set of local customizations, particularly if they are
>> running at any sort of scale. While it won't be feasible to support
>> any upgrade with these customizations, tools for the process (which
>> may only be used a straw man in complex cases) would go a long way.
> 
> I would like to take this a bit further. Documentation is a great first step,
> but I would actually like to have an actual Jenkins test that does the upgrade
> from essex to Folsom with live resources created. I think this the only way
> that we can be sure that the upgrade is working properly.

++

> The first version of this doesn't even have to be on a full cluster. I'm thinking
> something as simple as:
> 
> * configure devstack to checkout stable/essex from all of the projects
> * run the system, launch some instances and volumes
> * terminate the workers
> * upgrade all of the code to folsom
> * do any manual upgrade steps (nova-manage db sync, cinder migrate, etc.)
> * run all the workers
> * make sure the existing data still works and new api commands run
> 
> The manual upgrade steps should be contained in a single script so that the
> distress can use it to help make their package upgrades and deployers can
> use it for reference when upgrading their clusters.

Yes - especially if it's a self contained thing like devstack is currently.

For the "upgrade all the code to folsom" step - let's chat about making
sure that we get the right hooks/env vars in there so that we can make
that "upgrade to tip of trunk in most projects + proposed patch in one
of them"  - same as we do for devstack runs today.

> This is something we can start working on today and we can run after each
> commit. Then we will immediately know if we do something that breaks
> upgradability, and we will have a testable documented process for upgrading.

The creation of the self-contained script devstack was a HUGE step
forward for us for integration testing. I think a similar thing for
upgradability would similarly be huge.