openstack team mailing list archive

[Joshua Harlow <harlowja@xxxxxxxxxxxxx>]

So a useful question,

What mistakes can we learn from this to make sure that this doesn't happen again and that if it does we catch it much earlier?

On 4/13/12 4:17 PM, "Adam Gandelman" <adamg@xxxxxxxxxxxxx> wrote:

Hi-

We've added a patch to the Glance package in Ubuntu to help improve the
experience for users who are counting on a smooth upgrade from Diablo.
Since the release of Essex, I've been mostly focused on upgrade
testing.  Things seem generally okay with the exception of Glance and
the way it manages its database and migrations.  To ensure users can
reliably upgrade from Diablo, we've made a couple of changes that people
should know about as the end result is currently a bit different than
what you'd expect comparing to devstack/vanilla git repos.

1. Disabled database auto-creation

Glance is the only project that I know of that currently allows
sqlalchemy to auto-create database schema to match its database
models.   This is great for the developer use-case, but really not ideal
for anyone who wishes to maintain their database for the long-term.  The
original issue is that, on startup, glance-registry creates a database
schema that reflects its most recent view of the database.  When it
does, the schema is created outside the view of migrate and the
migration repository cannot be used to incrementally upgrade/downgrade
the schema.

Recent changes [1] in Glance provides a work-around to this isssue,
where we rely on glance-registry to automatically put the database under
version control after auto-creation and set its version to the most
recent (currently 13).  This works fine for development but there are
important details in the migration code that don't actually end up in
the database schema when auto-created (FK constraints, for example)
because the migration scripts are not actually run.

To avoid this entirely, we've prevented Glance from auto-migrating its
database and now require users to migrate the database on new install
and upgrade (via db_sync) in the same way Nova does.

2. Sync'ing a new database.

The changes also now allow users to set a specific version when
initializing version_control on a database with glance-manage, eg:

glance-manage version_control 9
glance-manage db_sync

This would properly migrate an non-version controlled Diablo database to
Essex. This is great, but it is still possible for Diablo users to
mistakenly start glance or run db_sync with no version control, and end
up in a situation where glance + migrate think it is up to date. In
reality the database is missing 3 migrations (one of which, 012 IDs to
UUIDs, is very essential!)   To prevent this, the Ubuntu patch
additionally requires explicitly version_control'ing the database a
before db_sync on both new databases and during upgrades of databases
that are not already version_control'd.

For example, on a new database:

# Start from the beginning of the migration repositroy
glance-manage version_control 0
glance-manage db_sync

or to upgrade from Diablo:

# Was our database properly version controlled to begin with?
if ! glance-manage db_version ; then
   # Set its version if not
   glance-manage version_control 9
fi
# Run all migrations 10 to 13
glance-manage db_sync

I understand this may have come a bit late to any users of the Precise
packages and I apologize for the lack of a heads up.  I've filed a new
bug [2] about the split brain between glance-registry and and hope to
address this at the database management session [3] at ODS next week.
Anne Gentle has also proposed an update to the documentation with a note
about this.   Please understand that our aim was to keep our patch delta
as minimal as possible in Ubuntu, but with the yesterday's Final Freeze
for the cycle 12.04 we needed something in place that ensures a sane
user story for previous users of Diablo and future Essex users alike.

Cheers,
Adam

[1] https://review.openstack.org/#/c/6166/
https://review.openstack.org/#/c/5980/1
[2] https://bugs.launchpad.net/bugs/981081
[3] http://summit.openstack.org/sessions/view/36



_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to     : openstack@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp