yahoo-eng-team team mailing list archive

[Guang Yee <1696417@xxxxxxxxxxxxxxxxxx>]

I also ran into this exact same issue. We have mysql as the backend. And
the datetime type is not timezone aware. Neither "--verbose" and "--
debug" are effective for the online_data_migrations command as they are
not being taken into consideration. I ended up manually printing out the
traceback by adding these two lines

import sys, traceback
traceback.print_exc(file=sys.stdout)

here
https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L897

And able to see more information on the failure. Here's a sample
traceback.


Running batches of 50 until complete
-------------- 2017-05-19 23:34:43+00:00 -----------------
Error attempting to run <function migrate_flavors at 0x7f5a0d758e60>
Traceback (most recent call last):
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/cmd/manage.py", line 892, in _run_migration
    found, done = migration_meth(ctxt, count)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 717, in migrate_flavors
    flavor._flavor_create(ctxt, flavor_values)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 463, in _flavor_create
    return _flavor_create(context, updates)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 824, in wrapper
    return fn(*args, **kwargs)
  File "/opt/stack/venv/nova-20170728T171245Z/lib/python2.7/site-packages/nova/objects/flavor.py", line 166, in _flavor_create
    raise db_exc.DBError(e)
DBError: (_mysql_exceptions.OperationalError) (1292, "Incorrect datetime value: '2017-05-19 23:34:43+00:00' for column 'created_at' at row 1") [SQL: u'INSERT INTO flavors (created_at, updated_at, id, name, memory_mb, vcpus, root_gb, ephemeral_gb, flavorid, swap, rxtx_factor, vcpu_weight, disabled, is_public) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2017, 5, 19, 23, 34, 43, tzinfo=<iso8601.Utc>), None, 16, 'foo', 1024, 1, 2, 0, '50aae60f-ba2a-40d8-a0c3-2117ba0dd2a6', 0, 1.0, 0, 0, 0)]

Looks like migrations code is attempting to insert a timezone aware
datetime field into mysql.


** Changed in: nova
       Status: Expired => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1696417

Title:
  nova-manage db online_data_migrations can fail when upgrading to
  newton under certain conditions

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Reproduceable under conditions:

  * Mitaka upgraded to Newton
  * online_data_migrations have not yet migrated flavors and/or aggregates from the nova database, to the nova-api database
  * One or more of the datetime-fields (created_at, updated_at, deleted_at) are set.
  ** Because a custom flavor has been created
  ** Because a flavor has been updated
  ** Because a flavor has been deleted (deleted flavors are probably not relevant, as the new table have no deleted flag, it just removes them altogether)

  Steps to reproduce:

  * Run 'nova-manage db online_data_migrations'

  It throws an error message like:
  Error attempting to run <function migrate_flavors at 0x488f398>

  Workaround:

  * Set created_at,updated_at and deleted_at to NULL
  * Run migration


  I have done quite a bit of troubleshooting, but haven't managed to
  write a patch so far.  As far as I can tell, inserting a flavor or
  aggregate to the new tables fail due to the datetime fields including
  a timezone.  There exists code for stripping away the timezone in
  nova/db/sqlalchemy/api.py (convert_objects_related_datetimes) - but
  the timezone reappears in nova/objects/flavor.py

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1696417/+subscriptions