yahoo-eng-team team mailing list archive

[Alvaro Uria <1824437@xxxxxxxxxxxxxxxxxx>]

** Changed in: charm-nova-compute
       Status: In Progress => Opinion

** Changed in: nova
       Status: In Progress => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1824437

Title:
  Issue with upgrade of nova-compute from Pike to Queens (RemoteError:
  Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList
  is not supported)

Status in OpenStack nova-compute charm:
  Opinion
Status in OpenStack Compute (nova):
  Opinion

Bug description:
  
  This issue is hapenning uring openstack upgrade on xenial. The procedure use was just changing the config "openstack-origin", and not using action-managed-upgrade, there were no
  instances spawned at the time of upgrade.

  Juju verion: 2.3.7-xenial-amd64

  When I upgrade from Pike to Queens the nova-compute charm gets blocked
  with error:

  
  nova-compute/0*       blocked      idle       6        10.10.245.9                        Services not running that should be: nova-compute

  When I saw this I decided to create a fresh Pike installation (instead
  of upgrading from Ocata). And then tried to upgrade it to queens. I
  git the same error. I do see many postings on internet where people
  had same issue but none of those posts were answered. I may need you
  help. I tried to post this on our launchpad channel, but I do not see
  any place to file a bug or post a question (probably will need your
  help with that too).

  
  After upgrade, I see below error and stack trace 

  
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [req-e37138fc-10f2-484d-9355-41d559d36e75 - - - - -] Error starting thread.: RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
  [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service Traceback (most recent call last):
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 729, in run_service
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     service.start()
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 161, in start
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     self.manager.init_host()
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1161, in init_host
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     evacuated_instances = self._destroy_evacuated_instances(context)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 658, in _destroy_evacuated_instances
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     filters)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     args, kwargs)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     args=args, kwargs=kwargs)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     retry=self.retry)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     timeout=timeout, retry=retry)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     retry=retry)
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service     raise result
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n    result = func(ctxt, **new_args)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
  2019-04-10 13:13:42.729 7883 ERROR oslo_service.service

  
  Here is the DB before and after upgrade

  Pre upgrade (at Pike)
  nova_api DB
  mysql> select * from migrate_version;
  +---------------+---------------------------------------------------------------------------------+---------+
  | repository_id | repository_path                                                                 | version |
  +---------------+---------------------------------------------------------------------------------+---------+
  | nova_api      | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo |      45 |
  +---------------+---------------------------------------------------------------------------------+---------+
  1 row in set (0.00 sec)

  nova DB
  mysql> select * from migrate_version;
  +---------------+------------------------------------------------------------------+---------+
  | repository_id | repository_path                                                  | version |
  +---------------+------------------------------------------------------------------+---------+
  | nova          | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo |     362 |
  +---------------+------------------------------------------------------------------+---------+
  1 row in set (0.00 sec)


  After upgrade
  nova_api DB
  mysql> select * from migrate_version;
  +---------------+---------------------------------------------------------------------------------+---------+
  | repository_id | repository_path                                                                 | version |
  +---------------+---------------------------------------------------------------------------------+---------+
  | nova_api      | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo |      52 |
  +---------------+---------------------------------------------------------------------------------+---------+
  1 row in set (0.00 sec)

  nova DB
  mysql> select * from migrate_version;
  +---------------+------------------------------------------------------------------+---------+
  | repository_id | repository_path                                                  | version |
  +---------------+------------------------------------------------------------------+---------+
  | nova          | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo |     378 |
  +---------------+------------------------------------------------------------------+---------+
  1 row in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-compute/+bug/1824437/+subscriptions