yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #77960
[Bug 1824437] [NEW] Issue with upgrade of nova-compute from Pike to Queens (RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported)
Public bug reported:
This issue is hapenning uring openstack upgrade on xenial. The procedure use was just changing the config "openstack-origin", and not using action-managed-upgrade, there were no
instances spawned at the time of upgrade.
Juju verion: 2.3.7-xenial-amd64
When I upgrade from Pike to Queens the nova-compute charm gets blocked
with error:
nova-compute/0* blocked idle 6 10.10.245.9 Services not running that should be: nova-compute
When I saw this I decided to create a fresh Pike installation (instead
of upgrading from Ocata). And then tried to upgrade it to queens. I git
the same error. I do see many postings on internet where people had same
issue but none of those posts were answered. I may need you help. I
tried to post this on our launchpad channel, but I do not see any place
to file a bug or post a question (probably will need your help with that
too).
After upgrade, I see below error and stack trace
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [req-e37138fc-10f2-484d-9355-41d559d36e75 - - - - -] Error starting thread.: RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service Traceback (most recent call last):
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 729, in run_service
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service service.start()
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 161, in start
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service self.manager.init_host()
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1161, in init_host
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service evacuated_instances = self._destroy_evacuated_instances(context)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 658, in _destroy_evacuated_instances
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service filters)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service args, kwargs)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service args=args, kwargs=kwargs)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service retry=self.retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service timeout=timeout, retry=retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service retry=retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service raise result
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service
Here is the DB before and after upgrade
Pre upgrade (at Pike)
nova_api DB
mysql> select * from migrate_version;
+---------------+---------------------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+---------------------------------------------------------------------------------+---------+
| nova_api | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 45 |
+---------------+---------------------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
nova DB
mysql> select * from migrate_version;
+---------------+------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+------------------------------------------------------------------+---------+
| nova | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo | 362 |
+---------------+------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
After upgrade
nova_api DB
mysql> select * from migrate_version;
+---------------+---------------------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+---------------------------------------------------------------------------------+---------+
| nova_api | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 52 |
+---------------+---------------------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
nova DB
mysql> select * from migrate_version;
+---------------+------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+------------------------------------------------------------------+---------+
| nova | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo | 378 |
+---------------+------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1824437
Title:
Issue with upgrade of nova-compute from Pike to Queens (RemoteError:
Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList
is not supported)
Status in OpenStack Compute (nova):
New
Bug description:
This issue is hapenning uring openstack upgrade on xenial. The procedure use was just changing the config "openstack-origin", and not using action-managed-upgrade, there were no
instances spawned at the time of upgrade.
Juju verion: 2.3.7-xenial-amd64
When I upgrade from Pike to Queens the nova-compute charm gets blocked
with error:
nova-compute/0* blocked idle 6 10.10.245.9 Services not running that should be: nova-compute
When I saw this I decided to create a fresh Pike installation (instead
of upgrading from Ocata). And then tried to upgrade it to queens. I
git the same error. I do see many postings on internet where people
had same issue but none of those posts were answered. I may need you
help. I tried to post this on our launchpad channel, but I do not see
any place to file a bug or post a question (probably will need your
help with that too).
After upgrade, I see below error and stack trace
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [req-e37138fc-10f2-484d-9355-41d559d36e75 - - - - -] Error starting thread.: RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
[u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service Traceback (most recent call last):
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 729, in run_service
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service service.start()
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 161, in start
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service self.manager.init_host()
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1161, in init_host
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service evacuated_instances = self._destroy_evacuated_instances(context)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 658, in _destroy_evacuated_instances
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service filters)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service args, kwargs)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service args=args, kwargs=kwargs)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service retry=self.retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service timeout=timeout, retry=retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service retry=retry)
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 550, in _send
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service raise result
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service RemoteError: Remote error: IncompatibleObjectVersion Version 1.4 of MigrationList is not supported
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 160, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 131, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 1.4 of MigrationList is not supported\n'].
2019-04-10 13:13:42.729 7883 ERROR oslo_service.service
Here is the DB before and after upgrade
Pre upgrade (at Pike)
nova_api DB
mysql> select * from migrate_version;
+---------------+---------------------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+---------------------------------------------------------------------------------+---------+
| nova_api | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 45 |
+---------------+---------------------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
nova DB
mysql> select * from migrate_version;
+---------------+------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+------------------------------------------------------------------+---------+
| nova | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo | 362 |
+---------------+------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
After upgrade
nova_api DB
mysql> select * from migrate_version;
+---------------+---------------------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+---------------------------------------------------------------------------------+---------+
| nova_api | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 52 |
+---------------+---------------------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
nova DB
mysql> select * from migrate_version;
+---------------+------------------------------------------------------------------+---------+
| repository_id | repository_path | version |
+---------------+------------------------------------------------------------------+---------+
| nova | /usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo | 378 |
+---------------+------------------------------------------------------------------+---------+
1 row in set (0.00 sec)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1824437/+subscriptions
Follow ups