yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #60009
[Bug 1652233] [NEW] mitaka is incompatible with newton
Public bug reported:
Description
===========
I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
Steps to reproduce
==================
1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
7) Try to place any VMs, it will fail
8) See nova-compute.log on the mitaka nodes
Expected result
===============
Successful upgrade one half of cluster, then another half
Actual result
=============
Nova can't place any VMs.
Compute logs:
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
nova-conductor:
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
Environment
===========
CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch
contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch
netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch
controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch
** Affects: nova
Importance: Undecided
Status: New
** Description changed:
Description
===========
I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
Steps to reproduce
==================
1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
- 3) Upgrade to newton on disable nodes
+ 3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
- 7) Try to place any VMs, it fill fail
+ 7) Try to place any VMs, it will fail
8) see nova-compute.log on the mitaka nodes
Expected result
===============
Successful upgrade one half of cluster, then another half
Actual result
=============
Nova can't place any VMs.
Compute logs:
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
-
nova-conductor:
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
-
Environment
===========
CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch
contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch
netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch
controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch
** Description changed:
Description
===========
I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
Steps to reproduce
==================
1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
7) Try to place any VMs, it will fail
- 8) see nova-compute.log on the mitaka nodes
+ 8) See nova-compute.log on the mitaka nodes
Expected result
===============
Successful upgrade one half of cluster, then another half
Actual result
=============
Nova can't place any VMs.
Compute logs:
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
nova-conductor:
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
Environment
===========
CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch
contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch
netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch
controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1652233
Title:
mitaka is incompatible with newton
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
Steps to reproduce
==================
1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
7) Try to place any VMs, it will fail
8) See nova-compute.log on the mitaka nodes
Expected result
===============
Successful upgrade one half of cluster, then another half
Actual result
=============
Nova can't place any VMs.
Compute logs:
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', u' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
nova-conductor:
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n :param incoming: incoming message\n', ' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
Environment
===========
CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch
contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch
netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch
controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1652233/+subscriptions
Follow ups
