← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1652233] [NEW] mitaka is incompatible with newton

 

Public bug reported:

Description
===========

I get an error after upgrade half of my cluster. Can't place any VMs.
"RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"

Steps to reproduce
==================

1) Install 4 nodes with mitaka
2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
3) Upgrade to newton on the disable nodes
4) compute=mitaka to [upgrade_levels]
5) db sync
6) Start newton
7) Try to place any VMs, it will fail
8) See nova-compute.log on the mitaka nodes

Expected result
===============

Successful upgrade one half of cluster, then another half

Actual result
=============

Nova can't place any VMs.

Compute logs:

2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     task(self, context)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     use_slave=True)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args, kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args=args, kwargs=kwargs)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=self.retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=retry)
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     raise result
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task

nova-conductor:

2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     :param incoming: incoming message
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     objname, object_versions[objname])
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     supported=latest_ver)
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', '  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']

Environment
===========

CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

mitaka nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-compute-13.1.2-1.el7.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-common-13.1.2-1.el7.noarch

contoller:
openstack-nova-scheduler-13.1.2-1.el7.noarch
openstack-nova-conductor-13.1.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-13.1.2-1.el7.noarch
openstack-nova-api-13.1.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-common-13.1.2-1.el7.noarch
openstack-nova-console-13.1.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
openstack-nova-novncproxy-13.1.2-1.el7.noarch

netonw nodes:
compute:
python2-novaclient-3.3.2-1.el7.noarch
python-nova-14.0.2-1.el7.noarch
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-compute-14.0.2-1.el7.noarch

controller:
openstack-nova-common-14.0.2-1.el7.noarch
openstack-nova-console-14.0.2-1.el7.noarch
novameter-2016.2-3.noarch
openstack-nova-conductor-14.0.2-1.el7.noarch
openstack-nova-novncproxy-14.0.2-1.el7.noarch
openstack-nova-api-14.0.2-1.el7.noarch
openstack-nova-scheduler-14.0.2-1.el7.noarch
python2-novaclient-3.3.2-1.el7.noarch
nova-solver-compute-lib-2016.2-2.noarch
python-nova-14.0.2-1.el7.noarch

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Description
  ===========
  
  I get an error after upgrade half of my cluster. Can't place any VMs.
  "RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
  
  Steps to reproduce
  ==================
  
  1) Install 4 nodes with mitaka
  2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
- 3) Upgrade to newton on disable nodes
+ 3) Upgrade to newton on the disable nodes
  4) compute=mitaka to [upgrade_levels]
  5) db sync
  6) Start newton
- 7) Try to place any VMs, it fill fail
+ 7) Try to place any VMs, it will fail
  8) see nova-compute.log on the mitaka nodes
  
  Expected result
  ===============
  
  Successful upgrade one half of cluster, then another half
  
  Actual result
  =============
  
  Nova can't place any VMs.
  
  Compute logs:
  
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     task(self, context)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     use_slave=True)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args, kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args=args, kwargs=kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=self.retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     raise result
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
  
- 
  nova-conductor:
  
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     :param incoming: incoming message
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     objname, object_versions[objname])
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     supported=latest_ver)
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
  2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', '  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
- 
  
  Environment
  ===========
  
  CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
  23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  
  mitaka nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-compute-13.1.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  
  contoller:
  openstack-nova-scheduler-13.1.2-1.el7.noarch
  openstack-nova-conductor-13.1.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-api-13.1.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  openstack-nova-console-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-novncproxy-13.1.2-1.el7.noarch
  
  netonw nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-14.0.2-1.el7.noarch
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-compute-14.0.2-1.el7.noarch
  
  controller:
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-console-14.0.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-conductor-14.0.2-1.el7.noarch
  openstack-nova-novncproxy-14.0.2-1.el7.noarch
  openstack-nova-api-14.0.2-1.el7.noarch
  openstack-nova-scheduler-14.0.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-14.0.2-1.el7.noarch

** Description changed:

  Description
  ===========
  
  I get an error after upgrade half of my cluster. Can't place any VMs.
  "RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"
  
  Steps to reproduce
  ==================
  
  1) Install 4 nodes with mitaka
  2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
  3) Upgrade to newton on the disable nodes
  4) compute=mitaka to [upgrade_levels]
  5) db sync
  6) Start newton
  7) Try to place any VMs, it will fail
- 8) see nova-compute.log on the mitaka nodes
+ 8) See nova-compute.log on the mitaka nodes
  
  Expected result
  ===============
  
  Successful upgrade one half of cluster, then another half
  
  Actual result
  =============
  
  Nova can't place any VMs.
  
  Compute logs:
  
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     task(self, context)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     use_slave=True)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args, kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args=args, kwargs=kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=self.retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     raise result
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task
  
  nova-conductor:
  
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     :param incoming: incoming message
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     objname, object_versions[objname])
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     supported=latest_ver)
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
  2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', '  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']
  
  Environment
  ===========
  
  CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
  23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  
  mitaka nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-compute-13.1.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  
  contoller:
  openstack-nova-scheduler-13.1.2-1.el7.noarch
  openstack-nova-conductor-13.1.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-api-13.1.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  openstack-nova-console-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-novncproxy-13.1.2-1.el7.noarch
  
  netonw nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-14.0.2-1.el7.noarch
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-compute-14.0.2-1.el7.noarch
  
  controller:
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-console-14.0.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-conductor-14.0.2-1.el7.noarch
  openstack-nova-novncproxy-14.0.2-1.el7.noarch
  openstack-nova-api-14.0.2-1.el7.noarch
  openstack-nova-scheduler-14.0.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-14.0.2-1.el7.noarch

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1652233

Title:
  mitaka is incompatible with newton

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  I get an error after upgrade half of my cluster. Can't place any VMs.
  "RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"

  Steps to reproduce
  ==================

  1) Install 4 nodes with mitaka
  2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
  3) Upgrade to newton on the disable nodes
  4) compute=mitaka to [upgrade_levels]
  5) db sync
  6) Start newton
  7) Try to place any VMs, it will fail
  8) See nova-compute.log on the mitaka nodes

  Expected result
  ===============

  Successful upgrade one half of cluster, then another half

  Actual result
  =============

  Nova can't place any VMs.

  Compute logs:

  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     task(self, context)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     use_slave=True)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args, kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args=args, kwargs=kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=self.retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     raise result
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task

  nova-conductor:

  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     :param incoming: incoming message
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     objname, object_versions[objname])
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     supported=latest_ver)
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
  2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', '  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']

  Environment
  ===========

  CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
  23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  mitaka nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-compute-13.1.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch

  contoller:
  openstack-nova-scheduler-13.1.2-1.el7.noarch
  openstack-nova-conductor-13.1.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-api-13.1.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  openstack-nova-console-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-novncproxy-13.1.2-1.el7.noarch

  netonw nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-14.0.2-1.el7.noarch
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-compute-14.0.2-1.el7.noarch

  controller:
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-console-14.0.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-conductor-14.0.2-1.el7.noarch
  openstack-nova-novncproxy-14.0.2-1.el7.noarch
  openstack-nova-api-14.0.2-1.el7.noarch
  openstack-nova-scheduler-14.0.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-14.0.2-1.el7.noarch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1652233/+subscriptions


Follow ups