yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1652233] Re: mitaka is incompatible with newton - IncompatibleObjectVersion Version 2.1 of InstanceList is not supported

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dan Smith <dms@xxxxxxxxxxxxx>
Date: Tue, 03 Jan 2017 15:08:02 -0000
Reply-to: Bug 1652233 <1652233@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Yeah, mixed-version controllers isn't supported. We've made some
progress towards being able to support it in master, but it's definitely
not going to work in mitaka/newton.

You have to upgrade your controllers simultaneously (well, most
critically, your conductor services), and then you can have any mix of
versions among your computes that you want.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1652233

Title:
  mitaka is incompatible with newton - IncompatibleObjectVersion Version
  2.1 of InstanceList is not supported

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========

  I get an error after upgrade half of my cluster. Can't place any VMs.
  "RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported"

  Steps to reproduce
  ==================

  1) Install 4 nodes with mitaka
  2) Disable 2 nodes (1 api controller and 1 compute): nova service-disable
  3) Upgrade to newton on the disable nodes
  4) compute=mitaka to [upgrade_levels]
  5) db sync
  6) Start newton
  7) Try to place any VMs, it will fail
  8) See nova-compute.log on the mitaka nodes

  Expected result
  ===============

  Successful upgrade one half of cluster, then another half

  Actual result
  =============

  Nova can't place any VMs.

  Compute logs:

  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [req-41e6df10-b33b-47f5-be0c-86793cbcae6e - - - - -] Error during ComputeManager._sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task Traceback (most recent call last):
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     task(self, context)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1637, in _sync_scheduler_instance_info
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     use_slave=True)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args, kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 236, in object_class_action_versions
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     args=args, kwargs=kwargs)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=self.retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     timeout=timeout, retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     retry=retry)
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task     raise result
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task RemoteError: Remote error: IncompatibleObjectVersion Version 2.1 of InstanceList is not supported
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', u'  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', u'  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', u'  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', u'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n'].
  2016-12-23 07:26:11.434 15392 ERROR oslo_service.periodic_task

  nova-conductor:

  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Exception during message handling: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     :param incoming: incoming message
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     objname, object_versions[objname])
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher     supported=latest_ver)
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported
  2016-12-23 08:01:00.489 9958 ERROR oslo_messaging.rpc.dispatcher
  2016-12-23 08:01:00.491 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] Returning exception Version 2.1 of InstanceList is not supported to caller
  2016-12-23 08:01:00.492 9958 ERROR oslo_messaging._drivers.common [req-7a6eccb3-ade3-4c9f-b6f2-1588f9eb7ed0 - - - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    :param incoming: incoming message\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 92, in object_class_action_versions\n    objname, object_versions[objname])\n', '  File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 374, in obj_class_from_name\n    supported=latest_ver)\n', 'IncompatibleObjectVersion: Version 2.1 of InstanceList is not supported\n']

  Environment
  ===========

  CentOS Linux 7 (Core) 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
  23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  mitaka nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-compute-13.1.2-1.el7.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-common-13.1.2-1.el7.noarch

  contoller:
  openstack-nova-scheduler-13.1.2-1.el7.noarch
  openstack-nova-conductor-13.1.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-13.1.2-1.el7.noarch
  openstack-nova-api-13.1.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-common-13.1.2-1.el7.noarch
  openstack-nova-console-13.1.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  openstack-nova-novncproxy-13.1.2-1.el7.noarch

  netonw nodes:
  compute:
  python2-novaclient-3.3.2-1.el7.noarch
  python-nova-14.0.2-1.el7.noarch
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-compute-14.0.2-1.el7.noarch

  controller:
  openstack-nova-common-14.0.2-1.el7.noarch
  openstack-nova-console-14.0.2-1.el7.noarch
  novameter-2016.2-3.noarch
  openstack-nova-conductor-14.0.2-1.el7.noarch
  openstack-nova-novncproxy-14.0.2-1.el7.noarch
  openstack-nova-api-14.0.2-1.el7.noarch
  openstack-nova-scheduler-14.0.2-1.el7.noarch
  python2-novaclient-3.3.2-1.el7.noarch
  nova-solver-compute-lib-2016.2-2.noarch
  python-nova-14.0.2-1.el7.noarch

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1652233/+subscriptions
References

[Bug 1652233] [NEW] mitaka is incompatible with newton
From: Jack Ivanov, 2016-12-23