yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1931209] [NEW] Circular reference detected during cold migration

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: norman shen <1931209@xxxxxxxxxxxxxxxxxx>
Date: Tue, 08 Jun 2021 06:43:13 -0000
Reply-to: Bug 1931209 <1931209@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

cold migration failed when server is specified with a numa topology

Steps to reproduce
==================

create server from a flavor specified with numa topology parameters and then do
a cold migrate or resize

Expected
========

success

Actual
======

failed with following messages

2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server [req-c299c521-2a07-483b-b19e-deb136572da0 dde6a5842265470a8e2f40938ae66097 f3d6994dfaf043479c9cf5bbac19ab87 - default default] Exception during message handling: ValueError: Circular reference detected
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 94, in wrapper
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/utils.py", line 1164, in decorated_function
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 298, in migrate_server
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     host_list)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 358, in _cold_migrate
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     updates, ex, request_spec)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     raise value
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 327, in _cold_migrate
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     task.execute()
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.rollback()
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.force_reraise()
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     raise value
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return original(self)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 42, in execute
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return self._execute()
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 295, in _execute
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_serialization/jsonutils.py", line 213, in dumps
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return json.dumps(obj, default=default, **kwargs)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     **kw).encode(obj)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     chunks = self.iterencode(o, _one_shot=True)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return _iterencode(o, 0)
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server ValueError: Circular reference detected
2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server

properties                 | SERVICE='ECS', SPEC='GENERAL',
hw:cpu_cores='4', hw:cpu_policy='dedicated', hw:cpu_sockets='1',
hw:cpu_thread_policy='isolate', hw:cpu_threads='1',
hw:emulator_threads_policy='share', quota:vif_inbound_average='1048576',
quota:vif_outbound_average='1048576' |

** Affects: nova
     Importance: Undecided
         Status: Invalid

** Changed in: nova
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1931209

Title:
  Circular reference detected during cold migration

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========

  cold migration failed when server is specified with a numa topology

  Steps to reproduce
  ==================

  create server from a flavor specified with numa topology parameters and then do
  a cold migrate or resize

  Expected
  ========

  success

  Actual
  ======

  failed with following messages

  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server [req-c299c521-2a07-483b-b19e-deb136572da0 dde6a5842265470a8e2f40938ae66097 f3d6994dfaf043479c9cf5bbac19ab87 - default default] Exception during message handling: ValueError: Circular reference detected
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return func(*args, **kwargs)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 94, in wrapper
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return fn(self, context, *args, **kwargs)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/compute/utils.py", line 1164, in decorated_function
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 298, in migrate_server
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     host_list)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 358, in _cold_migrate
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     updates, ex, request_spec)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     raise value
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/manager.py", line 327, in _cold_migrate
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     task.execute()
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.rollback()
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/six.py", line 693, in reraise
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     raise value
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return original(self)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/base.py", line 42, in execute
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return self._execute()
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/nova/conductor/tasks/migrate.py", line 295, in _execute
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/lib/python3.6/site-packages/oslo_serialization/jsonutils.py", line 213, in dumps
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return json.dumps(obj, default=default, **kwargs)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     **kw).encode(obj)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     chunks = self.iterencode(o, _one_shot=True)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server     return _iterencode(o, 0)
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server ValueError: Circular reference detected
  2021-06-08 04:24:52.963 19 ERROR oslo_messaging.rpc.server

  properties                 | SERVICE='ECS', SPEC='GENERAL',
  hw:cpu_cores='4', hw:cpu_policy='dedicated', hw:cpu_sockets='1',
  hw:cpu_thread_policy='isolate', hw:cpu_threads='1',
  hw:emulator_threads_policy='share',
  quota:vif_inbound_average='1048576',
  quota:vif_outbound_average='1048576' |

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1931209/+subscriptions