yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712336] Re: prep_resize cannot call reschedule correctly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 22 Aug 2017 18:45:12 -0000
Reply-to: Bug 1712336 <1712336@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Are you using devstack? If so, by default, devstack is running in
"superconductor" mode which won't allow "up-calls" from the nova-compute
service to the top-level conductor service to perform a reschedule.
That's explained here:

https://docs.openstack.org/nova/latest/user/cellsv2_layout.html
#operations-requiring-upcalls

And you can change devstack to perform reschedules by setting
CELLSV2_SETUP=singleconductor in your local.conf for devstack.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1712336

Title:
  prep_resize cannot call reschedule correctly

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  When the prep_resize method performs an exception to resend the
  command to conductor, the exception is as follows:

  
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Exception during message handling: Circular reference detected
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     reservations, clean_shutdown, request_spec)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     updates, ex, legacy_spec)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     task.execute()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.rollback()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return original(self)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._execute()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return json.dumps(obj, default=default, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return _iterencode(o, 0)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher ValueError: Circular reference detected
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher
  2017-08-22 20:31:46.957 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Returning exception Circular reference detected to caller
  2017-08-22 20:31:46.958 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server\n    reservations, clean_shutdown, request_spec)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate\n    updates, ex, legacy_spec)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate\n    task.execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap\n    self.rollback()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap\n    return original(self)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute\n    return self._execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute\n    legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))\n', '  File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps\n    return json.dumps(obj, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1712336/+subscriptions
References

[Bug 1712336] [NEW] prep_resize cannot call reschedule correctly
From: dongbin, 2017-08-22