yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712336] [NEW] prep_resize cannot call reschedule correctly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: dongbin <1712336@xxxxxxxxxxxxxxxxxx>
Date: Tue, 22 Aug 2017 12:33:55 -0000
Reply-to: Bug 1712336 <1712336@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When the prep_resize method performs an exception to resend the command
to conductor, the exception is as follows:


2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Exception during message handling: Circular reference detected
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     reservations, clean_shutdown, request_spec)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     updates, ex, legacy_spec)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     task.execute()
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.rollback()
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return original(self)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._execute()
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return json.dumps(obj, default=default, **kwargs)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return _iterencode(o, 0)
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher ValueError: Circular reference detected
2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher
2017-08-22 20:31:46.957 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Returning exception Circular reference detected to caller
2017-08-22 20:31:46.958 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server\n    reservations, clean_shutdown, request_spec)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate\n    updates, ex, legacy_spec)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate\n    task.execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap\n    self.rollback()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap\n    return original(self)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute\n    return self._execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute\n    legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))\n', '  File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps\n    return json.dumps(obj, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1712336

Title:
  prep_resize cannot call reschedule correctly

Status in OpenStack Compute (nova):
  New

Bug description:
  When the prep_resize method performs an exception to resend the
  command to conductor, the exception is as follows:

  
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Exception during message handling: Circular reference detected
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     reservations, clean_shutdown, request_spec)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     updates, ex, legacy_spec)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     task.execute()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.rollback()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return original(self)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return self._execute()
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return json.dumps(obj, default=default, **kwargs)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     sort_keys=sort_keys, **kw).encode(obj)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     chunks = self.iterencode(o, _one_shot=True)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher     return _iterencode(o, 0)
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher ValueError: Circular reference detected
  2017-08-22 20:31:46.955 2194187 ERROR oslo_messaging.rpc.dispatcher
  2017-08-22 20:31:46.957 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] Returning exception Circular reference detected to caller
  2017-08-22 20:31:46.958 2194187 ERROR oslo_messaging._drivers.common [req-203191f9-1633-491e-be04-2526a8d6c600 fa6a256122104fdc819a94202c96432a 415ece26aa5c4f1cb367607ed528df08 - - -] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 561, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 207, in migrate_server\n    reservations, clean_shutdown, request_spec)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 271, in _cold_migrate\n    updates, ex, legacy_spec)\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 240, in _cold_migrate\n    task.execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 27, in wrap\n    self.rollback()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n', '  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 24, in wrap\n    return original(self)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/base.py", line 42, in execute\n    return self._execute()\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/migrate.py", line 90, in _execute\n    legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec))\n', '  File "/usr/lib/python2.7/site-packages/oslo_serialization/jsonutils.py", line 178, in dumps\n    return json.dumps(obj, default=default, **kwargs)\n', '  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps\n    sort_keys=sort_keys, **kw).encode(obj)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n', '  File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode\n    return _iterencode(o, 0)\n', 'ValueError: Circular reference detected\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1712336/+subscriptions
Follow ups

[Bug 1712336] Re: prep_resize cannot call reschedule correctly
From: Matt Riedemann, 2017-08-22