yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1338451] Re: shelve api does not work in the nova-cell environment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Gandelman <1338451@xxxxxxxxxxxxxxxxxx>
Date: Mon, 29 Sep 2014 19:59:44 -0000
Reply-to: Bug 1338451 <1338451@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/icehouse
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1338451

Title:
  shelve api does not work in the nova-cell environment

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  New

Bug description:
  If you run nova shelve api in nova-cell environment It throws
  following error:

  Nova cell (n-cell-child) Logs:

  2014-07-06 23:57:13.445 ERROR nova.cells.messaging [req-a689a1a1-4634-4634-974a-7343b5554f46 admin admin] Error processing message locally: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging Traceback (most recent call last):
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 200, in _process_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1287, in _process_message_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 700, in run_compute_api_method
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 192, in wrapped
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 182, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 163, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 2458, in shelve
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     instance.save(expected_task_state=[None])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging TypeError: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging

  Nova compute log:

  2014-07-07 00:05:19.084 ERROR oslo.messaging.rpc.dispatcher [req-9539189d-239b-4e74-8aea-8076740
  31c2f admin admin] Exception during message handling: 'NoneType' object is not iterable
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
  dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
  dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
  do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
      system_metadata, extra_usage_info)

    File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
      ignore_missing_network_data)

    File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
      macs = [vif['address'] for vif in nw_info]

  TypeError: 'NoneType' object is not iterable

  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dis
  t-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     pass
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 308, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 3847, in shelve_instance
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     self.conductor_api.notify_usage_exists(
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/api.py", line 206, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 320, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     extra_usage_info=extra_usage_info_p)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=self.retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     raise result
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     ignore_missing_network_data)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     macs = [vif['address'] for vif in nw_info]
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.093 ERROR oslo.messaging._drivers.common [req-9539189d-239b-4e74-8aea-807674031c2f admin admin] Returning exception 'NoneType' object is not iterable
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
      system_metadata, extra_usage_info)

    File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
      ignore_missing_network_data)

    File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
      macs = [vif['address'] for vif in nw_info]

  TypeError: 'NoneType' object is not iterable
   to caller


  Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve
  methods expect an Instance object, but cell is still passing the sqlalchemy form.

  Also 'info_cache' and 'metadata' are not present in instance-object and shelve requires these
  properties to be present in Instance object.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1338451/+subscriptions
References

[Bug 1338451] [NEW] shelve api does not work in the nova-cell environment
From: Abhijeet Malawade, 2014-07-07