yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1338451] [NEW] shelve api does not work in the nova-cell environment

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Abhijeet Malawade <Abhijeet.Malawade@xxxxxxxxxxx>
Date: Mon, 07 Jul 2014 07:23:06 -0000
Reply-to: Bug 1338451 <1338451@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If you run nova shelve api in nova-cell environment It throws following
error:

Nova cell (n-cell-child) Logs:

2014-07-06 23:57:13.445 ERROR nova.cells.messaging [req-a689a1a1-4634-4634-974a-7343b5554f46 admin admin] Error processing message locally: save() got an unexpected keyword argument 'expected_task_state'
2014-07-06 23:57:13.445 TRACE nova.cells.messaging Traceback (most recent call last):
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 200, in _process_locally
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1287, in _process_message_locally
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 700, in run_compute_api_method
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 192, in wrapped
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 182, in inner
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 163, in inner
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 2458, in shelve
2014-07-06 23:57:13.445 TRACE nova.cells.messaging     instance.save(expected_task_state=[None])
2014-07-06 23:57:13.445 TRACE nova.cells.messaging TypeError: save() got an unexpected keyword argument 'expected_task_state'
2014-07-06 23:57:13.445 TRACE nova.cells.messaging

Nova compute log:

2014-07-07 00:05:19.084 ERROR oslo.messaging.rpc.dispatcher [req-9539189d-239b-4e74-8aea-8076740
31c2f admin admin] Exception during message handling: 'NoneType' object is not iterable
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
    system_metadata, extra_usage_info)

  File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
    ignore_missing_network_data)

  File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
    macs = [vif['address'] for vif in nw_info]

TypeError: 'NoneType' object is not iterable

2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dis
t-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     payload)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     pass
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 308, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 3847, in shelve_instance
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     self.conductor_api.notify_usage_exists(
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/api.py", line 206, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 320, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     extra_usage_info=extra_usage_info_p)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=self.retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout, retry=retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     raise result
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     ignore_missing_network_data)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     macs = [vif['address'] for vif in nw_info]
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.093 ERROR oslo.messaging._drivers.common [req-9539189d-239b-4e74-8aea-807674031c2f admin admin] Returning exception 'NoneType' object is not iterable
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
    system_metadata, extra_usage_info)

  File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
    ignore_missing_network_data)

  File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
    macs = [vif['address'] for vif in nw_info]

TypeError: 'NoneType' object is not iterable
 to caller


Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve
methods expect an Instance object, but cell is still passing the sqlalchemy form.

Also 'info_cache' and 'metadata' are not present in instance-object and shelve requires these
properties to be present in Instance object.

** Affects: nova
     Importance: Undecided
     Assignee: Abhijeet Malawade (abhijeet-malawade)
         Status: New


** Tags: ntt

** Description changed:

  If you run nova shelve api in nova-cell environment It throws following
  error:
  
  Nova cell (n-cell-child) Logs:
  
  2014-07-06 23:57:13.445 ERROR nova.cells.messaging [req-a689a1a1-4634-4634-974a-7343b5554f46 admin admin] Error processing message locally: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging Traceback (most recent call last):
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 200, in _process_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1287, in _process_message_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 700, in run_compute_api_method
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 192, in wrapped
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 182, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 163, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 2458, in shelve
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     instance.save(expected_task_state=[None])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging TypeError: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging
  
- 
  Nova compute log:
  
  2014-07-07 00:05:19.084 ERROR oslo.messaging.rpc.dispatcher [req-9539189d-239b-4e74-8aea-8076740
  31c2f admin admin] Exception during message handling: 'NoneType' object is not iterable
  Traceback (most recent call last):
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
  dispatch_and_reply
-     incoming.message))
+     incoming.message))
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
  dispatch
-     return self._do_dispatch(endpoint, method, ctxt, args)
+     return self._do_dispatch(endpoint, method, ctxt, args)
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
  do_dispatch
-     result = getattr(endpoint, method)(ctxt, **new_args)
+     result = getattr(endpoint, method)(ctxt, **new_args)
  
-   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
-     system_metadata, extra_usage_info)
+   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
+     system_metadata, extra_usage_info)
  
-   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
-     ignore_missing_network_data)
+   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
+     ignore_missing_network_data)
  
-   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
-     macs = [vif['address'] for vif in nw_info]
+   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
+     macs = [vif['address'] for vif in nw_info]
  
  TypeError: 'NoneType' object is not iterable
  
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dis
  t-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     pass
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 308, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 3847, in shelve_instance
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     self.conductor_api.notify_usage_exists(
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/api.py", line 206, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 320, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     extra_usage_info=extra_usage_info_p)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=self.retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     raise result
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     ignore_missing_network_data)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     macs = [vif['address'] for vif in nw_info]
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.093 ERROR oslo.messaging._drivers.common [req-9539189d-239b-4e74-8aea-807674031c2f admin admin] Returning exception 'NoneType' object is not iterable
  Traceback (most recent call last):
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
-     incoming.message))
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
+     incoming.message))
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
-     return self._do_dispatch(endpoint, method, ctxt, args)
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
+     return self._do_dispatch(endpoint, method, ctxt, args)
  
-   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
-     result = getattr(endpoint, method)(ctxt, **new_args)
+   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
+     result = getattr(endpoint, method)(ctxt, **new_args)
  
-   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
-     system_metadata, extra_usage_info)
+   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
+     system_metadata, extra_usage_info)
  
-   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
-     ignore_missing_network_data)
+   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
+     ignore_missing_network_data)
  
-   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
-     macs = [vif['address'] for vif in nw_info]
+   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
+     macs = [vif['address'] for vif in nw_info]
  
  TypeError: 'NoneType' object is not iterable
-  to caller
+  to caller
  
-  
- Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve 
+ 
+ Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve
  methods expect an Instance object, but cell is still passing the sqlalchemy form.
  
  Also 'info_cache' and 'metadata' are not present in instance-object and shelve requires these
  properties to be present in Instance object.

** Changed in: nova
     Assignee: (unassigned) => Abhijeet Malawade (abhijeet-malawade)

** Tags added: ntt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1338451

Title:
  shelve api does not work in the nova-cell environment

Status in OpenStack Compute (Nova):
  New

Bug description:
  If you run nova shelve api in nova-cell environment It throws
  following error:

  Nova cell (n-cell-child) Logs:

  2014-07-06 23:57:13.445 ERROR nova.cells.messaging [req-a689a1a1-4634-4634-974a-7343b5554f46 admin admin] Error processing message locally: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging Traceback (most recent call last):
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 200, in _process_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 1287, in _process_message_locally
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/cells/messaging.py", line 700, in run_compute_api_method
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 192, in wrapped
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 182, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 163, in inner
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging   File "/opt/stack/nova/nova/compute/api.py", line 2458, in shelve
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging     instance.save(expected_task_state=[None])
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging TypeError: save() got an unexpected keyword argument 'expected_task_state'
  2014-07-06 23:57:13.445 TRACE nova.cells.messaging

  Nova compute log:

  2014-07-07 00:05:19.084 ERROR oslo.messaging.rpc.dispatcher [req-9539189d-239b-4e74-8aea-8076740
  31c2f admin admin] Exception during message handling: 'NoneType' object is not iterable
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
  dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
  dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
  do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
      system_metadata, extra_usage_info)

    File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
      ignore_missing_network_data)

    File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
      macs = [vif['address'] for vif in nw_info]

  TypeError: 'NoneType' object is not iterable

  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dis
  t-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     pass
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 308, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 3847, in shelve_instance
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     self.conductor_api.notify_usage_exists(
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/api.py", line 206, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 320, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     extra_usage_info=extra_usage_info_p)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=self.retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     retry=retry)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     raise result
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     system_metadata, extra_usage_info)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     ignore_missing_network_data)
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher     macs = [vif['address'] for vif in nw_info]
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
  2014-07-07 00:05:19.093 ERROR oslo.messaging._drivers.common [req-9539189d-239b-4e74-8aea-807674031c2f admin admin] Returning exception 'NoneType' object is not iterable
  Traceback (most recent call last):

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      incoming.message))

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
      system_metadata, extra_usage_info)

    File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
      ignore_missing_network_data)

    File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
      macs = [vif['address'] for vif in nw_info]

  TypeError: 'NoneType' object is not iterable
   to caller


  Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve
  methods expect an Instance object, but cell is still passing the sqlalchemy form.

  Also 'info_cache' and 'metadata' are not present in instance-object and shelve requires these
  properties to be present in Instance object.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1338451/+subscriptions
Follow ups

[Bug 1338451] Re: shelve api does not work in the nova-cell environment
From: Adam Gandelman, 2014-10-02
[Bug 1338451] Re: shelve api does not work in the nova-cell environment
From: Adam Gandelman, 2014-09-29
[Bug 1338451] Re: shelve api does not work in the nova-cell environment
From: Russell Bryant, 2014-07-23
[Bug 1338451] [NEW] shelve api does not work in the nova-cell environment
From: Abhijeet Malawade, 2014-07-07
References

[Bug 1338451] [NEW] shelve api does not work in the nova-cell environment
From: Abhijeet Malawade, 2014-07-07