yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1418509] Re: nova.cells.messaging: Error processing message locally: 'dict' object has no attribute '_sa_instance_state'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Davanum Srinivas \(DIMS\)" <davanum@xxxxxxxxx>
Date: Wed, 04 Mar 2015 21:59:02 -0000
Reply-to: Bug 1418509 <1418509@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
looks like this was a transient failure.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1418509

Title:
  nova.cells.messaging:  Error processing message locally: 'dict' object
  has no attribute '_sa_instance_state'

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Noticed this in 'check-devstack-dsvm-cells' job failure:

  Specific errors from tracebacks in logs (screen-n-cell-region.txt.gz,
  screen-n-cell-child.txt.gz:

  [. . .]
  2015-02-05 10:03:43.694 ERROR nova.cells.messaging [req-140b752f-0e02-4fa3-a781-e3bc39018238 demo demo] Error processing message locally: 'dict' object has no attribute '_sa_instance_state'
  [. . .]
  2015-02-05 10:05:30.905 ERROR nova.cells.messaging [req-9fa83855-01db-455c-92cc-be73b85dc57f demo demo] Error processing message locally: Parent instance <InstanceExtra at 0x7f2ebb250850> is not bound to a Session; deferred load operation of attribute 'numa_topology' cannot proceed
  [. . .]

  
  Traceback from  'screen-n-cell-region.txt.gz'[1]:
  ---------------------------------------------------------------------
  2015-02-05 10:03:43.694 ERROR nova.cells.messaging [req-140b752f-0e02-4fa3-a781-e3bc39018238 demo demo] Error processing message locally: 'dict' object has no attribute '_sa_instance_state'
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging Traceback (most recent call last):
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 200, in _process_locally
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 1296, in _process_message_locally
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 1061, in instance_update_at_top
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     instance, update_cells=False)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/db/api.py", line 749, in instance_update
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     rv = IMPL.instance_update(context, instance_uuid, values)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 146, in wrapper
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     return f(*args, **kwargs)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2375, in instance_update
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     instance_ref = _instance_update(context, instance_uuid, values)[1]
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 184, in wrapped
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     return f(*args, **kwargs)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 2491, in _instance_update
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     instance_ref.update(values)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 86, in update
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     setattr(self, k, v)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 226, in __set__
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     instance_dict(instance), value, None)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 812, in set
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     value = self.fire_replace_event(state, dict_, value, old, initiator)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 832, in fire_replace_event
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     state, value, previous, initiator or self._replace_token)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 84, in set_
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging     newvalue_state = attributes.instance_state(newvalue)
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging AttributeError: 'dict' object has no attribute '_sa_instance_state'
  2015-02-05 10:03:43.694 28965 TRACE nova.cells.messaging 
  ---------------------------------------------------------------------

  
  Traceback, from 'screen-n-cell-child.txt.gz'[2]:
  ---------------------------------------------------------------------
  2015-02-05 10:05:30.905 ERROR nova.cells.messaging [req-9fa83855-01db-455c-92cc-be73b85dc57f demo demo] Error processing message locally: Parent instance <InstanceExtra at 0x7f2ebb250850> is not bound to a Session; deferred load operation of attribute 'numa_topology' cannot proceed
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging Traceback (most recent call last):
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 200, in _process_locally
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 1296, in _process_message_locally
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 830, in instance_update_from_api
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     expected_task_state=expected_task_state)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/base.py", line 206, in wrapper
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return fn(self, ctxt, *args, **kwargs)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/instance.py", line 825, in save
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     cells_api.instance_update_at_top(context, inst_ref)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/rpcapi.py", line 177, in instance_update_at_top
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     instance_p = jsonutils.to_primitive(instance)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 152, in to_primitive
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return recursive(dict(value.iteritems()), level=level + 1)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 139, in to_primitive
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return dict((k, recursive(v)) for k, v in six.iteritems(value))
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 139, in <genexpr>
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return dict((k, recursive(v)) for k, v in six.iteritems(value))
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 152, in to_primitive
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return recursive(dict(value.iteritems()), level=level + 1)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 93, in iteritems
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     local = dict(self)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/six.py", line 535, in next
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return type(self).__next__(self)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/models.py", line 112, in __next__
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return n, getattr(self.model, n)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     return self.impl.get(instance_state(instance), dict_)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 591, in get
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     value = self.callable_(state, passive)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 278, in _load_for_state
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging     (orm_util.state_str(state), self.key)
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging DetachedInstanceError: Parent instance <InstanceExtra at 0x7f2ebb250850> is not bound to a Session; deferred load operation of attribute 'numa_topology' cannot proceed
  2015-02-05 10:05:30.905 28974 TRACE nova.cells.messaging 
  ---------------------------------------------------------------------

  
  [1] http://logs.openstack.org/30/148630/7/check/check-devstack-dsvm-cells/b5f4d82/logs/screen-n-cell-region.txt.gz
  [2] http://logs.openstack.org/30/148630/7/check/check-devstack-dsvm-cells/b5f4d82/logs/screen-n-cell-child.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1418509/+subscriptions
References

[Bug 1418509] [NEW] nova.cells.messaging: Error processing message locally: 'dict' object has no attribute '_sa_instance_state'
From: Kashyap Chamarthy, 2015-02-05