← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #32375

[Bug 1448302] [NEW] cells: intermittent KeyError when deleting instance metadata

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This shows up quite often in the cells jobs but it's not contributing to
overall failures (from what I can tell) but we should get the ugly trace
cleaned up:

http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
cells/ce91d7e/logs/screen-n-cell-
child.txt.gz?level=TRACE#_2015-04-24_15_28_53_498

2015-04-24 15:28:53.498 ERROR nova.cells.messaging [req-6918b2e1-899d-49ff-a42e-b0c84e61b7ff AuthorizationTestJSON-1347876921 AuthorizationTestJSON-1581495686] Error processing message locally
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging Traceback (most recent call last):
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 200, in _process_locally
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 1285, in _process_message_locally
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 698, in run_compute_api_method
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 226, in wrapped
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 215, in inner
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 196, in inner
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 3150, in delete_instance_metadata
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     instance.delete_metadata_key(key)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/base.py", line 208, in wrapper
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(self, *args, **kwargs)
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/instance.py", line 1058, in delete_metadata_key
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     del self.metadata[key]
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging KeyError: u'meta1'
2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging 

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3IgcHJvY2Vzc2luZyBtZXNzYWdlIGxvY2FsbHlcIiBBTkQgbWVzc2FnZTpcImRlbGV0ZV9tZXRhZGF0YV9rZXlcIiBBTkQgbWVzc2FnZTpcIktleUVycm9yXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jZWxsLWNoaWxkLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5OTA3ODIzMTAzfQ==

** Affects: nova
     Importance: Low
         Status: Confirmed


** Tags: cells

** Changed in: nova
   Importance: Undecided => Low

** Changed in: nova
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1448302

Title:
  cells: intermittent KeyError when deleting instance metadata

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  This shows up quite often in the cells jobs but it's not contributing
  to overall failures (from what I can tell) but we should get the ugly
  trace cleaned up:

  http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
  cells/ce91d7e/logs/screen-n-cell-
  child.txt.gz?level=TRACE#_2015-04-24_15_28_53_498

  2015-04-24 15:28:53.498 ERROR nova.cells.messaging [req-6918b2e1-899d-49ff-a42e-b0c84e61b7ff AuthorizationTestJSON-1347876921 AuthorizationTestJSON-1581495686] Error processing message locally
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging Traceback (most recent call last):
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 200, in _process_locally
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     resp_value = self.msg_runner._process_message_locally(self)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 1285, in _process_message_locally
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(message, **message.method_kwargs)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/cells/messaging.py", line 698, in run_compute_api_method
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(message.ctxt, *args, **method_info['method_kwargs'])
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 226, in wrapped
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return func(self, context, target, *args, **kwargs)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 215, in inner
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return function(self, context, instance, *args, **kwargs)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 196, in inner
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return f(self, context, instance, *args, **kw)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/compute/api.py", line 3150, in delete_instance_metadata
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     instance.delete_metadata_key(key)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/base.py", line 208, in wrapper
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     return fn(self, *args, **kwargs)
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging   File "/opt/stack/new/nova/nova/objects/instance.py", line 1058, in delete_metadata_key
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging     del self.metadata[key]
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging KeyError: u'meta1'
  2015-04-24 15:28:53.498 32510 TRACE nova.cells.messaging 

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3IgcHJvY2Vzc2luZyBtZXNzYWdlIGxvY2FsbHlcIiBBTkQgbWVzc2FnZTpcImRlbGV0ZV9tZXRhZGF0YV9rZXlcIiBBTkQgbWVzc2FnZTpcIktleUVycm9yXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jZWxsLWNoaWxkLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5OTA3ODIzMTAzfQ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1448302/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next