yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1521340] Re: BDMNotFound race while building resources since 11/27

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Doug Hellmann <doug@xxxxxxxxxxxxxxxx>
Date: Thu, 03 Dec 2015 21:33:50 -0000
Reply-to: Bug 1521340 <1521340@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1521340

Title:
  BDMNotFound race while building resources since 11/27

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  http://logs.openstack.org/94/242594/2/check/gate-tempest-dsvm-
  nova-v20-api/085f243/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-11-30_18_03_48_003

  2015-11-30 18:03:48.003 ERROR nova.compute.manager [req-b3bc5257-4673-4850-9ba0-e1e136c131b3 tempest-DeleteServersTestJSON-651179004 tempest-DeleteServersTestJSON-684647951] [instance: 75986322-d59d-411c-a60c-776415cdd6d1] Failure prepping block device
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] Traceback (most recent call last):
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2132, in _build_resources
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     block_device_mapping)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1689, in _default_block_device_names
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     root_bdm.save()
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 206, in wrapper
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     ctxt, self, fn.__name__, args, kwargs)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 246, in object_action
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     objmethod=objmethod, args=args, kwargs=kwargs)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     retry=self.retry)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     timeout=timeout, retry=retry)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 464, in send
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     retry=retry)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 455, in _send
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     raise result
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] BDMNotFound_Remote: No Block Device Mapping with id 48.
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] Traceback (most recent call last):
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/opt/stack/new/nova/nova/conductor/manager.py", line 85, in _object_dispatch
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     return getattr(target, method)(*args, **kwargs)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 222, in wrapper
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     return fn(self, *args, **kwargs)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]   File "/opt/stack/new/nova/nova/objects/block_device.py", line 183, in save
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1]     raise exception.BDMNotFound(id=self.id)
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] BDMNotFound: No Block Device Mapping with id 48.
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 
  2015-11-30 18:03:48.003 4993 ERROR nova.compute.manager [instance: 75986322-d59d-411c-a60c-776415cdd6d1] 

  This started spiking on 11/27.  I suspect
  https://review.openstack.org/#/c/194063/ introduced the race, which
  was a 2nd attempt a change which was reverted previously because it
  introduced a race.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22BDMNotFound:%20No%20Block%20Device%20Mapping%20with%20id%5C%22%20AND%20message:%5C%22_default_block_device_names%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22%20AND%20NOT%20build_queue:%5C%22experimental%5C%22

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1521340/+subscriptions
References

[Bug 1521340] [NEW] BDMNotFound race while building resources since 11/27
From: Matt Riedemann, 2015-11-30