yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

There is a lock being held in c-vol that is taking awhile, 16 seconds in
one case, I'm not sure if that's a lot or not:

http://logs.openstack.org/10/230510/2/gate/gate-grenade-
dsvm/e796b39/logs/new/screen-c-vol.txt.gz#_2015-10-06_05_40_15_936

2015-10-06 05:40:15.936 DEBUG oslo_concurrency.lockutils [req-
6b402bc1-3c24-47f8-be65-609bfb98c8fd tempest-
TestVolumeBootPatternV2-836916434] Lock "35fbbb48-90a3-45e2-9275
-d66c9b8e9b2b-delete_snapshot" released by
"cinder.volume.manager._run_flow_locked" :: held 16.103s inner
/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:265

** Also affects: cinder
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1503290

Title:
  grenade failures on mitaka side with failure prepping block device due
  to RPC timeout

Status in Cinder:
  New
Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/10/230510/2/gate/gate-grenade-
  dsvm/e796b39/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2015-10-06_05_39_36_682

  2015-10-06 05:39:36.682 ERROR nova.compute.manager [req-03e13a69-3da6-4bc6-bcab-3b2a3fe4d236 tempest-ServersTestManualDisk-1771971010 tempest-ServersTestManualDisk-1239478904] [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device.
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Traceback (most recent call last):
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1906, in _do_build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     filter_properties)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2034, in _build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     'create.error', fault=e)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     six.reraise(self.type_, self.value, self.tb)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1999, in _build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     block_device_mapping) as resources:
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     return self.gen.next()
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2165, in _build_resources
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     reason=msg)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] BuildAbortException: Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device.
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] 


  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2RvX2J1aWxkX2FuZF9ydW5faW5zdGFuY2VcIiBBTkQgbWVzc2FnZTpcImFib3J0ZWQ6IEZhaWx1cmUgcHJlcHBpbmcgYmxvY2sgZGV2aWNlXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDQxMzg1OTE3NjV9

  7 hits in 7 days, check and gate, all failures, master branch only.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1503290/+subscriptions