yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Cinder is not involved here, we're hitting the rpc timeout when calling
instance.save() in nova.compute.manager _build_resources here:

            LOG.debug('Start building block device mappings for instance.',
                      instance=instance)
            instance.vm_state = vm_states.BUILDING
            instance.task_state = task_states.BLOCK_DEVICE_MAPPING
            instance.save()

And conductor is in bad shape:

http://logs.openstack.org/10/230510/2/gate/gate-grenade-
dsvm/e796b39/logs/new/screen-n-cond.txt.gz?level=TRACE

When we failed, we're falling in between:

http://logs.openstack.org/10/230510/2/gate/gate-grenade-
dsvm/e796b39/logs/new/screen-n-cond.txt.gz#_2015-10-06_05_39_34_313

And

http://logs.openstack.org/10/230510/2/gate/gate-grenade-
dsvm/e796b39/logs/new/screen-n-cond.txt.gz#_2015-10-06_05_39_43_085

When nothing is happening.

** No longer affects: cinder

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1503290

Title:
  grenade failures on mitaka side with failure prepping block device due
  to RPC timeout

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/10/230510/2/gate/gate-grenade-
  dsvm/e796b39/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2015-10-06_05_39_36_682

  2015-10-06 05:39:36.682 ERROR nova.compute.manager [req-03e13a69-3da6-4bc6-bcab-3b2a3fe4d236 tempest-ServersTestManualDisk-1771971010 tempest-ServersTestManualDisk-1239478904] [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device.
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] Traceback (most recent call last):
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1906, in _do_build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     filter_properties)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2034, in _build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     'create.error', fault=e)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     six.reraise(self.type_, self.value, self.tb)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1999, in _build_and_run_instance
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     block_device_mapping) as resources:
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     return self.gen.next()
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2165, in _build_resources
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959]     reason=msg)
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] BuildAbortException: Build of instance 816e60bb-0085-4b0b-9a00-dbd24c339959 aborted: Failure prepping block device.
  2015-10-06 05:39:36.682 5634 ERROR nova.compute.manager [instance: 816e60bb-0085-4b0b-9a00-dbd24c339959] 


  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2RvX2J1aWxkX2FuZF9ydW5faW5zdGFuY2VcIiBBTkQgbWVzc2FnZTpcImFib3J0ZWQ6IEZhaWx1cmUgcHJlcHBpbmcgYmxvY2sgZGV2aWNlXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDQxMzg1OTE3NjV9

  7 hits in 7 days, check and gate, all failures, master branch only.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1503290/+subscriptions