← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #25446

[Bug 1401288] [NEW] "Failure prepping block device" error when creating volumes

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I get a "Failure prepping block device" error whenever I create an
instance with a volume or just a volume. It has been happening a few
times after setup, then I was able to create an instance once, now it
again occurs every time. I tried at least 10 times with various
configurations (empty volume, from image), also tried rebooting in
between.

I run RDO Juno, CentOS 7, XFS + MD-RAID1. It is a fresh installation, I
didn't really change much other than adding a few images.

SSH access can be provided for debugging purposes.

Relevant nova-compute log:
https://gist.githubusercontent.com/jgillich/fac1667fac71f5f459a6/raw/2794e3434f39d0333e70145a30a5c18ebbe34db5/gistfile1.txt

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  I get a "Failure prepping block device" error whenever I create an
  instance with a volume or just a volume. It has been happening a few
  times after setup, then I was able to create an instance once, now it
  again occurs every time. I tried at least 10 times with various
  configurations (empty volume, from image), also tried rebooting in
  between.
  
  I run RDO Juno, CentOS 7, XFS + MD-RAID1. It is a fresh installation, I
  didn't really change much other than adding a few images.
  
  SSH access can be provided for debugging purposes.
  
  Relevant nova-compute log:
- 
- 2014-12-10 22:25:20.978 3841 AUDIT nova.compute.manager [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Starting instance...
- 2014-12-10 22:25:21.133 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Attempting claim: memory 512 MB, disk 5 GB
- 2014-12-10 22:25:21.133 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Total memory: 7716 MB, used: 6656.00 MB
- 2014-12-10 22:25:21.134 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] memory limit: 11574.00 MB, free: 4918.00 MB
- 2014-12-10 22:25:21.134 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Total disk: 922 GB, used: 60.00 GB
- 2014-12-10 22:25:21.134 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] disk limit not specified, defaulting to unlimited
- 2014-12-10 22:25:21.158 3841 AUDIT nova.compute.claims [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Claim successful
- 2014-12-10 22:25:21.500 3841 INFO nova.scheduler.client.report [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] Compute_service record updated for ('openstack.localdomain', 'openstack.localdomain')
- 2014-12-10 22:25:21.703 3841 INFO nova.scheduler.client.report [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] Compute_service record updated for ('openstack.localdomain', 'openstack.localdomain')
- 2014-12-10 22:25:22.001 3841 AUDIT nova.virt.block_device [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Booting with volume None at /dev/vda
- 2014-12-10 22:25:22.002 3841 WARNING nova.volume.cinder [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] Cinder V1 API is deprecated as of the Juno release, and Nova is still configured to use it. Enable the V2 API in Cinder and set cinder_catalog_info in nova.conf to use it.
- 2014-12-10 22:25:22.808 3841 INFO nova.scheduler.client.report [-] Compute_service record updated for ('openstack.localdomain', 'openstack.localdomain')
- 2014-12-10 22:25:23.626 3841 WARNING nova.compute.manager [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] Volume id: b9642e1b-a964-41da-a722-521b96cec023 finished being created but was not set as 'available'
- 2014-12-10 22:25:23.811 3841 ERROR nova.compute.manager [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Instance failed block device setup
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Traceback (most recent call last):
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1818, in _prep_block_device
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     do_check_attach=do_check_attach) +
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 407, in attach_block_devices
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     map(_log_and_attach, block_device_mapping)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 405, in _log_and_attach
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     bdm.attach(*attach_args, **attach_kwargs)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 339, in attach
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     do_check_attach=do_check_attach)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 46, in wrapped
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     ret_val = method(obj, context, *args, **kwargs)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 229, in attach
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     volume_api.check_attach(context, volume, instance=instance)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 305, in check_attach
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     raise exception.InvalidVolume(reason=msg)
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] InvalidVolume: Invalid volume: status must be 'available'
- 2014-12-10 22:25:23.811 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] 
- 2014-12-10 22:25:23.827 3841 ERROR nova.compute.manager [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Failure prepping block device
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Traceback (most recent call last):
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2206, in _build_resources
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     block_device_mapping)
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1846, in _prep_block_device
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     raise exception.InvalidBDM()
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] InvalidBDM: Block Device Mapping is Invalid.
- 2014-12-10 22:25:23.827 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] 
- 2014-12-10 22:25:23.943 3841 INFO nova.scheduler.client.report [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] Compute_service record updated for ('openstack.localdomain', 'openstack.localdomain')
- 2014-12-10 22:25:23.953 3841 ERROR nova.compute.manager [req-62588275-2fdb-4257-ba95-050b6c68c2e6 None] [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Build of instance 74998751-c233-45d2-8582-2dfdaa7cdb90 aborted: Failure prepping block device.
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] Traceback (most recent call last):
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2014, in do_build_and_run_instance
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     filter_properties)
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2117, in _build_and_run_instance
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     'create.error', fault=e)
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     six.reraise(self.type_, self.value, self.tb)
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2091, in _build_and_run_instance
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     block_device_mapping) as resources:
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     return self.gen.next()
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2228, in _build_resources
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90]     reason=msg)
- 2014-12-10 22:25:23.953 3841 TRACE nova.compute.manager [instance: 74998751-c233-45d2-8582-2dfdaa7cdb90] BuildAbortException: Build of instance 74998751-c233-45d2-8582-2dfdaa7cdb90 aborted: Failure prepping block device.
+ https://gist.githubusercontent.com/jgillich/fac1667fac71f5f459a6/raw/2794e3434f39d0333e70145a30a5c18ebbe34db5/gistfile1.txt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1401288

Title:
  "Failure prepping block device" error when creating volumes

Status in OpenStack Compute (Nova):
  New

Bug description:
  I get a "Failure prepping block device" error whenever I create an
  instance with a volume or just a volume. It has been happening a few
  times after setup, then I was able to create an instance once, now it
  again occurs every time. I tried at least 10 times with various
  configurations (empty volume, from image), also tried rebooting in
  between.

  I run RDO Juno, CentOS 7, XFS + MD-RAID1. It is a fresh installation,
  I didn't really change much other than adding a few images.

  SSH access can be provided for debugging purposes.

  Relevant nova-compute log:
  https://gist.githubusercontent.com/jgillich/fac1667fac71f5f459a6/raw/2794e3434f39d0333e70145a30a5c18ebbe34db5/gistfile1.txt

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1401288/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next