yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1522791] [NEW] horizon boot from image (create a new volume) fails with ceph backend

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Mihai Gheorghe <m.capsali@xxxxxxxxx>
Date: Fri, 04 Dec 2015 11:21:15 -0000
Reply-to: Bug 1522791 <1522791@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi,

This is bugging me since Kilo. Whenever i try to launch an instance from
horizon with boot from image (create a new volume), it imediatly enters
error state with invalid block device mapping as an error in dashboard.
I am using ceph rbd as backend for cinder and glance.

When booting from cli the process works as expected with the instace
booting from a newly created volume. From what i can tell it is a wait
time problem. Nova tries to attach the volume before the image gets
downloaded from glance and it tries to attach volume None as /dev/vda.

The only errors i found where in nova-compute.log : 
"2015-12-04 12:58:42.924 5957 WARNING nova.virt.libvirt.driver [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Ignoring supplied device name: /dev/vda. Libvirt can't honour user-supplied dev names
2015-12-04 12:58:43.004 5957 INFO nova.virt.block_device [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Booting with volume None at /dev/vda
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Instance failed block device setup
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Traceback (most recent call last):
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1738, in _prep_block_device
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     wait_func=self._await_block_device_map_created)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 476, in attach_block_devices
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     map(_log_and_attach, block_device_mapping)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 474, in _log_and_attach
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     bdm.attach(*attach_args, **attach_kwargs)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 383, in attach
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     availability_zone=av_zone)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 385, in create
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     raise exception.InvalidInput(reason=e)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] InvalidInput: Invalid input received: Invalid input received: Availability zone 'compute3' is invalid. (HTTP 400) (Request-ID: req-b00c4435-26dc-4e2e-b346-3b26f3a972f4)
2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] 
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Build of instance f8bd22e2-0826-4f80-b6d8-beba72488acc aborted: Block Device Mapping is Invalid.
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Traceback (most recent call last):
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     filter_properties)
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2025, in _build_and_run_instance
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     'create.error', fault=e)
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     six.reraise(self.type_, self.value, self.tb)
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1996, in _build_and_run_instance
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     block_device_mapping) as resources:
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     return self.gen.next()
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2143, in _build_resources
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     reason=e.format_message())
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] BuildAbortException: Build of instance f8bd22e2-0826-4f80-b6d8-beba72488acc aborted: Block Device Mapping is Invalid.
2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] 
"

Steps to reproduce: 
1. Ceph is backend to cinder and glance
2. From dashboard launch an instance with "boot from image (create a new volume)".
3. Get the error saying invalid block device mapping!

Sorry if this bug has already been posted but i couldn't find it
anywhere!

** Affects: horizon
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1522791

Title:
  horizon boot from image (create a new volume) fails with ceph backend

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  Hi,

  This is bugging me since Kilo. Whenever i try to launch an instance
  from horizon with boot from image (create a new volume), it imediatly
  enters error state with invalid block device mapping as an error in
  dashboard. I am using ceph rbd as backend for cinder and glance.

  When booting from cli the process works as expected with the instace
  booting from a newly created volume. From what i can tell it is a wait
  time problem. Nova tries to attach the volume before the image gets
  downloaded from glance and it tries to attach volume None as /dev/vda.

  The only errors i found where in nova-compute.log : 
  "2015-12-04 12:58:42.924 5957 WARNING nova.virt.libvirt.driver [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Ignoring supplied device name: /dev/vda. Libvirt can't honour user-supplied dev names
  2015-12-04 12:58:43.004 5957 INFO nova.virt.block_device [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Booting with volume None at /dev/vda
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Instance failed block device setup
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Traceback (most recent call last):
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1738, in _prep_block_device
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     wait_func=self._await_block_device_map_created)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 476, in attach_block_devices
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     map(_log_and_attach, block_device_mapping)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 474, in _log_and_attach
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     bdm.attach(*attach_args, **attach_kwargs)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 383, in attach
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     availability_zone=av_zone)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 385, in create
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     raise exception.InvalidInput(reason=e)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] InvalidInput: Invalid input received: Invalid input received: Availability zone 'compute3' is invalid. (HTTP 400) (Request-ID: req-b00c4435-26dc-4e2e-b346-3b26f3a972f4)
  2015-12-04 12:58:43.226 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] 
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [req-f7951875-56e0-4205-8cab-29b6509ea00b 9d6fed2183e54ce69de81341e86b37f9 610d740d5e0148b481401e6c4c79ddca - - -] [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Build of instance f8bd22e2-0826-4f80-b6d8-beba72488acc aborted: Block Device Mapping is Invalid.
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] Traceback (most recent call last):
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     filter_properties)
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2025, in _build_and_run_instance
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     'create.error', fault=e)
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     six.reraise(self.type_, self.value, self.tb)
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1996, in _build_and_run_instance
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     block_device_mapping) as resources:
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     return self.gen.next()
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2143, in _build_resources
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc]     reason=e.format_message())
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] BuildAbortException: Build of instance f8bd22e2-0826-4f80-b6d8-beba72488acc aborted: Block Device Mapping is Invalid.
  2015-12-04 12:58:43.374 5957 ERROR nova.compute.manager [instance: f8bd22e2-0826-4f80-b6d8-beba72488acc] 
  "

  Steps to reproduce: 
  1. Ceph is backend to cinder and glance
  2. From dashboard launch an instance with "boot from image (create a new volume)".
  3. Get the error saying invalid block device mapping!

  Sorry if this bug has already been posted but i couldn't find it
  anywhere!

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1522791/+subscriptions
Follow ups

[Bug 1522791] Re: horizon boot from image (create a new volume) fails with ceph backend
From: Launchpad Bug Tracker, 2017-10-16