yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1318544] Re: XenServer - Nova-Compute StorageError Waiting for device

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 17 Sep 2014 11:45:25 -0000
Reply-to: Bug 1318544 <1318544@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Long incomplete bug

** Changed in: nova
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1318544

Title:
  XenServer - Nova-Compute StorageError Waiting for device

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Hi All,

  I started building a Openstack cloud based on the new LTS version of
  Ubuntu (14.04). I installed both Control and Compute nodes as VM's on
  a XenServer. I selected the option 'Other Operating System' so this is
  running as HVM.

  The cluster is up and running: i can start instances, create storage
  and allocate IP addresses. But the error i got is popping up when the
  instance is in spawning state. The error i attached is from the nova-
  compute node.

  I use a IBM storwize backend with Cinder connection using iSCSI. I can
  see the instances being created, the storage being connected but the
  following error is keep coming back.

  Locally on the XenServer SDB and SDC existed, which leaded to a
  earlier error in which HDB was HDC. I removed SDC locally from the
  XenServer using: "echo 1 > /sys/block/sdc/device/delete". This changed
  the HDC error message to HDB. The same thing i did for SDB. Both where
  virtual drives from DRAC. But still this message is keep popping up
  with HDB.

  I read an other article about a similar error, this guy was pointed in
  the direction of a HVM vs PV solution because the VM is communicating
  with the DOM0 via it's kernel. I checked and after installing the
  XenServer tools the UUID is there and is readable.

  While trying the solution above i also focused on the HDB reference,
  this was strange because with all the rewrite options in nova i would
  expect a reference to xvda: "xenapi_remap_vbd_dev=false"

  Could you guys point me in the right direction or help me solve this
  problem? I think this isn't a mis-configuration because i tried all
  possible configurations and this message keeps popping up.

   This is the last step towards a working openstack cloud with Ubuntu
  14.04 and XenServer 6.2 all patches applied.

  2014-05-12 09:00:30.918 6660 ERROR nova.compute.manager [req-595bc50b-e8b2-4de4-8c38-09887c8d4c82 f8609ec1f5254fcfb1fdf6df76876805 41971b840c7a405d9da5052a2506a1c9] [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Error: Timeout waiting for device hdb to be created
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] Traceback (most recent call last):
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     set_access_ip=set_access_ip)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     return function(self, context, *args, **kwargs)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     six.reraise(self.type_, self.value, self.tb)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     block_device_info)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 230, in spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     admin_password, network_info, block_device_info)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 357, in spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     network_info, block_device_info, name_label, rescue)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 526, in _spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     undo_mgr.rollback_and_reraise(msg=msg, instance=instance)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 830, in rollback_and_reraise
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     self._rollback()
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     six.reraise(self.type_, self.value, self.tb)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 509, in _spawn
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     attach_devices_step(undo_mgr, vm_ref, vdis, disk_image_type)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 146, in inner
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     rv = f(*args, **kwargs)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 443, in attach_devices_step
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     attach_disks(undo_mgr, vm_ref, vdis, disk_image_type)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 390, in attach_disks
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     injected_files)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 595, in _attach_disks
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     flavor['root_gb'])
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1037, in try_auto_configure_disk
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _auto_configure_disk(session, vdi_ref, new_gb)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 1018, in _auto_configure_disk
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     with vdi_attached_here(session, vdi_ref, read_only=False) as dev:
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     return self.gen.next()
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2200, in vdi_attached_here
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _wait_for_device(dev)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]   File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vm_utils.py", line 2155, in _wait_for_device
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9]     _('Timeout waiting for device %s to be created') % dev)
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] StorageError: Timeout waiting for device hdb to be created
  2014-05-12 09:00:30.918 6660 TRACE nova.compute.manager [instance: 00e2c60e-372b-43d2-a2fd-bcc3628c24a9] 

  Best,
  - Thijs

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1318544/+subscriptions
References

[Bug 1318544] [NEW] XenServer - Nova-Compute StorageError Waiting for device
From: Thijs van Leeuwen, 2014-05-12