yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1518007] [NEW] volume attach test race fails in dsvm-tempest-xen CI due to: "libvirtError: internal error: libxenlight failed to attach disk 'xvdb'"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 19 Nov 2015 17:55:37 -0000
Reply-to: Bug 1518007 <1518007@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seen in a CI run here:

http://logs.openstack.xenproject.org/79/241179/9/check/dsvm-tempest-
xen/6901c94/logs/screen-n-cpu.txt.gz

This was unrelated to the change being tested:
https://review.openstack.org/#/c/241179/

2015-11-19 05:41:56.395 ERROR nova.compute.manager [req-0f579e1a-f928-4f24-8beb-8288a2699284 tempest-AttachVolumeTestJSON-174551048 tempest-AttachVolumeTestJSON-1798726384] [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] Failed to attach 853230d9-8e71-416b-a27b-0107ca181020 at /dev/xvdb
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] Traceback (most recent call last):
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4653, in _attach_volume
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     do_check_attach=False, do_driver_attach=True)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 56, in wrapped
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     ret_val = method(obj, context, *args, **kwargs)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 297, in attach
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     connector)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(self.type_, self.value, self.tb)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 288, in attach
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     device_type=self['device_type'], encryption=encryption)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1123, in attach_volume
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     self._disconnect_volume(connection_info, disk_dev)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(self.type_, self.value, self.tb)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1112, in attach_volume
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     guest.attach_device(conf, persistent=True, live=live)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 233, in attach_device
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     rv = execute(f, *args, **kwargs)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(c, e, tb)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     rv = meth(*args, **kwargs)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 554, in attachDeviceFlags
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] libvirtError: internal error: libxenlight failed to attach disk 'xvdb'
2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt xen

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1518007

Title:
  volume attach test race fails in dsvm-tempest-xen CI due to:
  "libvirtError: internal error: libxenlight failed to attach disk
  'xvdb'"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seen in a CI run here:

  http://logs.openstack.xenproject.org/79/241179/9/check/dsvm-tempest-
  xen/6901c94/logs/screen-n-cpu.txt.gz

  This was unrelated to the change being tested:
  https://review.openstack.org/#/c/241179/

  2015-11-19 05:41:56.395 ERROR nova.compute.manager [req-0f579e1a-f928-4f24-8beb-8288a2699284 tempest-AttachVolumeTestJSON-174551048 tempest-AttachVolumeTestJSON-1798726384] [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] Failed to attach 853230d9-8e71-416b-a27b-0107ca181020 at /dev/xvdb
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] Traceback (most recent call last):
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4653, in _attach_volume
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     do_check_attach=False, do_driver_attach=True)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 56, in wrapped
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     ret_val = method(obj, context, *args, **kwargs)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 297, in attach
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     connector)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(self.type_, self.value, self.tb)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/block_device.py", line 288, in attach
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     device_type=self['device_type'], encryption=encryption)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1123, in attach_volume
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     self._disconnect_volume(connection_info, disk_dev)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(self.type_, self.value, self.tb)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1112, in attach_volume
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     guest.attach_device(conf, persistent=True, live=live)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 233, in attach_device
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     self._domain.attachDeviceFlags(conf.to_xml(), flags=flags)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     rv = execute(f, *args, **kwargs)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     six.reraise(c, e, tb)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     rv = meth(*args, **kwargs)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 554, in attachDeviceFlags
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d] libvirtError: internal error: libxenlight failed to attach disk 'xvdb'
  2015-11-19 05:41:56.395 22173 ERROR nova.compute.manager [instance: 264d4140-70ad-4eae-b8b7-2602f52bcd3d]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1518007/+subscriptions