yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1673483] [NEW] libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 16 Mar 2017 15:01:57 -0000
Reply-to: Bug 1673483 <1673483@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563

The test teardown times out waiting for the volume to be detached from
the server because the volume detach fails due to a libvirt i/o error
during detach:

http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
full-ubuntu-
xenial/479c7bf/logs/screen-n-cpu.txt.gz?#_2017-03-16_01_01_32_897

2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [req-ac93339e-9eb1-4104-8743-5e5626253ab8 tempest-AttachVolumeShelveTestJSON-116820638 tempest-AttachVolumeShelveTestJSON-116820638] [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Failed to detach volume dc49318a-e51f-4812-97a3-64bf93e522eb from /dev/vdc
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Traceback (most recent call last):
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4833, in _driver_detach_volume
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     encryption=encryption)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1334, in detach_volume
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     live=live)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 414, in detach_device_with_retry
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     _try_detach_device(conf, persistent, live)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 408, in _try_detach_device
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     raise exception.DeviceNotFound(device=device)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.force_reraise()
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(self.type_, self.value, self.tb)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 393, in _try_detach_device
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.detach_device(conf, persistent=persistent, live=live)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 450, in detach_device
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self._domain.detachDeviceFlags(device_xml, flags=flags)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = execute(f, *args, **kwargs)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(c, e, tb)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = meth(*args, **kwargs)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1192, in detachDeviceFlags
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] libvirtError: End of file while reading data: Input/output error
2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] 

It's not clear if we're trying the detach twice, and the first time it's
OK and then the 2nd fails with the i/o error, which is not in the list
of expected libvirt errors we handle in this case.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1673483

Title:
  libvirt: test_attach_volume_shelved_or_offload_server times out
  waiting for device detach (which fails)

Status in OpenStack Compute (nova):
  New

Bug description:
  http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
  full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563

  The test teardown times out waiting for the volume to be detached from
  the server because the volume detach fails due to a libvirt i/o error
  during detach:

  http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/479c7bf/logs/screen-n-cpu.txt.gz?#_2017-03-16_01_01_32_897

  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [req-ac93339e-9eb1-4104-8743-5e5626253ab8 tempest-AttachVolumeShelveTestJSON-116820638 tempest-AttachVolumeShelveTestJSON-116820638] [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Failed to detach volume dc49318a-e51f-4812-97a3-64bf93e522eb from /dev/vdc
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Traceback (most recent call last):
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4833, in _driver_detach_volume
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     encryption=encryption)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1334, in detach_volume
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     live=live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 414, in detach_device_with_retry
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     _try_detach_device(conf, persistent, live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 408, in _try_detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     raise exception.DeviceNotFound(device=device)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.force_reraise()
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(self.type_, self.value, self.tb)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 393, in _try_detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.detach_device(conf, persistent=persistent, live=live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 450, in detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self._domain.detachDeviceFlags(device_xml, flags=flags)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = execute(f, *args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(c, e, tb)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = meth(*args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1192, in detachDeviceFlags
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] libvirtError: End of file while reading data: Input/output error
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] 

  It's not clear if we're trying the detach twice, and the first time
  it's OK and then the 2nd fails with the i/o error, which is not in the
  list of expected libvirt errors we handle in this case.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1673483/+subscriptions
Follow ups

[Bug 1673483] Re: libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)
From: Kashyap Chamarthy, 2017-03-17