yahoo-eng-team team mailing list archive

[Kashyap Chamarthy <kchamart@xxxxxxxxxx>]

** Also affects: libvirt (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1673483

Title:
  libvirt: test_attach_volume_shelved_or_offload_server times out
  waiting for device detach (which fails)

Status in OpenStack Compute (nova):
  Confirmed
Status in libvirt package in Ubuntu:
  New

Bug description:
  http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
  full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563

  The test teardown times out waiting for the volume to be detached from
  the server because the volume detach fails due to a libvirt i/o error
  during detach:

  http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-
  full-ubuntu-
  xenial/479c7bf/logs/screen-n-cpu.txt.gz?#_2017-03-16_01_01_32_897

  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [req-ac93339e-9eb1-4104-8743-5e5626253ab8 tempest-AttachVolumeShelveTestJSON-116820638 tempest-AttachVolumeShelveTestJSON-116820638] [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Failed to detach volume dc49318a-e51f-4812-97a3-64bf93e522eb from /dev/vdc
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] Traceback (most recent call last):
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/compute/manager.py", line 4833, in _driver_detach_volume
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     encryption=encryption)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1334, in detach_volume
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     live=live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 414, in detach_device_with_retry
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     _try_detach_device(conf, persistent, live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 408, in _try_detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     raise exception.DeviceNotFound(device=device)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.force_reraise()
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(self.type_, self.value, self.tb)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 393, in _try_detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self.detach_device(conf, persistent=persistent, live=live)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 450, in detach_device
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     self._domain.detachDeviceFlags(device_xml, flags=flags)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = execute(f, *args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     six.reraise(c, e, tb)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     rv = meth(*args, **kwargs)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1192, in detachDeviceFlags
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9]     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] libvirtError: End of file while reading data: Input/output error
  2017-03-16 01:01:32.897 7447 ERROR nova.compute.manager [instance: 5f67c47b-f7d7-4063-88f1-8f7b6177fcb9] 

  It's not clear if we're trying the detach twice, and the first time
  it's OK and then the 2nd fails with the i/o error, which is not in the
  list of expected libvirt errors we handle in this case.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1673483/+subscriptions