yahoo-eng-team team mailing list archive

[OpenStack Infra <1815949@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/641480
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2a8ee40fccc65b177275d6fe80c10fdb83b86e1f
Submitter: Zuul
Branch:    master

commit 2a8ee40fccc65b177275d6fe80c10fdb83b86e1f
Author: Chris Friesen <chris.friesen@xxxxxxxxxxxxx>
Date:   Wed Mar 6 14:19:01 2019 -0600

    Add missing libvirt exception during device detach
    
    It turns out that when detaching a device libvirt can raise a
    libvirt.VIR_ERR_INTERNAL_ERROR exception with an error log of
    "unable to execute QEMU command 'device_del': Device <foo> not found".
    
    Add this exception to the existing "not found" case which currently
    handles only libvirt.VIR_ERR_OPERATION_FAILED.
    
    Change-Id: I3055cd7641de92ab188de73733ca9288a9ca730a
    Closes-Bug: #1815949
    Signed-off-by: Chris Friesen <chris.friesen@xxxxxxxxxxxxx>


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815949

Title:
  missing special-case libvirt exception during device detach

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) pike series:
  Confirmed
Status in OpenStack Compute (nova) queens series:
  Confirmed
Status in OpenStack Compute (nova) rocky series:
  Confirmed

Bug description:
  In Pike a customer has run into the following issue:

  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed: libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall Traceback (most recent call last):
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 143, in _run_loop
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 363, in _func
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     result = f(*args, **kwargs)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 505, in _do_wait_and_retry_detach
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     _try_detach_device(config, persistent=False, host=host)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 467, in _try_detach_device
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     device=alternative_device_name)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     self.force_reraise()
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 451, in _try_detach_device
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     self.detach_device(conf, persistent=persistent, live=live)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 530, in detach_device
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     self._domain.detachDeviceFlags(device_xml, flags=flags)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     rv = execute(f, *args, **kwargs)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     six.reraise(c, e, tb)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     rv = meth(*args, **kwargs)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1217, in detachDeviceFlags
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall libvirtError: internal error: unable to execute QEMU command 'device_del': Device 'virtio-disk15' not found
  2019-02-12 07:34:43.728 23425 ERROR oslo.service.loopingcall

  
  Based on discussion with Melanie Witt, it seems likely that nova is missing a special-case in Guest.detach_device_with_retry().  It seems likely we need to modify the conditional at line 409 of virt/libvirt/guest.py to look like 'if errcode in (libvirt.VIR_ERR_OPERATION_FAILED, libvirt.VIR_ERR_INTERNAL_ERROR):'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815949/+subscriptions