← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #21012

[Bug 1368404] [NEW] Uncaught 'libvirtError: Domain not found' errors during destroy

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Some uncaught libvirt errors may result in instances being set to ERROR
state and is causing sporadic gate failures. This can happen for any of
the code paths that use _destroy().  Here is a recent example of a
failed resize:

[req-06dd4908-382e-455e-854e-e4d42a4bf62b TestServerAdvancedOps-724416891 TestServerAdvancedOps-711228572] [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] Setting instance vm_state to ERROR
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] Traceback (most recent call last):
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 5902, in _error_out_instance_on_exception
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     yield
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3658, in resize_instance
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     timeout, retry_interval)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5468, in migrate_disk_and_power_off
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self.power_off(instance, timeout, retry_interval)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2400, in power_off
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self._destroy(instance)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 998, in _destroy
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     timer.start(interval=0.5).wait()
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     return hubs.get_hub().switch()
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 293, in switch
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     return self.greenlet.switch()
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/openstack/common/loopingcall.py", line 81, in _inner
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self.f(*self.args, **self.kw)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 971, in _wait_for_destroy
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     dom_info = self.get_info(instance)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3922, in get_info
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     dom_info = virt_dom.info()
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     rv = execute(f, *args, **kwargs)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     six.reraise(c, e, tb)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     rv = meth(*args, **kwargs)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1068, in info
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] libvirtError: Domain not found: no domain with matching uuid '525f4f95-f631-4fbb-a884-20c37711fb0d' (instance-00000097)
2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]

** Affects: nova
     Importance: High
     Assignee: Hans Lindgren (hanlind)
         Status: In Progress


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1368404

Title:
  Uncaught 'libvirtError: Domain not found' errors during destroy

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  Some uncaught libvirt errors may result in instances being set to
  ERROR state and is causing sporadic gate failures. This can happen for
  any of the code paths that use _destroy().  Here is a recent example
  of a failed resize:

  [req-06dd4908-382e-455e-854e-e4d42a4bf62b TestServerAdvancedOps-724416891 TestServerAdvancedOps-711228572] [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] Setting instance vm_state to ERROR
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] Traceback (most recent call last):
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 5902, in _error_out_instance_on_exception
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     yield
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 3658, in resize_instance
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     timeout, retry_interval)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5468, in migrate_disk_and_power_off
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self.power_off(instance, timeout, retry_interval)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2400, in power_off
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self._destroy(instance)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 998, in _destroy
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     timer.start(interval=0.5).wait()
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     return hubs.get_hub().switch()
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 293, in switch
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     return self.greenlet.switch()
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/openstack/common/loopingcall.py", line 81, in _inner
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     self.f(*self.args, **self.kw)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 971, in _wait_for_destroy
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     dom_info = self.get_info(instance)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3922, in get_info
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     dom_info = virt_dom.info()
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     rv = execute(f, *args, **kwargs)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     six.reraise(c, e, tb)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     rv = meth(*args, **kwargs)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1068, in info
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]     if ret is None: raise libvirtError ('virDomainGetInfo() failed', dom=self)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d] libvirtError: Domain not found: no domain with matching uuid '525f4f95-f631-4fbb-a884-20c37711fb0d' (instance-00000097)
  2014-09-05 01:08:37.123 26984 TRACE nova.compute.manager [instance: 525f4f95-f631-4fbb-a884-20c37711fb0d]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1368404/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next