yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1585858] Re: InterfaceDetachFailed: resources.server: Failed to detach interface

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rabi Mishra <1585858@xxxxxxxxxxxxxxxxxx>
Date: Tue, 06 Jun 2017 06:08:33 -0000
Reply-to: Bug 1585858 <1585858@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is happening very frequently now. Every time with a different
libvirt error.

Like the one below:

http://logs.openstack.org/93/462293/3/gate/gate-heat-dsvm-functional-
convg-mysql-lbaasv2-non-apache-ubuntu-
xenial/ea8c446/logs/screen-n-cpu.txt.gz?level=ERROR


Jun 06 05:25:27.340500 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed: libvirtError: internal error: End of file from qemu monitor
Jun 06 05:25:27.340614 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall Traceback (most recent call last):
Jun 06 05:25:27.340696 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 137, in _run_loop
Jun 06 05:25:27.340773 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
Jun 06 05:25:27.340850 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 394, in _func
Jun 06 05:25:27.340927 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     result = f(*args, **kwargs)
Jun 06 05:25:27.341001 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 446, in _do_wait_and_retry_detach
Jun 06 05:25:27.341091 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     _try_detach_device(config, persistent=False, live=live)
Jun 06 05:25:27.341172 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 427, in _try_detach_device
Jun 06 05:25:27.341257 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     device=alternative_device_name)
Jun 06 05:25:27.341347 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 06 05:25:27.341424 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     self.force_reraise()
Jun 06 05:25:27.341509 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 06 05:25:27.341602 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     six.reraise(self.type_, self.value, self.tb)
Jun 06 05:25:27.341688 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 407, in _try_detach_device
Jun 06 05:25:27.341766 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     self.detach_device(conf, persistent=persistent, live=live)
Jun 06 05:25:27.341843 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 471, in detach_device
Jun 06 05:25:27.341930 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     self._domain.detachDeviceFlags(device_xml, flags=flags)
Jun 06 05:25:27.342391 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
Jun 06 05:25:27.342483 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jun 06 05:25:27.342570 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
Jun 06 05:25:27.342661 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     rv = execute(f, *args, **kwargs)
Jun 06 05:25:27.342738 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
Jun 06 05:25:27.342814 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     six.reraise(c, e, tb)
Jun 06 05:25:27.342897 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Jun 06 05:25:27.342985 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     rv = meth(*args, **kwargs)
Jun 06 05:25:27.343068 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1215, in detachDeviceFlags
Jun 06 05:25:27.343149 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall     if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
Jun 06 05:25:27.343224 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall libvirtError: internal error: End of file from qemu monitor
Jun 06 05:25:27.343298 ubuntu-xenial-osic-cloud1-s3500-9151671 nova-compute[4359]: ERROR oslo.service.loopingcall 


** Changed in: heat
   Importance: Undecided => High

** Also affects: nova
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1585858

Title:
  InterfaceDetachFailed: resources.server: Failed to detach interface

Status in heat:
  Triaged
Status in OpenStack Compute (nova):
  New

Bug description:
  It seems the number of attempts to detach interfaces in not enough in
  some cases.

  heat_integrationtests.functional.test_create_update.UpdateStackTest.test_stack_update_with_replacing_userdata
  2016-05-25 10:37:16.151 | 2016-05-25 10:37:16.020 | -------------------------------------------------------------------------------------------------------------
  2016-05-25 10:37:16.151 | 2016-05-25 10:37:16.022 | 
  2016-05-25 10:37:16.152 | 2016-05-25 10:37:16.025 | Captured traceback:
  2016-05-25 10:37:16.152 | 2016-05-25 10:37:16.031 | ~~~~~~~~~~~~~~~~~~~
  2016-05-25 10:37:16.152 | 2016-05-25 10:37:16.034 |     Traceback (most recent call last):
  2016-05-25 10:37:16.152 | 2016-05-25 10:37:16.038 |       File "/opt/stack/new/heat/heat_integrationtests/functional/test_create_update.py", line 534, in test_stack_update_with_replacing_userdata
  2016-05-25 10:37:16.152 | 2016-05-25 10:37:16.040 |         parameters=parms_updated)
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.044 |       File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 402, in update_stack
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.046 |         self._wait_for_stack_status(**kwargs)
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.048 |       File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 334, in _wait_for_stack_status
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.051 |         fail_regexp):
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.054 |       File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 295, in _verify_status
  2016-05-25 10:37:16.153 | 2016-05-25 10:37:16.056 |         stack_status_reason=stack.stack_status_reason)
  2016-05-25 10:37:16.154 | 2016-05-25 10:37:16.058 |     heat_integrationtests.common.exceptions.StackBuildErrorException: Stack UpdateStackTest-2129656241/3c284793-623d-47bd-901a-6d71e75bdbca is in UPDATE_FAILED status due to 'InterfaceDetachFailed: resources.server: Failed to detach interface (bdbc4a8c-114b-42c0-a49a-a27311f5f2e2) from server (259f3fff-6e38-4722-9a51-7c7166d78812)'

To manage notifications about this bug go to:
https://bugs.launchpad.net/heat/+bug/1585858/+subscriptions