yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1696125] Re: Detach interface failed - Unable to detach from guest transient domain (pike)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Tue, 13 Jun 2017 20:37:13 -0000
Reply-to: Bug 1696125 <1696125@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
There is a weird 5 minute gap in the neutron L2 agent logs here:

http://logs.openstack.org/94/471494/2/check/gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-
xenial/e0332b9/logs/screen-q-agt.txt.gz#_Jun_12_22_24_33_893133

Jun 12 22:19:32.086692 ubuntu-xenial-internap-mtl01-9270284 neutron-linuxbridge-agent[18735]: DEBUG oslo_concurrency.lockutils [None req-2b21118e-0da6-4c4a-a9d5-5d0e2cf4ccf8 None None] Lock "ebtables" released by "neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect.delete_arp_spoofing_protection" :: held 0.033s {{(pid=18735) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}}
Jun 12 22:24:33.893133 ubuntu-xenial-internap-mtl01-9270284 neutron-linuxbridge-agent[18735]: DEBUG neutron.plugins.ml2.drivers.agent._common_agent [None req-2b21118e-0da6-4c4a-a9d5-5d0e2cf4ccf8 None None] Agent loop found changes! {'current': set(['tap3578cb80-7a', 'tap8b6dc92f-75', 'tapb7c1b8db-1e', 'tap999bcb19-81', 'tap839063fe-6d', 'tap3a6b0b5c-cb', 'tap14bcf3bb-af', 'tap0f5cf6b7-63']), 'timestamps': {'tap3578cb80-7a': 9, 'tap8b6dc92f-75': 6, 'tapb7c1b8db-1e': 10, 'tap999bcb19-81': 478, 'tap839063fe-6d': 484, 'tap3a6b0b5c-cb': 480, 'tap14bcf3bb-af': 479, 'tap0f5cf6b7-63': 11}, 'removed': set(['tape6c02d6c-e4']), 'added': set([]), 'updated': set([])} {{(pid=18735) daemon_loop /opt/stack/new/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py:452}}

** Also affects: neutron
   Importance: Undecided
       Status: New

** Summary changed:

- Detach interface failed - Unable to detach from guest transient domain (pike)
+ Detach interface failed - timeout waiting to detach tap device in linuxbridge job (pike)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1696125

Title:
  Detach interface failed - timeout waiting to detach tap device in
  linuxbridge job (pike)

Status in neutron:
  New
Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seeing this in Tempest runs on master (pike):

  http://logs.openstack.org/24/471024/2/check/gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-
  xenial/6b98d38/logs/screen-n-cpu.txt.gz?level=TRACE#_Jun_06_02_16_02_855503

  Jun 06 02:16:02.855503 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: WARNING nova.compute.manager [None req-b4a50024-a2fd-4279-b284-340d2074f1c1 tempest-TestNetworkBasicOps-1479445685 tempest-TestNetworkBasicOps-1479445685] [instance: 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8] Detach interface failed, port_id=3843caa3-ab04-45f1-94d8-f330390e40fe, reason: Device detach failed for fa:16:3e:ab:e3:3f: Unable to detach from guest transient domain.: DeviceDetachFailed: Device detach failed for fa:16:3e:ab:e3:3f: Unable to detach from guest transient domain.
  Jun 06 02:16:02.884007 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server [None req-b4a50024-a2fd-4279-b284-340d2074f1c1 tempest-TestNetworkBasicOps-1479445685 tempest-TestNetworkBasicOps-1479445685] Exception during message handling: InterfaceDetachFailed: Failed to detach network adapter device from 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8
  Jun 06 02:16:02.884180 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  Jun 06 02:16:02.884286 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 157, in _process_incoming
  Jun 06 02:16:02.884395 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  Jun 06 02:16:02.884538 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 213, in dispatch
  Jun 06 02:16:02.884669 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  Jun 06 02:16:02.884777 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _do_dispatch
  Jun 06 02:16:02.884869 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  Jun 06 02:16:02.884968 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 77, in wrapped
  Jun 06 02:16:02.885069 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  Jun 06 02:16:02.885171 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 06 02:16:02.885272 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 06 02:16:02.885367 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 06 02:16:02.885461 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 06 02:16:02.885554 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/exception_wrapper.py", line 68, in wrapped
  Jun 06 02:16:02.885649 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  Jun 06 02:16:02.885755 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 214, in decorated_function
  Jun 06 02:16:02.885856 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  Jun 06 02:16:02.885950 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 06 02:16:02.886053 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     self.force_reraise()
  Jun 06 02:16:02.886143 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 06 02:16:02.886232 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  Jun 06 02:16:02.886322 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 202, in decorated_function
  Jun 06 02:16:02.886415 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  Jun 06 02:16:02.886505 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server   File "/opt/stack/new/nova/nova/compute/manager.py", line 5224, in detach_interface
  Jun 06 02:16:02.886601 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server     raise exception.InterfaceDetachFailed(instance_uuid=instance.uuid)
  Jun 06 02:16:02.886695 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server InterfaceDetachFailed: Failed to detach network adapter device from 2668bcb9-b13d-4b5b-8ee5-edbdee3b15a8
  Jun 06 02:16:02.886794 ubuntu-xenial-ovh-bhs1-9149075 nova-compute[24118]: ERROR oslo_messaging.rpc.server 

  This probably started with: https://review.openstack.org/#/c/349014/
  (merged 5/31)

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Detach%20interface%20failed%5C%22%20AND%20message:%5C%22reason:%20Device%20detach%20failed%20for%5C%22%20AND%20message:%5C%22Unable%20to%20detach%20from%20guest%20transient%20domain%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22&from=10d

  Might need something like this:
  https://review.openstack.org/#/c/441204/

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1696125/+subscriptions
References

[Bug 1696125] [NEW] Detach interface failed - Unable to detach from guest transient domain (pike)
From: Matt Riedemann, 2017-06-06