yahoo-eng-team team mailing list archive

[Miguel Lavalle <miguel@xxxxxxxxxxxx>]

Public bug reported:

We are seeing a spike of failures in job gate-tempest-dsvm-neutron-
linuxbridge-ubuntu-xenial, test
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic.

As of this writing, there are 48 instances of this failure in Kibana:

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Failed%20to%20delete%20resource%5C%22%20AND%20message:%5C%22within%20the%20required%20time%20%5C%22%20AND%20message:%5C%22TestNetworkBasicOps:_run_cleanups%5C%22%20AND%20tags:%5C%22console%5C%22&from=7d

I analyzed several cases and all seem to be the same. Compute manager
calls libvirt to detach the interface and gets a failure (in this
example, tap tap05567d09-b3 is being detached), so it raises an
InterfaceDetachFailed exception:

http://paste.openstack.org/show/611347/

The code involved is here:

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5224

Since the exception is raised, the call to Neutron to delete the port is
never executed:
https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5227.
Since the tempest test was setup to wait for the port deletion to
ascertain success of the interface removal
(https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L253)
a timeout occurs and the test / job fails.

It is worth noting that the Linuxbridge agent actually detected the
removal of the interface: http://paste.openstack.org/show/611346/.
Please note that tap05567d09-b3 shows up in the removed set of the
changes detected by the agent

It is also worth noting that this failure starts to show up in Kibana on
5/31/2017

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: compute gate-failure libvirt

** Tags added: compute gate-failure libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1696006

Title:
  Libvirt fails to detach network interface with Linux bridge

Status in OpenStack Compute (nova):
  New

Bug description:
  We are seeing a spike of failures in job gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-xenial, test
  tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic.

  As of this writing, there are 48 instances of this failure in Kibana:

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Failed%20to%20delete%20resource%5C%22%20AND%20message:%5C%22within%20the%20required%20time%20%5C%22%20AND%20message:%5C%22TestNetworkBasicOps:_run_cleanups%5C%22%20AND%20tags:%5C%22console%5C%22&from=7d

  I analyzed several cases and all seem to be the same. Compute manager
  calls libvirt to detach the interface and gets a failure (in this
  example, tap tap05567d09-b3 is being detached), so it raises an
  InterfaceDetachFailed exception:

  http://paste.openstack.org/show/611347/

  The code involved is here:

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5224

  Since the exception is raised, the call to Neutron to delete the port
  is never executed:
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5227.
  Since the tempest test was setup to wait for the port deletion to
  ascertain success of the interface removal
  (https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L253)
  a timeout occurs and the test / job fails.

  It is worth noting that the Linuxbridge agent actually detected the
  removal of the interface: http://paste.openstack.org/show/611346/.
  Please note that tap05567d09-b3 shows up in the removed set of the
  changes detected by the agent

  It is also worth noting that this failure starts to show up in Kibana
  on 5/31/2017

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1696006/+subscriptions