yahoo-eng-team team mailing list archive

[Miguel Lavalle <miguel@xxxxxxxxxxxx>]

** Also affects: neutron
   Importance: Undecided
       Status: New

** Changed in: neutron
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1696006

Title:
  Libvirt fails to detach network interface with Linux bridge

Status in neutron:
  New
Status in OpenStack Compute (nova):
  New

Bug description:
  We are seeing a spike of failures in job gate-tempest-dsvm-neutron-
  linuxbridge-ubuntu-xenial, test
  tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic.

  As of this writing, there are 48 instances of this failure in Kibana:

  http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Failed%20to%20delete%20resource%5C%22%20AND%20message:%5C%22within%20the%20required%20time%20%5C%22%20AND%20message:%5C%22TestNetworkBasicOps:_run_cleanups%5C%22%20AND%20tags:%5C%22console%5C%22&from=7d

  I analyzed several cases and all seem to be the same. Compute manager
  calls libvirt to detach the interface and gets a failure (in this
  example, tap tap05567d09-b3 is being detached), so it raises an
  InterfaceDetachFailed exception:

  http://paste.openstack.org/show/611347/

  The code involved is here:

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5224

  Since the exception is raised, the call to Neutron to delete the port
  is never executed:
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5227.
  Since the tempest test was setup to wait for the port deletion to
  ascertain success of the interface removal
  (https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_basic_ops.py#L253)
  a timeout occurs and the test / job fails.

  It is worth noting that the Linuxbridge agent actually detected the
  removal of the interface: http://paste.openstack.org/show/611346/.
  Please note that tap05567d09-b3 shows up in the removed set of the
  changes detected by the agent

  It is also worth noting that this failure starts to show up in Kibana
  on 5/31/2017

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1696006/+subscriptions