yahoo-eng-team team mailing list archive

["Collin M." <ziggit@xxxxxxxxx>]

Public bug reported:

The neutron linux bridge ml2 plugin fails to clean up brq bridges on
nodes hosting dhcp agents.

Environment:
Mirantis OpenStack 8 / Liberty
Linux Bridge ml2 plugin
vxlan segmentation

The workload on the environment involves large numbers of network and
subnet creation and deletion over very short periods of time.

It appears that the issue arises from the network getting marked as
deleted before the cleanup has an opportunity to take place:

An example log section:

2017-06-15 00:19:11.594 449231 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] network_delete received network_delete /usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:770
2017-06-15 00:19:11.595 449231 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] Network f834212c-1e11-4bc1-b3ee-89a61b5ee6ed is not available.


A collection of example bridges that failed to get cleaned up:
brqfed1d4bd-de 8000.92bb5a5feba5 no vxlan-77488 
brqfeebf10d-df 8000.12e256acb4d7 no vxlan-72765 
brqff281599-d2 8000.f26313c6ba80 no vxlan-66828 
brqff2dca83-db 8000.c6f2263ea94d no vxlan-71111 
brqff40fcfa-8e 8000.3621679c4a97 no vxlan-75989 
brqff96bdd2-5c 8000.4efaf9b1ce1f no vxlan-69185 
brqffe6412f-20 8000.ca59d8a1a1aa no vxlan-66860 
brqffea1c9e-82 8000.8a917769c6e4 no vxlan-67359

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  The neutron linux bridge ml2 plugin fails to clean up brq bridges on
  nodes hosting dhcp agents.
  
  Environment:
  Mirantis OpenStack 8 / Liberty
  Linux Bridge ml2 plugin
  vxlan segmentation
  
  The workload on the environment involves large numbers of network and
  subnet creation and deletion over very short periods of time.
  
  It appears that the issue arises from the network getting marked as
  deleted before the cleanup has an opportunity to take place:
  
+ An example log section:
+ 
  2017-06-15 00:19:11.594 449231 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] network_delete received network_delete /usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:770
  2017-06-15 00:19:11.595 449231 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] Network f834212c-1e11-4bc1-b3ee-89a61b5ee6ed is not available.
+ 
+ 
+ A collection of example bridges that failed to get cleaned up:
+ brqfed1d4bd-de 8000.92bb5a5feba5 no vxlan-77488 
+ brqfeebf10d-df 8000.12e256acb4d7 no vxlan-72765 
+ brqff281599-d2 8000.f26313c6ba80 no vxlan-66828 
+ brqff2dca83-db 8000.c6f2263ea94d no vxlan-71111 
+ brqff40fcfa-8e 8000.3621679c4a97 no vxlan-75989 
+ brqff96bdd2-5c 8000.4efaf9b1ce1f no vxlan-69185 
+ brqffe6412f-20 8000.ca59d8a1a1aa no vxlan-66860 
+ brqffea1c9e-82 8000.8a917769c6e4 no vxlan-67359

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1698271

Title:
  Neutron ml2 linux bridge agent fails to clean up bridges on high
  volumes of deletes

Status in neutron:
  New

Bug description:
  The neutron linux bridge ml2 plugin fails to clean up brq bridges on
  nodes hosting dhcp agents.

  Environment:
  Mirantis OpenStack 8 / Liberty
  Linux Bridge ml2 plugin
  vxlan segmentation

  The workload on the environment involves large numbers of network and
  subnet creation and deletion over very short periods of time.

  It appears that the issue arises from the network getting marked as
  deleted before the cleanup has an opportunity to take place:

  An example log section:

  2017-06-15 00:19:11.594 449231 DEBUG neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] network_delete received network_delete /usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py:770
  2017-06-15 00:19:11.595 449231 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [req-d682e11a-f441-4417-b584-07ecc6f7737a ebfe3ac655d0babdd553ea5f8e1ef87bc94dece509a5bc26cd7d26fc58532cfc a5c292ffb8cf484ab74d7954a94cfad1 - - -] Network f834212c-1e11-4bc1-b3ee-89a61b5ee6ed is not available.

  
  A collection of example bridges that failed to get cleaned up:
  brqfed1d4bd-de 8000.92bb5a5feba5 no vxlan-77488 
  brqfeebf10d-df 8000.12e256acb4d7 no vxlan-72765 
  brqff281599-d2 8000.f26313c6ba80 no vxlan-66828 
  brqff2dca83-db 8000.c6f2263ea94d no vxlan-71111 
  brqff40fcfa-8e 8000.3621679c4a97 no vxlan-75989 
  brqff96bdd2-5c 8000.4efaf9b1ce1f no vxlan-69185 
  brqffe6412f-20 8000.ca59d8a1a1aa no vxlan-66860 
  brqffea1c9e-82 8000.8a917769c6e4 no vxlan-67359

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1698271/+subscriptions