yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1449048] [NEW] Cannot find device "vxlan-nnnn" when deleting VxLAN based networks

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Richard Winters <riwinter@xxxxxxxxx>
Date: Mon, 27 Apr 2015 13:52:24 -0000
Reply-to: Bug 1449048 <1449048@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Running base line mutli-node test shows tracebacks in q-agt.log when
deleting test networks

Test Bed:
   - Kilo based
   - LinuxBridge with VxLAN
   - Mutli node - node1: controller/compute, node2: compute

Test details:
 - Create tenant/router - attach to public network
 - Create two tenant networks.
 - Create VM on each network on each compute node - i.e. 2 nets with 2 VMs each
 - Verify traffic to all VMs, VM to VM
 - Delete all VMs, networks, routers  
 - Check logs for Tracebacks.

Even though the traceback shows - the vxlan-1006 interface still gets
deleted.  Another traceback shown below also shows a similar problem
when trying to delete a bridge.


2015-04-27 06:30:21.012 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting vxlan interface vxlan-1006 for vlan from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_a
gent.py:500
2015-04-27 06:30:21.012 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1006', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2015-04-27 06:30:21.021 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
Exit code: 0
Stdin: 
Stdout: 
Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:30:21.023 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'vxlan-1006'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2015-04-27 06:30:21.025 DEBUG neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] 
Command: ['ip', '-o', 'link', 'show', 'vxlan-1006']
Exit code: 0
Stdin: 
Stdout: 71: vxlan-1006: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default \    link/ether 26:6e:5b:e3:6d:81 brd ff:ff:ff:ff:ff:ff

Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:30:21.026 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Deleting vxlan interface vxlan-1006 for vlan from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridg
e/agent/linuxbridge_neutron_agent.py:500
2015-04-27 06:30:21.026 DEBUG neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1006', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2015-04-27 06:30:21.044 ERROR neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] 
Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
Exit code: 1
Stdin: 
Stdout: 
Stderr: Cannot find device "vxlan-1006"

2015-04-27 06:30:21.044 ERROR oslo_messaging.rpc.dispatcher [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Exception during message handling: 
Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
Exit code: 1
Stdin: 
Stdout: 
Stderr: Cannot find device "vxlan-1006"

2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 653, in network_delete
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self.agent.br_mgr.delete_vlan_bridge(bridge_name)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 427, in delete_vlan_bridge
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self.delete_vxlan(interface)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 502, in delete_vxlan
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     int_vxlan.link.set_down()
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 279, in set_down
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self._as_root([], ('set', self.name, 'down'))
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     use_root_namespace=use_root_namespace)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=self.log_fail_as_error)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=log_fail_as_error)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     raise RuntimeError(m)
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher RuntimeError: 
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Exit code: 1
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stdin: 
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stdout: 
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stderr: Cannot find device "vxlan-1006"
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher 
2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher 
2015-04-27 06:30:21.060 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
Command: ['ip', 'link', 'delete', u'vxlan-1006']
Exit code: 0
Stdin: 
Stdout: 
Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:30:21.060 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Done deleting vxlan interface vxlan-1006 from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:504
2015-04-27 06:30:21.061 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting bridge brq239ff965-41 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:443
2015-04-27 06:30:21.061 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'brq239ff965-41', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2015-04-27 06:30:21.066 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 


=======================================================
2nd run of same test show similar issue just complians about can't find bridge. The bridge does get deleted


Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:40:45.032 DEBUG neutron.agent.linux.utils [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] Running command (rootwrap daemon): ['brctl', 'delbr', 'brq84bbf230-51'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/ut
ils.py:100
2015-04-27 06:40:45.057 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
Command: ['ip', 'link', 'delete', u'vxlan-1006']
Exit code: 0
Stdin: 
Stdout: 
Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:40:45.059 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Done deleting vxlan interface vxlan-1006 from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent
.py:504
2015-04-27 06:40:45.059 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting bridge brq84bbf230-51 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:
443
2015-04-27 06:40:45.059 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'brq84bbf230-51', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
2015-04-27 06:40:45.068 ERROR neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
Command: ['ip', 'link', 'set', u'brq84bbf230-51', 'down']
Exit code: 1
Stdin: 
Stdout: 
Stderr: Cannot find device "brq84bbf230-51"

2015-04-27 06:40:45.068 ERROR neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Error in agent loop. Devices info: {'current': set(['tapec8cb37a-0f', 'tap605d7373-35', 'tapb21755ac-f7', 'tap450252a8-4a', 'tap6ccb9041-3c', 'tap5a8
ace63-a6']), 'removed': set(['tap8477fb1f-65', 'tap68e9bd5e-b4', 'tap2f05553b-08', 'tap0e8ae5ae-fd', 'tap9be3a1a9-75']), 'added': set([]), 'updated': set([])}
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 984, in daemon_loop
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     sync = self.process_network_devices(device_info)
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 843, in process_network_devices
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     resync_b = self.treat_devices_removed(device_info['removed'])
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 918, in treat_devices_removed
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     self.br_mgr.remove_empty_bridges()
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 460, in remove_empty_bridges
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     self.delete_vlan_bridge(bridge_name)
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 445, in delete_vlan_bridge
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     run_as_root=True):
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     raise RuntimeError(m)
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError: 
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Command: ['ip', 'link', 'set', u'brq84bbf230-51', 'down']
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Exit code: 1
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdin: 
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdout: 
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stderr: Cannot find device "brq84bbf230-51"
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent 
2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent 
2015-04-27 06:40:45.084 DEBUG neutron.agent.linux.utils [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] 
Command: ['brctl', 'delbr', u'brq84bbf230-51']
Exit code: 0
Stdin: 
Stdout: 
Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
2015-04-27 06:40:45.084 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] Done deleting bridge brq84bbf230-51 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:450
2015-04-27 06:40:46.563 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Agent out of sync with plugin!
2015-04-27 06:40:46.563 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Agent loop found changes! {'current': set(['tap450252a8-4a', 'tap605d7373-35', 'tapec8cb37a-0f']), 'removed': set(['tap2f05553b-08', 'tap8477fb1f-65', 'tap68e9bd5e-b4', 'tapb21755ac-f7', 'tap0e8ae5ae-fd', 'tap9be3a1a9-75', 'tap6ccb9041-3c', 'tap5a8ace63-a6']), 'added': set(['tap450252a8-4a', 'tap605d7373-35', 'tapec8cb37a-0f']), 'updated': set([])} from (pid=16503) daemon_loop /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:982
2015-04-27 06:40:46.564 INFO neutron.agent.securitygroups_rpc [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Preparing filters for devices set(['tap450252a8-4a', 'tap605d7373-35', 'tape

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1449048

Title:
  Cannot find device "vxlan-nnnn" when deleting VxLAN based networks

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Running base line mutli-node test shows tracebacks in q-agt.log when
  deleting test networks

  Test Bed:
     - Kilo based
     - LinuxBridge with VxLAN
     - Mutli node - node1: controller/compute, node2: compute

  Test details:
   - Create tenant/router - attach to public network
   - Create two tenant networks.
   - Create VM on each network on each compute node - i.e. 2 nets with 2 VMs each
   - Verify traffic to all VMs, VM to VM
   - Delete all VMs, networks, routers  
   - Check logs for Tracebacks.

  Even though the traceback shows - the vxlan-1006 interface still gets
  deleted.  Another traceback shown below also shows a similar problem
  when trying to delete a bridge.


  2015-04-27 06:30:21.012 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting vxlan interface vxlan-1006 for vlan from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_a
  gent.py:500
  2015-04-27 06:30:21.012 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1006', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2015-04-27 06:30:21.021 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
  Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
  Exit code: 0
  Stdin: 
  Stdout: 
  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:30:21.023 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'vxlan-1006'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2015-04-27 06:30:21.025 DEBUG neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] 
  Command: ['ip', '-o', 'link', 'show', 'vxlan-1006']
  Exit code: 0
  Stdin: 
  Stdout: 71: vxlan-1006: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default \    link/ether 26:6e:5b:e3:6d:81 brd ff:ff:ff:ff:ff:ff

  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:30:21.026 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Deleting vxlan interface vxlan-1006 for vlan from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridg
  e/agent/linuxbridge_neutron_agent.py:500
  2015-04-27 06:30:21.026 DEBUG neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Running command (rootwrap daemon): ['ip', 'link', 'set', 'vxlan-1006', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2015-04-27 06:30:21.044 ERROR neutron.agent.linux.utils [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] 
  Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: Cannot find device "vxlan-1006"

  2015-04-27 06:30:21.044 ERROR oslo_messaging.rpc.dispatcher [req-cf0ecf5c-b75b-4891-8962-303e9f8c3c96 admin c28923a14e0d4ee9badd971171e3567c] Exception during message handling: 
  Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: Cannot find device "vxlan-1006"

  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 653, in network_delete
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self.agent.br_mgr.delete_vlan_bridge(bridge_name)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 427, in delete_vlan_bridge
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self.delete_vxlan(interface)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 502, in delete_vxlan
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     int_vxlan.link.set_down()
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 279, in set_down
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     self._as_root([], ('set', self.name, 'down'))
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     use_root_namespace=use_root_namespace)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=self.log_fail_as_error)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     log_fail_as_error=log_fail_as_error)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher     raise RuntimeError(m)
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher RuntimeError: 
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Command: ['ip', 'link', 'set', u'vxlan-1006', 'down']
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Exit code: 1
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stdin: 
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stdout: 
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher Stderr: Cannot find device "vxlan-1006"
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher 
  2015-04-27 06:30:21.044 TRACE oslo_messaging.rpc.dispatcher 
  2015-04-27 06:30:21.060 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
  Command: ['ip', 'link', 'delete', u'vxlan-1006']
  Exit code: 0
  Stdin: 
  Stdout: 
  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:30:21.060 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Done deleting vxlan interface vxlan-1006 from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:504
  2015-04-27 06:30:21.061 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting bridge brq239ff965-41 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:443
  2015-04-27 06:30:21.061 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'brq239ff965-41', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2015-04-27 06:30:21.066 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 


  
  =======================================================
  2nd run of same test show similar issue just complians about can't find bridge. The bridge does get deleted


  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:40:45.032 DEBUG neutron.agent.linux.utils [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] Running command (rootwrap daemon): ['brctl', 'delbr', 'brq84bbf230-51'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/ut
  ils.py:100
  2015-04-27 06:40:45.057 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
  Command: ['ip', 'link', 'delete', u'vxlan-1006']
  Exit code: 0
  Stdin: 
  Stdout: 
  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:40:45.059 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Done deleting vxlan interface vxlan-1006 from (pid=16503) delete_vxlan /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent
  .py:504
  2015-04-27 06:40:45.059 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Deleting bridge brq84bbf230-51 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:
  443
  2015-04-27 06:40:45.059 DEBUG neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Running command (rootwrap daemon): ['ip', 'link', 'set', 'brq84bbf230-51', 'down'] from (pid=16503) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:100
  2015-04-27 06:40:45.068 ERROR neutron.agent.linux.utils [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] 
  Command: ['ip', 'link', 'set', u'brq84bbf230-51', 'down']
  Exit code: 1
  Stdin: 
  Stdout: 
  Stderr: Cannot find device "brq84bbf230-51"

  2015-04-27 06:40:45.068 ERROR neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Error in agent loop. Devices info: {'current': set(['tapec8cb37a-0f', 'tap605d7373-35', 'tapb21755ac-f7', 'tap450252a8-4a', 'tap6ccb9041-3c', 'tap5a8
  ace63-a6']), 'removed': set(['tap8477fb1f-65', 'tap68e9bd5e-b4', 'tap2f05553b-08', 'tap0e8ae5ae-fd', 'tap9be3a1a9-75']), 'added': set([]), 'updated': set([])}
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Traceback (most recent call last):
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 984, in daemon_loop
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     sync = self.process_network_devices(device_info)
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 843, in process_network_devices
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     resync_b = self.treat_devices_removed(device_info['removed'])
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 918, in treat_devices_removed
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     self.br_mgr.remove_empty_bridges()
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 460, in remove_empty_bridges
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     self.delete_vlan_bridge(bridge_name)
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 445, in delete_vlan_bridge
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     run_as_root=True):
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent   File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 137, in execute
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent     raise RuntimeError(m)
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent RuntimeError: 
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Command: ['ip', 'link', 'set', u'brq84bbf230-51', 'down']
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Exit code: 1
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdin: 
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stdout: 
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent Stderr: Cannot find device "brq84bbf230-51"
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent 
  2015-04-27 06:40:45.068 TRACE neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent 
  2015-04-27 06:40:45.084 DEBUG neutron.agent.linux.utils [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] 
  Command: ['brctl', 'delbr', u'brq84bbf230-51']
  Exit code: 0
  Stdin: 
  Stdout: 
  Stderr:  from (pid=16503) execute /opt/stack/neutron/neutron/agent/linux/utils.py:134
  2015-04-27 06:40:45.084 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-7ec93e55-f857-44c5-a692-78b7423aee3e admin c28923a14e0d4ee9badd971171e3567c] Done deleting bridge brq84bbf230-51 from (pid=16503) delete_vlan_bridge /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:450
  2015-04-27 06:40:46.563 INFO neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Agent out of sync with plugin!
  2015-04-27 06:40:46.563 DEBUG neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Agent loop found changes! {'current': set(['tap450252a8-4a', 'tap605d7373-35', 'tapec8cb37a-0f']), 'removed': set(['tap2f05553b-08', 'tap8477fb1f-65', 'tap68e9bd5e-b4', 'tapb21755ac-f7', 'tap0e8ae5ae-fd', 'tap9be3a1a9-75', 'tap6ccb9041-3c', 'tap5a8ace63-a6']), 'added': set(['tap450252a8-4a', 'tap605d7373-35', 'tapec8cb37a-0f']), 'updated': set([])} from (pid=16503) daemon_loop /opt/stack/neutron/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py:982
  2015-04-27 06:40:46.564 INFO neutron.agent.securitygroups_rpc [req-0cf41562-c9f9-4153-a5e8-2800838dc2f6 None None] Preparing filters for devices set(['tap450252a8-4a', 'tap605d7373-35', 'tape

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1449048/+subscriptions
Follow ups

[Bug 1449048] [NEW] Cannot find device "vxlan-nnnn" when deleting VxLAN based networks
From: Richard Winters, 2015-04-27
References

[Bug 1449048] [NEW] Cannot find device "vxlan-nnnn" when deleting VxLAN based networks
From: Richard Winters, 2015-04-27