yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1468251] [NEW] network_delete on agent causes race

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kevin Benton <1468251@xxxxxxxxxxxxxxxxxx>
Date: Wed, 24 Jun 2015 09:14:59 -0000
Reply-to: Bug 1468251 <1468251@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

The OVS agent tries to reclaim a local VLAN on a network delete event.
This is racey with any recent port events it has received.

2015-06-24 08:24:46.722 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Delete d27a0308-a37c-496c-8f5f-45b7284075da network_delete /opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:324
2015-06-24 08:24:46.722 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Reclaiming vlan = 15 from net-id = d27a0308-a37c-496c-8f5f-45b7284075da
2015-06-24 08:24:46.722 DEBUG neutron.agent.linux.utils [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', 'br-tun', '-'] execute_rootwrap_daemon /opt/stack/old/neutron/neutron/agent/linux/utils.py:100
2015-06-24 08:24:46.818 DEBUG neutron.agent.linux.utils [req-bd6fd648-0eb7-4ebd-9a99-6e120ccef3b0 None None] 
Command: ['ovs-ofctl', 'add-flows', 'br-tun', '-']
Exit code: 0
Stdin: hard_timeout=0,idle_timeout=0,priority=1,table=4,tun_id=1016,actions=mod_vlan_vid:15,resubmit(,10)
Stdout:
Stderr:  execute /opt/stack/old/neutron/neutron/agent/linux/utils.py:134
2015-06-24 08:24:46.818 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-bd6fd648-0eb7-4ebd-9a99-6e120ccef3b0 None None] Error while processing VIF ports
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1605, in rpc_loop
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1379, in process_network_ports
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     devices_added_updated, ovs_restarted)
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1272, in treat_devices_added_or_updated
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1177, in treat_vif_port
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     fixed_ips, device_owner, ovs_restarted)
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 711, in port_bound
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     lvm = self.local_vlan_map[net_uuid]
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent KeyError: u'd27a0308-a37c-496c-8f5f-45b7284075da'
2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent

** Affects: neutron
     Importance: Undecided
     Assignee: Kevin Benton (kevinbenton)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Kevin Benton (kevinbenton)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1468251

Title:
  network_delete on agent causes race

Status in OpenStack Neutron (virtual network service):
  In Progress

Bug description:
  The OVS agent tries to reclaim a local VLAN on a network delete event.
  This is racey with any recent port events it has received.

  2015-06-24 08:24:46.722 DEBUG neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Delete d27a0308-a37c-496c-8f5f-45b7284075da network_delete /opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py:324
  2015-06-24 08:24:46.722 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Reclaiming vlan = 15 from net-id = d27a0308-a37c-496c-8f5f-45b7284075da
  2015-06-24 08:24:46.722 DEBUG neutron.agent.linux.utils [req-608504e3-aba8-4dab-93da-9c2ef35f8586 NetworksIpV6TestAttrs-1441763649 NetworksIpV6TestAttrs-1909199491] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', 'br-tun', '-'] execute_rootwrap_daemon /opt/stack/old/neutron/neutron/agent/linux/utils.py:100
  2015-06-24 08:24:46.818 DEBUG neutron.agent.linux.utils [req-bd6fd648-0eb7-4ebd-9a99-6e120ccef3b0 None None] 
  Command: ['ovs-ofctl', 'add-flows', 'br-tun', '-']
  Exit code: 0
  Stdin: hard_timeout=0,idle_timeout=0,priority=1,table=4,tun_id=1016,actions=mod_vlan_vid:15,resubmit(,10)
  Stdout:
  Stderr:  execute /opt/stack/old/neutron/neutron/agent/linux/utils.py:134
  2015-06-24 08:24:46.818 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-bd6fd648-0eb7-4ebd-9a99-6e120ccef3b0 None None] Error while processing VIF ports
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1605, in rpc_loop
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1379, in process_network_ports
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     devices_added_updated, ovs_restarted)
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1272, in treat_devices_added_or_updated
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     ovs_restarted)
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1177, in treat_vif_port
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     fixed_ips, device_owner, ovs_restarted)
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/opt/stack/old/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 711, in port_bound
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     lvm = self.local_vlan_map[net_uuid]
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent KeyError: u'd27a0308-a37c-496c-8f5f-45b7284075da'
  2015-06-24 08:24:46.818 9657 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1468251/+subscriptions
Follow ups

[Bug 1468251] [NEW] network_delete on agent causes race
From: Kevin Benton, 2015-06-24
References

[Bug 1468251] [NEW] network_delete on agent causes race
From: Kevin Benton, 2015-06-24