yahoo-eng-team team mailing list archive

[Jakub Libosvar <libosvar@xxxxxxxxxx>]

Public bug reported:

There is a race condition in ovs-agent if l2pop is not used.

If ovs agent receives tunnel_update rpc message and starts handling it,
it searches in local vlan map and calls ovs-ofctl leading to yield in
eventlet. If in another coroutine ovs agent creates a new
LocalVLANMapping due to creation of new port for new tenant, local vlan
map changes and causes exception, after eventlet starts handling
coroutine of tunnel_update.

    2015-04-09 17:41:47.771 6067 ERROR oslo.messaging.rpc.dispatcher [req-1c5353f1-38bd-49f6-8ad1-31ea14484156 ] Exception during message handling: dictionary changed size during iteration
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 347, in tunnel_update
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     tunnel_type)
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1055, in _setup_tunnel_port
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     for network_id, vlan_mapping in self.local_vlan_map.iteritems():
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher RuntimeError: dictionary changed size during iteration
    2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher

** Affects: neutron
     Importance: Undecided
     Assignee: Jakub Libosvar (libosvar)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Jakub Libosvar (libosvar)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1449944

Title:
  Tunnel creation race in ovs-agent, if tunnel_update is called from ml2
  plugin

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  There is a race condition in ovs-agent if l2pop is not used.

  If ovs agent receives tunnel_update rpc message and starts handling
  it, it searches in local vlan map and calls ovs-ofctl leading to yield
  in eventlet. If in another coroutine ovs agent creates a new
  LocalVLANMapping due to creation of new port for new tenant, local
  vlan map changes and causes exception, after eventlet starts handling
  coroutine of tunnel_update.

      2015-04-09 17:41:47.771 6067 ERROR oslo.messaging.rpc.dispatcher [req-1c5353f1-38bd-49f6-8ad1-31ea14484156 ] Exception during message handling: dictionary changed size during iteration
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 347, in tunnel_update
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     tunnel_type)
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1055, in _setup_tunnel_port
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher     for network_id, vlan_mapping in self.local_vlan_map.iteritems():
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher RuntimeError: dictionary changed size during iteration
      2015-04-09 17:41:47.771 6067 TRACE oslo.messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1449944/+subscriptions