yahoo-eng-team team mailing list archive

[OpenStack Infra <1631513@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/385617
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d40322c7d4aa1dd6d595dfe415278c9f252f4da2
Submitter: Jenkins
Branch:    master

commit d40322c7d4aa1dd6d595dfe415278c9f252f4da2
Author: Swaminathan Vasudevan <swaminathan.vasudevan@xxxxxxx>
Date:   Fri Oct 7 10:30:40 2016 -0700

    DVR: Fix race condition in creation of fip gateway
    
    In large-scale environments, we have seen a router update
    arrive for one tenant while we are still creating the
    router for a different tenant and initializing the shared
    floating IP gateway port.  Sometimes these updates can
    get scheduled simultaneously, with the second running
    before we are done creating all the resources in the
    first, causing an exception when trying to set the
    default route since either the interface or IP address
    does not exist yet.
    
    Add a lock to better synchronize these functions so
    a create can finish before an update can be done.
    
    If it still fails, we will throw an exception so that
    the namespace will be cleaned-up and the update can be
    re-scheduled for the next iteration.
    
    Closes-Bug: #1631513
    Change-Id: Ia8c92cea2f8798582c39ad3450ab3b3c45a356f7


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1631513

Title:
  DVR: Fix race conditions when trying to add default gateway for fip
  gateway port.

Status in neutron:
  Fix Released

Bug description:
  There seems to be a race condition when trying to add default gateway
  route in fip namespace for the fip agent gateway port.

  The way it happens is at high scale testing, when there is a router
  update that is currently happening for the Router-A which has a
  floatingip, a fip namespace is getting created and gateway ports
  plugged to the external bridge in the context of the fip namespace.
  While it is getting created, if there is another router update for the
  same Router-A, then it calls 'update-gateway-port' and tries to set
  the default gateway and fails.

  We do find a log message in the l3-agent with  'Failed to process compatible router' and also a TRACE in the l3-agent.
  Traceback (most recent call last):
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
       self._process_router_if_compatible(router)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 440, in _process_router_if_compatible
       self._process_updated_router(router)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 454, in _process_updated_router
       ri.process(self)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 538, in process
       super(DvrLocalRouter, self).process(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process
       super(DvrRouterBase, self).process(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 396, in call
       self.logger(e)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
       self.force_reraise()
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
       six.reraise(self.type_, self.value, self.tb)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 393, in call
       return func(*args, **kwargs)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 989, in process
       self.process_external(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 491, in process_external
       self.create_dvr_fip_interfaces(ex_gw_port)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 522, in create_dvr_fip_interfaces
       self.fip_ns.update_gateway_port(fip_agent_port)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 243, in update_gateway_port
       ipd.route.add_gateway(gw_ip)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 690, in add_gateway
       self._as_root([ip_version], tuple(args))
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
       use_root_namespace=use_root_namespace)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
       log_fail_as_error=self.log_fail_as_error)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
       log_fail_as_error=log_fail_as_error)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
       raise RuntimeError(msg)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1631513/+subscriptions