yahoo-eng-team team mailing list archive

[Swaminathan Vasudevan <swaminathan.vasudevan@xxxxxx>]

Public bug reported:

There seems to be a race condition when trying to add default gateway
route in fip namespace for the fip agent gateway port.

The way it happens is at high scale testing, when there is a router
update that is currently happening for the Router-A which has a
floatingip, a fip namespace is getting created and gateway ports plugged
to the external bridge in the context of the fip namespace. While it is
getting created, if there is another router update for the same
Router-A, then it calls 'update-gateway-port' and tries to set the
default gateway and fails.

We do find a log message in the l3-agent with  'Failed to process compatible router' and also a TRACE in the l3-agent.
Traceback (most recent call last):
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
     self._process_router_if_compatible(router)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 440, in _process_router_if_compatible
     self._process_updated_router(router)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 454, in _process_updated_router
     ri.process(self)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 538, in process
     super(DvrLocalRouter, self).process(agent)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process
     super(DvrRouterBase, self).process(agent)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 396, in call
     self.logger(e)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
     self.force_reraise()
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
     six.reraise(self.type_, self.value, self.tb)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 393, in call
     return func(*args, **kwargs)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 989, in process
     self.process_external(agent)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 491, in process_external
     self.create_dvr_fip_interfaces(ex_gw_port)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 522, in create_dvr_fip_interfaces
     self.fip_ns.update_gateway_port(fip_agent_port)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 243, in update_gateway_port
     ipd.route.add_gateway(gw_ip)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 690, in add_gateway
     self._as_root([ip_version], tuple(args))
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
     use_root_namespace=use_root_namespace)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
     log_fail_as_error=self.log_fail_as_error)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
     log_fail_as_error=log_fail_as_error)
   File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
     raise RuntimeError(msg)

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3-dvr-backlog mitaka-backport-potential newton-backport-potential

** Summary changed:

- Fix race conditions when trying to add default gateway for fip gateway port.
+ DVR: Fix race conditions when trying to add default gateway for fip gateway port.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1631513

Title:
  DVR: Fix race conditions when trying to add default gateway for fip
  gateway port.

Status in neutron:
  New

Bug description:
  There seems to be a race condition when trying to add default gateway
  route in fip namespace for the fip agent gateway port.

  The way it happens is at high scale testing, when there is a router
  update that is currently happening for the Router-A which has a
  floatingip, a fip namespace is getting created and gateway ports
  plugged to the external bridge in the context of the fip namespace.
  While it is getting created, if there is another router update for the
  same Router-A, then it calls 'update-gateway-port' and tries to set
  the default gateway and fails.

  We do find a log message in the l3-agent with  'Failed to process compatible router' and also a TRACE in the l3-agent.
  Traceback (most recent call last):
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 501, in _process_router_update
       self._process_router_if_compatible(router)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 440, in _process_router_if_compatible
       self._process_updated_router(router)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 454, in _process_updated_router
       ri.process(self)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 538, in process
       super(DvrLocalRouter, self).process(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_router_base.py", line 31, in process
       super(DvrRouterBase, self).process(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 396, in call
       self.logger(e)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
       self.force_reraise()
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
       six.reraise(self.type_, self.value, self.tb)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/common/utils.py", line 393, in call
       return func(*args, **kwargs)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 989, in process
       self.process_external(agent)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 491, in process_external
       self.create_dvr_fip_interfaces(ex_gw_port)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 522, in create_dvr_fip_interfaces
       self.fip_ns.update_gateway_port(fip_agent_port)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 243, in update_gateway_port
       ipd.route.add_gateway(gw_ip)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 690, in add_gateway
       self._as_root([ip_version], tuple(args))
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
       use_root_namespace=use_root_namespace)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
       log_fail_as_error=self.log_fail_as_error)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
       log_fail_as_error=log_fail_as_error)
     File "/opt/stack/venv/neutron-20160927T090820Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
       raise RuntimeError(msg)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1631513/+subscriptions