yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1883089] [NEW] [L3] floating IP failed to bind due to no agent gateway port(fip-ns)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: LIU Yulong <1883089@xxxxxxxxxxxxxxxxxx>
Date: Thu, 11 Jun 2020 10:44:43 -0000
Reply-to: Bug 1883089 <1883089@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In patch [1] it introduced a binding of DB uniq constraint for L3
agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
is in DB while the gateway port not. The current code path only checks
the binding existence which will pass a "None" port to the following
code path that results an AttributeError.

[1] https://review.opendev.org/#/c/702547/


Exception log:

2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     admin_ctx, network_id, host)
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self._populate_mtu_and_subnets_for_ports(context, [agent_port])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     for p in self._each_port_having_fixed_ips(ports)]
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     fixed_ips = port.get('fixed_ips', [])
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server

** Affects: neutron
     Importance: Medium
     Assignee: LIU Yulong (dragon889)
         Status: Confirmed

** Changed in: neutron
     Assignee: (unassigned) => LIU Yulong (dragon889)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1883089

Title:
  [L3] floating IP failed to bind due to no agent gateway port(fip-ns)

Status in neutron:
  Confirmed

Bug description:
  In patch [1] it introduced a binding of DB uniq constraint for L3
  agent gateway. In some extreme case the DvrFipGatewayPortAgentBinding
  is in DB while the gateway port not. The current code path only checks
  the binding existence which will pass a "None" port to the following
  code path that results an AttributeError.

  [1] https://review.opendev.org/#/c/702547/

  
  Exception log:

  2020-06-11 15:39:28.361 1285214 INFO neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 does not exist on host host-compute-1. Creating one.
  2020-06-11 15:39:28.370 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port for network 3fcb7702-ae0b-46b4-807f-8ae94d656dd3 already exists on host host-compute-1. Probably it was just created by other worker. create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:927
  2020-06-11 15:39:28.390 1285214 DEBUG neutron.db.l3_dvr_db [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Floating IP Agent Gateway port None found for the destination host: host-compute-1 create_fip_agent_gw_port_if_not_exists /usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py:933
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server [None req-d6a41187-2495-46bf-a424-ab7195c0ecb1 - - - - -] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'get'
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 170, in _process_incoming
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     setattr(e, '_RETRY_EXCEEDED', True)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 87, in wrapped
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     ectxt.value = e.inner_exc
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     LOG.debug("Retry wrapper got retriable exception: %s", e)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 122, in wrapped
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     return f(*dup_args, **dup_kwargs)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 348, in get_agent_gateway_port
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     admin_ctx, network_id, host)
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_dvr_db.py", line 953, in create_fip_agent_gw_port_if_not_exists
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     self._populate_mtu_and_subnets_for_ports(context, [agent_port])
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1978, in _populate_mtu_and_subnets_for_ports
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     for p in self._each_port_having_fixed_ips(ports)]
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/neutron/db/l3_db.py", line 1925, in _each_port_having_fixed_ips
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server     fixed_ips = port.get('fixed_ips', [])
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get'
  2020-06-11 15:39:28.391 1285214 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1883089/+subscriptions
Follow ups

[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Hemanth Nakkina, 2021-06-21
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Hemanth Nakkina, 2021-06-21
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Launchpad Bug Tracker, 2021-05-24
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Edward Hope-Morley, 2021-04-27
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Hemanth Nakkina, 2021-04-27
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: Hemanth Nakkina, 2021-04-27
[Bug 1883089] Re: [L3] floating IP failed to bind due to no agent gateway port(fip-ns)
From: OpenStack Infra, 2020-07-02