yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1991501] [NEW] l3 agent fails to init router breaking floating ips

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: sean mooney <1991501@xxxxxxxxxxxxxxxxxx>
Date: Mon, 03 Oct 2022 02:36:25 -0000
Reply-to: Bug 1991501 <1991501@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 639, in _process_router_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 719, in _process_updated_router
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     ri.process()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.logger(e)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.force_reraise()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     raise self.value
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 174, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.process_external()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return caller(func, *(extras + args), **kw)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return f(*a, **k)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 931, in _process_external_gateway
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.external_gateway_updated(ex_gw_port, interface_name)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 884, in external_gateway_updated
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._external_gateway_added(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 822, in _external_gateway_added
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._external_gateway_settings(ex_gw_port, interface_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 835, in _external_gateway_settings
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.driver.init_router_port(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 171, in init_router_port
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.init_l3(device_name=device_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 146, in init_l3
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     device.delete_addr_and_conntrack_state(ip_cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 361, in delete_addr_and_conntrack_state
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.addr.delete(cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 540, in delete
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     delete_ip_address(cidr, self.name, self._parent.namespace)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     privileged.delete_ip_address(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent 
2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3


version info
yoga deoployed with kolla-ansible using debian ubuntu or centos image and popos 22.04 host

popos 22.04 is based on ubuntu 22.04 and uses nftables intead of legacy
ip tables

in the contianer i noticed that iptable -L is slow but iptable -S is fast
it is not clare if this is a race or some other issue but intially the router is fine and hte floating ips work and restarting the l3 agent fixes the issue for a time then it breaks with this message.

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: l3 ovs

** Tags added: l3 ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1991501

Title:
  l3 agent fails to init router breaking floating ips

Status in neutron:
  New

Bug description:
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 639, in _process_router_if_compatible
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 719, in _process_updated_router
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     ri.process()
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.logger(e)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.force_reraise()
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     raise self.value
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 174, in call
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.process_external()
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return caller(func, *(extras + args), **kw)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py", line 78, in _synchronized
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return f(*a, **k)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 931, in _process_external_gateway
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.external_gateway_updated(ex_gw_port, interface_name)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 884, in external_gateway_updated
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._external_gateway_added(
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 822, in _external_gateway_added
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self._external_gateway_settings(ex_gw_port, interface_name,
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 835, in _external_gateway_settings
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.driver.init_router_port(
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 171, in init_router_port
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.init_l3(device_name=device_name,
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 146, in init_l3
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     device.delete_addr_and_conntrack_state(ip_cidr)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 361, in delete_addr_and_conntrack_state
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     self.addr.delete(cidr)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 540, in delete
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     delete_ip_address(cidr, self.name, self._parent.namespace)
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     privileged.delete_ip_address(
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs,
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
  2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent 
  2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3

  
  version info
  yoga deoployed with kolla-ansible using debian ubuntu or centos image and popos 22.04 host

  popos 22.04 is based on ubuntu 22.04 and uses nftables intead of
  legacy ip tables

  in the contianer i noticed that iptable -L is slow but iptable -S is fast
  it is not clare if this is a race or some other issue but intially the router is fine and hte floating ips work and restarting the l3 agent fixes the issue for a time then it breaks with this message.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1991501/+subscriptions
Follow ups

[Bug 1991501] Re: l3 agent fails to process router breaking floating ips when updating them on the router qg port via pyroute2
From: Brian Haley, 2023-01-24