yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #86733
[Bug 1938191] [NEW] L3 agent fails to process a DVR router external network change
Public bug reported:
I ran into a problem when L3 agent fails to process the external network change on the router and was hitting the retry limit.
I'm using a devstack deployment over the master branch.
* Pre-conditions:
L3 agent in DVR mode
mechanism driver is openvswitch
* Step-by-step reproduction steps:
- create two external networks and three internal
- create three routers and add the corresponding internal networks
- connect external networks to the routers (according to the scheme: net1->r1, net2->r2, net1->r3)
- switch the external network of the r3 router from net1 to net2
Here are the CLI commands:
openstack network create phys-net1 --external
openstack network create phys-net2 --external
openstack network create priv-net1
openstack network create priv-net2
openstack network create priv-net3
openstack subnet create --network phys-net1 --subnet-range 192.168.1.0/24 phys-sub1
openstack subnet create --network phys-net2 --subnet-range 192.168.2.0/24 phys-sub2
openstack subnet create --network priv-net1 --subnet-range 192.168.10.0/24 priv-sub1
openstack subnet create --network priv-net2 --subnet-range 192.168.20.0/24 priv-sub2
openstack subnet create --network priv-net3 --subnet-range 192.168.30.0/24 priv-sub3
openstack router create r1
openstack router create r2
openstack router create r3
openstack router add subnet r1 priv-sub1
openstack router add subnet r2 priv-sub2
openstack router add subnet r3 priv-sub3
openstack router set r1 --external-gateway phys-net1
openstack router set r2 --external-gateway phys-net2
openstack router set r3 --external-gateway phys-net1
# Switch r3 external network from phys-net1 to phys-net2:
openstack router set r3 --external-gateway phys-net2
After switching in the l3 agent logs one can observe unsuccessful attempts to process the changes and the message
(see the logs of router processing below):
'Hit retry limit with router update for <router_id>, action 3'
The state of resources and net devices:
[root@devstack ~]# openstack router list
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| 6cb4a81f-9b5a-4f98-9ef2-705b369d4240 | r2 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
| 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 | r3 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
| c37e75aa-4bc1-4d56-95a1-3045d8817c26 | r1 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
[root@devstack ~]# openstack network list
+--------------------------------------+-----------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-----------+--------------------------------------+
| 34cf22a5-8368-4935-a5a6-47bf2763d6a1 | priv-net2 | 2f067140-d6a8-4341-ac53-aef48be15877 |
| 86f5bceb-a945-48c0-ad50-ae3e395fd21f | phys-net1 | d03016ee-5724-47ea-891c-018cdd8338f1 |
| 8bbaff79-4e40-4341-b48d-76b8a62f80cd | priv-net1 | ef7dca63-29f8-4483-af7f-8ab9661232f2 |
| a3704615-3e3e-4a03-a425-5851a381e702 | phys-net2 | 647ed571-c6ee-4f7f-8ecf-8a78b5f0b534 |
| f142ca45-9cce-4619-9964-ad68b64aa0a2 | priv-net3 | e386cfdd-d52c-4830-a90b-bdc5cb656ad7 |
+--------------------------------------+-----------+--------------------------------------+
[root@devstack ~]# openstack router show r3 -c external_gateway_info
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| external_gateway_info | {"network_id": "a3704615-3e3e-4a03-a425-5851a381e702", "external_fixed_ips": [{"subnet_id": "647ed571-c6ee-4f7f-8ecf-8a78b5f0b534", "ip_address": "192.168.2.42"}], "enable_snat": true} |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@devstack ~]#
[root@devstack ~]# ip netns
snat-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 12)
snat-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 11)
fip-a3704615-3e3e-4a03-a425-5851a381e702 (id: 10)
snat-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 9)
fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 8)
qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 7)
qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 6)
qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 5)
qdhcp-f142ca45-9cce-4619-9964-ad68b64aa0a2 (id: 4)
qdhcp-34cf22a5-8368-4935-a5a6-47bf2763d6a1 (id: 3)
qdhcp-8bbaff79-4e40-4341-b48d-76b8a62f80cd (id: 2)
qdhcp-a3704615-3e3e-4a03-a425-5851a381e702 (id: 1)
qdhcp-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 0)
[root@devstack ~]#
[root@devstack ~]# ip netns exec qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-9e15faf3-8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 1e:bb:61:ef:bf:63 brd ff:ff:ff:ff:ff:ff link-netns fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f
inet 169.254.127.96/31 scope global rfp-9e15faf3-8
valid_lft forever preferred_lft forever
inet6 fe80::1cbb:61ff:feef:bf63/64 scope link
valid_lft forever preferred_lft forever
53: qr-fb010c2b-df: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:78:37:48 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.1/24 brd 192.168.30.255 scope global qr-fb010c2b-df
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe78:3748/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
[root@devstack ~]# ip netns exec fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-c37e75aa-4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether b6:51:de:a8:6c:9b brd ff:ff:ff:ff:ff:ff link-netns qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26
inet 169.254.107.95/31 scope global fpr-c37e75aa-4
valid_lft forever preferred_lft forever
inet6 fe80::b451:deff:fea8:6c9b/64 scope link
valid_lft forever preferred_lft forever
3: fpr-9e15faf3-8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether e2:4e:93:f2:4a:3a brd ff:ff:ff:ff:ff:ff link-netns qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4
inet 169.254.127.97/31 scope global fpr-9e15faf3-8
valid_lft forever preferred_lft forever
inet6 fe80::e04e:93ff:fef2:4a3a/64 scope link
valid_lft forever preferred_lft forever
54: fg-9c125c91-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:31:8e:73 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.83/24 brd 192.168.1.255 scope global fg-9c125c91-bd
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe31:8e73/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
[root@devstack ~]# ip netns exec fip-a3704615-3e3e-4a03-a425-5851a381e702 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-6cb4a81f-9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether ee:52:27:fb:62:5f brd ff:ff:ff:ff:ff:ff link-netns qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240
inet 169.254.127.97/31 scope global fpr-6cb4a81f-9
valid_lft forever preferred_lft forever
inet 169.254.117.23/31 scope global fpr-6cb4a81f-9
valid_lft forever preferred_lft forever
inet6 fe80::ec52:27ff:fefb:625f/64 scope link
valid_lft forever preferred_lft forever
57: fg-700fa25e-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:9f:0f:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.2.56/24 brd 192.168.2.255 scope global fg-700fa25e-34
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9f:fbd/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
You can see that the 'fpr-9e15faf3' device is stuck in fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (phys-net1) namespace
while r3 should be connected to fip-a3704615-3e3e-4a03-a425-5851a381e702 namespace (phys-net2)
L3 agent logs:
Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.000
Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.001
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info [-] Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 819, in _process_routers_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self._process_updated_router(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 693, in _process_updated_router
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent ri.process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 914, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 168, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.logger(e)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.force_reraise()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent raise self.value
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 2, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 1.085
<cut retries>
Jul 27 14:30:46 devstack neutron-l3-agent[163379]: WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3
* Version:
* devstack deployment (master branch) based on centos-stream OS
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1938191
Title:
L3 agent fails to process a DVR router external network change
Status in neutron:
New
Bug description:
I ran into a problem when L3 agent fails to process the external network change on the router and was hitting the retry limit.
I'm using a devstack deployment over the master branch.
* Pre-conditions:
L3 agent in DVR mode
mechanism driver is openvswitch
* Step-by-step reproduction steps:
- create two external networks and three internal
- create three routers and add the corresponding internal networks
- connect external networks to the routers (according to the scheme: net1->r1, net2->r2, net1->r3)
- switch the external network of the r3 router from net1 to net2
Here are the CLI commands:
openstack network create phys-net1 --external
openstack network create phys-net2 --external
openstack network create priv-net1
openstack network create priv-net2
openstack network create priv-net3
openstack subnet create --network phys-net1 --subnet-range 192.168.1.0/24 phys-sub1
openstack subnet create --network phys-net2 --subnet-range 192.168.2.0/24 phys-sub2
openstack subnet create --network priv-net1 --subnet-range 192.168.10.0/24 priv-sub1
openstack subnet create --network priv-net2 --subnet-range 192.168.20.0/24 priv-sub2
openstack subnet create --network priv-net3 --subnet-range 192.168.30.0/24 priv-sub3
openstack router create r1
openstack router create r2
openstack router create r3
openstack router add subnet r1 priv-sub1
openstack router add subnet r2 priv-sub2
openstack router add subnet r3 priv-sub3
openstack router set r1 --external-gateway phys-net1
openstack router set r2 --external-gateway phys-net2
openstack router set r3 --external-gateway phys-net1
# Switch r3 external network from phys-net1 to phys-net2:
openstack router set r3 --external-gateway phys-net2
After switching in the l3 agent logs one can observe unsuccessful attempts to process the changes and the message
(see the logs of router processing below):
'Hit retry limit with router update for <router_id>, action 3'
The state of resources and net devices:
[root@devstack ~]# openstack router list
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| ID | Name | Status | State | Project | Distributed | HA |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| 6cb4a81f-9b5a-4f98-9ef2-705b369d4240 | r2 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
| 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 | r3 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
| c37e75aa-4bc1-4d56-95a1-3045d8817c26 | r1 | ACTIVE | UP | f3f8c288836f47ca930e13620f27a8c8 | True | False |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
[root@devstack ~]# openstack network list
+--------------------------------------+-----------+--------------------------------------+
| ID | Name | Subnets |
+--------------------------------------+-----------+--------------------------------------+
| 34cf22a5-8368-4935-a5a6-47bf2763d6a1 | priv-net2 | 2f067140-d6a8-4341-ac53-aef48be15877 |
| 86f5bceb-a945-48c0-ad50-ae3e395fd21f | phys-net1 | d03016ee-5724-47ea-891c-018cdd8338f1 |
| 8bbaff79-4e40-4341-b48d-76b8a62f80cd | priv-net1 | ef7dca63-29f8-4483-af7f-8ab9661232f2 |
| a3704615-3e3e-4a03-a425-5851a381e702 | phys-net2 | 647ed571-c6ee-4f7f-8ecf-8a78b5f0b534 |
| f142ca45-9cce-4619-9964-ad68b64aa0a2 | priv-net3 | e386cfdd-d52c-4830-a90b-bdc5cb656ad7 |
+--------------------------------------+-----------+--------------------------------------+
[root@devstack ~]# openstack router show r3 -c external_gateway_info
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| external_gateway_info | {"network_id": "a3704615-3e3e-4a03-a425-5851a381e702", "external_fixed_ips": [{"subnet_id": "647ed571-c6ee-4f7f-8ecf-8a78b5f0b534", "ip_address": "192.168.2.42"}], "enable_snat": true} |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@devstack ~]#
[root@devstack ~]# ip netns
snat-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 12)
snat-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 11)
fip-a3704615-3e3e-4a03-a425-5851a381e702 (id: 10)
snat-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 9)
fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 8)
qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 7)
qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 6)
qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 5)
qdhcp-f142ca45-9cce-4619-9964-ad68b64aa0a2 (id: 4)
qdhcp-34cf22a5-8368-4935-a5a6-47bf2763d6a1 (id: 3)
qdhcp-8bbaff79-4e40-4341-b48d-76b8a62f80cd (id: 2)
qdhcp-a3704615-3e3e-4a03-a425-5851a381e702 (id: 1)
qdhcp-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 0)
[root@devstack ~]#
[root@devstack ~]# ip netns exec qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: rfp-9e15faf3-8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 1e:bb:61:ef:bf:63 brd ff:ff:ff:ff:ff:ff link-netns fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f
inet 169.254.127.96/31 scope global rfp-9e15faf3-8
valid_lft forever preferred_lft forever
inet6 fe80::1cbb:61ff:feef:bf63/64 scope link
valid_lft forever preferred_lft forever
53: qr-fb010c2b-df: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:78:37:48 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.1/24 brd 192.168.30.255 scope global qr-fb010c2b-df
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe78:3748/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
[root@devstack ~]# ip netns exec fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-c37e75aa-4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether b6:51:de:a8:6c:9b brd ff:ff:ff:ff:ff:ff link-netns qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26
inet 169.254.107.95/31 scope global fpr-c37e75aa-4
valid_lft forever preferred_lft forever
inet6 fe80::b451:deff:fea8:6c9b/64 scope link
valid_lft forever preferred_lft forever
3: fpr-9e15faf3-8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether e2:4e:93:f2:4a:3a brd ff:ff:ff:ff:ff:ff link-netns qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4
inet 169.254.127.97/31 scope global fpr-9e15faf3-8
valid_lft forever preferred_lft forever
inet6 fe80::e04e:93ff:fef2:4a3a/64 scope link
valid_lft forever preferred_lft forever
54: fg-9c125c91-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:31:8e:73 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.83/24 brd 192.168.1.255 scope global fg-9c125c91-bd
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe31:8e73/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
[root@devstack ~]# ip netns exec fip-a3704615-3e3e-4a03-a425-5851a381e702 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: fpr-6cb4a81f-9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether ee:52:27:fb:62:5f brd ff:ff:ff:ff:ff:ff link-netns qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240
inet 169.254.127.97/31 scope global fpr-6cb4a81f-9
valid_lft forever preferred_lft forever
inet 169.254.117.23/31 scope global fpr-6cb4a81f-9
valid_lft forever preferred_lft forever
inet6 fe80::ec52:27ff:fefb:625f/64 scope link
valid_lft forever preferred_lft forever
57: fg-700fa25e-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:16:3e:9f:0f:bd brd ff:ff:ff:ff:ff:ff
inet 192.168.2.56/24 brd 192.168.2.255 scope global fg-700fa25e-34
valid_lft forever preferred_lft forever
inet6 fe80::f816:3eff:fe9f:fbd/64 scope link
valid_lft forever preferred_lft forever
[root@devstack ~]#
You can see that the 'fpr-9e15faf3' device is stuck in fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (phys-net1) namespace
while r3 should be connected to fip-a3704615-3e3e-4a03-a425-5851a381e702 namespace (phys-net2)
L3 agent logs:
Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.000
Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.001
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info [-] Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 819, in _process_routers_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self._process_updated_router(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 693, in _process_updated_router
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent ri.process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 914, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent super(DvrLocalRouter, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent super(DvrRouterBase, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 168, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.logger(e)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.force_reraise()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent raise self.value
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 2, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 1.085
<cut retries>
Jul 27 14:30:46 devstack neutron-l3-agent[163379]: WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3
* Version:
* devstack deployment (master branch) based on centos-stream OS
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1938191/+subscriptions
