← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1938191] [NEW] L3 agent fails to process a DVR router external network change

 

Public bug reported:

I ran into a problem when L3 agent fails to process the external network change on the router and was hitting the retry limit.
I'm using a devstack deployment over the master branch.

* Pre-conditions:
L3 agent in DVR mode
mechanism driver is openvswitch

* Step-by-step reproduction steps:
- create two external networks and three internal
- create three routers and add the corresponding internal networks
- connect external networks to the routers (according to the scheme: net1->r1, net2->r2, net1->r3)
- switch the external network of the r3 router from net1 to net2

Here are the CLI commands:

openstack network create phys-net1 --external
openstack network create phys-net2 --external
openstack network create priv-net1
openstack network create priv-net2
openstack network create priv-net3

openstack subnet create --network phys-net1 --subnet-range 192.168.1.0/24 phys-sub1
openstack subnet create --network phys-net2 --subnet-range 192.168.2.0/24 phys-sub2
openstack subnet create --network priv-net1 --subnet-range 192.168.10.0/24 priv-sub1
openstack subnet create --network priv-net2 --subnet-range 192.168.20.0/24 priv-sub2
openstack subnet create --network priv-net3 --subnet-range 192.168.30.0/24 priv-sub3

openstack router create r1
openstack router create r2
openstack router create r3

openstack router add subnet r1 priv-sub1
openstack router add subnet r2 priv-sub2
openstack router add subnet r3 priv-sub3

openstack router set r1 --external-gateway phys-net1
openstack router set r2 --external-gateway phys-net2
openstack router set r3 --external-gateway phys-net1

# Switch r3 external network from phys-net1 to phys-net2:
openstack router set r3 --external-gateway phys-net2

After switching in the l3 agent logs one can observe unsuccessful attempts to process the changes and the message
(see the logs of router processing below):
'Hit retry limit with router update for <router_id>, action 3'

The state of resources and net devices:

[root@devstack ~]# openstack router list
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| ID                                   | Name | Status | State | Project                          | Distributed | HA    |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
| 6cb4a81f-9b5a-4f98-9ef2-705b369d4240 | r2   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
| 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 | r3   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
| c37e75aa-4bc1-4d56-95a1-3045d8817c26 | r1   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
+--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
[root@devstack ~]# openstack network list
+--------------------------------------+-----------+--------------------------------------+
| ID                                   | Name      | Subnets                              |
+--------------------------------------+-----------+--------------------------------------+
| 34cf22a5-8368-4935-a5a6-47bf2763d6a1 | priv-net2 | 2f067140-d6a8-4341-ac53-aef48be15877 |
| 86f5bceb-a945-48c0-ad50-ae3e395fd21f | phys-net1 | d03016ee-5724-47ea-891c-018cdd8338f1 |
| 8bbaff79-4e40-4341-b48d-76b8a62f80cd | priv-net1 | ef7dca63-29f8-4483-af7f-8ab9661232f2 |
| a3704615-3e3e-4a03-a425-5851a381e702 | phys-net2 | 647ed571-c6ee-4f7f-8ecf-8a78b5f0b534 |
| f142ca45-9cce-4619-9964-ad68b64aa0a2 | priv-net3 | e386cfdd-d52c-4830-a90b-bdc5cb656ad7 |
+--------------------------------------+-----------+--------------------------------------+
[root@devstack ~]# openstack router show r3 -c external_gateway_info
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                                                                    |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| external_gateway_info | {"network_id": "a3704615-3e3e-4a03-a425-5851a381e702", "external_fixed_ips": [{"subnet_id": "647ed571-c6ee-4f7f-8ecf-8a78b5f0b534", "ip_address": "192.168.2.42"}], "enable_snat": true} |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
[root@devstack ~]#

[root@devstack ~]# ip netns
snat-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 12)
snat-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 11)
fip-a3704615-3e3e-4a03-a425-5851a381e702 (id: 10)
snat-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 9)
fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 8)
qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 7)
qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 6)
qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 5)
qdhcp-f142ca45-9cce-4619-9964-ad68b64aa0a2 (id: 4)
qdhcp-34cf22a5-8368-4935-a5a6-47bf2763d6a1 (id: 3)
qdhcp-8bbaff79-4e40-4341-b48d-76b8a62f80cd (id: 2)
qdhcp-a3704615-3e3e-4a03-a425-5851a381e702 (id: 1)
qdhcp-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 0)
[root@devstack ~]#

[root@devstack ~]# ip netns exec qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: rfp-9e15faf3-8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:bb:61:ef:bf:63 brd ff:ff:ff:ff:ff:ff link-netns fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f
    inet 169.254.127.96/31 scope global rfp-9e15faf3-8
       valid_lft forever preferred_lft forever
    inet6 fe80::1cbb:61ff:feef:bf63/64 scope link
       valid_lft forever preferred_lft forever
53: qr-fb010c2b-df: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:78:37:48 brd ff:ff:ff:ff:ff:ff
    inet 192.168.30.1/24 brd 192.168.30.255 scope global qr-fb010c2b-df
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe78:3748/64 scope link
       valid_lft forever preferred_lft forever
[root@devstack ~]#

[root@devstack ~]# ip netns exec fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: fpr-c37e75aa-4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether b6:51:de:a8:6c:9b brd ff:ff:ff:ff:ff:ff link-netns qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26
    inet 169.254.107.95/31 scope global fpr-c37e75aa-4
       valid_lft forever preferred_lft forever
    inet6 fe80::b451:deff:fea8:6c9b/64 scope link
       valid_lft forever preferred_lft forever
3: fpr-9e15faf3-8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether e2:4e:93:f2:4a:3a brd ff:ff:ff:ff:ff:ff link-netns qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4
    inet 169.254.127.97/31 scope global fpr-9e15faf3-8
       valid_lft forever preferred_lft forever
    inet6 fe80::e04e:93ff:fef2:4a3a/64 scope link
       valid_lft forever preferred_lft forever
54: fg-9c125c91-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:31:8e:73 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.83/24 brd 192.168.1.255 scope global fg-9c125c91-bd
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe31:8e73/64 scope link
       valid_lft forever preferred_lft forever
[root@devstack ~]#

[root@devstack ~]# ip netns exec fip-a3704615-3e3e-4a03-a425-5851a381e702 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: fpr-6cb4a81f-9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:52:27:fb:62:5f brd ff:ff:ff:ff:ff:ff link-netns qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240
    inet 169.254.127.97/31 scope global fpr-6cb4a81f-9
       valid_lft forever preferred_lft forever
    inet 169.254.117.23/31 scope global fpr-6cb4a81f-9
       valid_lft forever preferred_lft forever
    inet6 fe80::ec52:27ff:fefb:625f/64 scope link
       valid_lft forever preferred_lft forever
57: fg-700fa25e-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:9f:0f:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.56/24 brd 192.168.2.255 scope global fg-700fa25e-34
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe9f:fbd/64 scope link
       valid_lft forever preferred_lft forever
[root@devstack ~]#

You can see that the 'fpr-9e15faf3' device is stuck in fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (phys-net1) namespace
while r3 should be connected to fip-a3704615-3e3e-4a03-a425-5851a381e702 namespace (phys-net2)

L3 agent logs:

Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.000
Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.001
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info [-] Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 819, in _process_routers_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self._process_updated_router(router)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 693, in _process_updated_router
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     ri.process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 914, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/common/utils.py", line 168, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.logger(e)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.force_reraise()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     raise self.value
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.process_external()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.connect_rtr_2_fip()
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.fip_ns.create_rtr_2_fip_link(self)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     fip_2_rtr_dev.link.set_mtu(mtu)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.name, self._parent.namespace, mtu=mtu_size)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     r_call_timeout)
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent
Jul 27 14:30:41 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 2, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 1.085
<cut retries>
Jul 27 14:30:46 devstack neutron-l3-agent[163379]: WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3

* Version:
* devstack deployment (master branch) based on centos-stream OS

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1938191

Title:
  L3 agent fails to process a DVR router external network change

Status in neutron:
  New

Bug description:
  I ran into a problem when L3 agent fails to process the external network change on the router and was hitting the retry limit.
  I'm using a devstack deployment over the master branch.

  * Pre-conditions:
  L3 agent in DVR mode
  mechanism driver is openvswitch

  * Step-by-step reproduction steps:
  - create two external networks and three internal
  - create three routers and add the corresponding internal networks
  - connect external networks to the routers (according to the scheme: net1->r1, net2->r2, net1->r3)
  - switch the external network of the r3 router from net1 to net2

  Here are the CLI commands:

  openstack network create phys-net1 --external
  openstack network create phys-net2 --external
  openstack network create priv-net1
  openstack network create priv-net2
  openstack network create priv-net3

  openstack subnet create --network phys-net1 --subnet-range 192.168.1.0/24 phys-sub1
  openstack subnet create --network phys-net2 --subnet-range 192.168.2.0/24 phys-sub2
  openstack subnet create --network priv-net1 --subnet-range 192.168.10.0/24 priv-sub1
  openstack subnet create --network priv-net2 --subnet-range 192.168.20.0/24 priv-sub2
  openstack subnet create --network priv-net3 --subnet-range 192.168.30.0/24 priv-sub3

  openstack router create r1
  openstack router create r2
  openstack router create r3

  openstack router add subnet r1 priv-sub1
  openstack router add subnet r2 priv-sub2
  openstack router add subnet r3 priv-sub3

  openstack router set r1 --external-gateway phys-net1
  openstack router set r2 --external-gateway phys-net2
  openstack router set r3 --external-gateway phys-net1

  # Switch r3 external network from phys-net1 to phys-net2:
  openstack router set r3 --external-gateway phys-net2

  After switching in the l3 agent logs one can observe unsuccessful attempts to process the changes and the message
  (see the logs of router processing below):
  'Hit retry limit with router update for <router_id>, action 3'

  The state of resources and net devices:

  [root@devstack ~]# openstack router list
  +--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
  | ID                                   | Name | Status | State | Project                          | Distributed | HA    |
  +--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
  | 6cb4a81f-9b5a-4f98-9ef2-705b369d4240 | r2   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
  | 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 | r3   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
  | c37e75aa-4bc1-4d56-95a1-3045d8817c26 | r1   | ACTIVE | UP    | f3f8c288836f47ca930e13620f27a8c8 | True        | False |
  +--------------------------------------+------+--------+-------+----------------------------------+-------------+-------+
  [root@devstack ~]# openstack network list
  +--------------------------------------+-----------+--------------------------------------+
  | ID                                   | Name      | Subnets                              |
  +--------------------------------------+-----------+--------------------------------------+
  | 34cf22a5-8368-4935-a5a6-47bf2763d6a1 | priv-net2 | 2f067140-d6a8-4341-ac53-aef48be15877 |
  | 86f5bceb-a945-48c0-ad50-ae3e395fd21f | phys-net1 | d03016ee-5724-47ea-891c-018cdd8338f1 |
  | 8bbaff79-4e40-4341-b48d-76b8a62f80cd | priv-net1 | ef7dca63-29f8-4483-af7f-8ab9661232f2 |
  | a3704615-3e3e-4a03-a425-5851a381e702 | phys-net2 | 647ed571-c6ee-4f7f-8ecf-8a78b5f0b534 |
  | f142ca45-9cce-4619-9964-ad68b64aa0a2 | priv-net3 | e386cfdd-d52c-4830-a90b-bdc5cb656ad7 |
  +--------------------------------------+-----------+--------------------------------------+
  [root@devstack ~]# openstack router show r3 -c external_gateway_info
  +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | Field                 | Value                                                                                                                                                                                    |
  +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  | external_gateway_info | {"network_id": "a3704615-3e3e-4a03-a425-5851a381e702", "external_fixed_ips": [{"subnet_id": "647ed571-c6ee-4f7f-8ecf-8a78b5f0b534", "ip_address": "192.168.2.42"}], "enable_snat": true} |
  +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  [root@devstack ~]#

  [root@devstack ~]# ip netns
  snat-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 12)
  snat-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 11)
  fip-a3704615-3e3e-4a03-a425-5851a381e702 (id: 10)
  snat-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 9)
  fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 8)
  qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 (id: 7)
  qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240 (id: 6)
  qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26 (id: 5)
  qdhcp-f142ca45-9cce-4619-9964-ad68b64aa0a2 (id: 4)
  qdhcp-34cf22a5-8368-4935-a5a6-47bf2763d6a1 (id: 3)
  qdhcp-8bbaff79-4e40-4341-b48d-76b8a62f80cd (id: 2)
  qdhcp-a3704615-3e3e-4a03-a425-5851a381e702 (id: 1)
  qdhcp-86f5bceb-a945-48c0-ad50-ae3e395fd21f (id: 0)
  [root@devstack ~]#

  [root@devstack ~]# ip netns exec qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4 ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: rfp-9e15faf3-8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
      link/ether 1e:bb:61:ef:bf:63 brd ff:ff:ff:ff:ff:ff link-netns fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f
      inet 169.254.127.96/31 scope global rfp-9e15faf3-8
         valid_lft forever preferred_lft forever
      inet6 fe80::1cbb:61ff:feef:bf63/64 scope link
         valid_lft forever preferred_lft forever
  53: qr-fb010c2b-df: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:78:37:48 brd ff:ff:ff:ff:ff:ff
      inet 192.168.30.1/24 brd 192.168.30.255 scope global qr-fb010c2b-df
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe78:3748/64 scope link
         valid_lft forever preferred_lft forever
  [root@devstack ~]#

  [root@devstack ~]# ip netns exec fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: fpr-c37e75aa-4@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
      link/ether b6:51:de:a8:6c:9b brd ff:ff:ff:ff:ff:ff link-netns qrouter-c37e75aa-4bc1-4d56-95a1-3045d8817c26
      inet 169.254.107.95/31 scope global fpr-c37e75aa-4
         valid_lft forever preferred_lft forever
      inet6 fe80::b451:deff:fea8:6c9b/64 scope link
         valid_lft forever preferred_lft forever
  3: fpr-9e15faf3-8@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
      link/ether e2:4e:93:f2:4a:3a brd ff:ff:ff:ff:ff:ff link-netns qrouter-9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4
      inet 169.254.127.97/31 scope global fpr-9e15faf3-8
         valid_lft forever preferred_lft forever
      inet6 fe80::e04e:93ff:fef2:4a3a/64 scope link
         valid_lft forever preferred_lft forever
  54: fg-9c125c91-bd: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:31:8e:73 brd ff:ff:ff:ff:ff:ff
      inet 192.168.1.83/24 brd 192.168.1.255 scope global fg-9c125c91-bd
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe31:8e73/64 scope link
         valid_lft forever preferred_lft forever
  [root@devstack ~]#

  [root@devstack ~]# ip netns exec fip-a3704615-3e3e-4a03-a425-5851a381e702 ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
      inet 127.0.0.1/8 scope host lo
         valid_lft forever preferred_lft forever
      inet6 ::1/128 scope host
         valid_lft forever preferred_lft forever
  2: fpr-6cb4a81f-9@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
      link/ether ee:52:27:fb:62:5f brd ff:ff:ff:ff:ff:ff link-netns qrouter-6cb4a81f-9b5a-4f98-9ef2-705b369d4240
      inet 169.254.127.97/31 scope global fpr-6cb4a81f-9
         valid_lft forever preferred_lft forever
      inet 169.254.117.23/31 scope global fpr-6cb4a81f-9
         valid_lft forever preferred_lft forever
      inet6 fe80::ec52:27ff:fefb:625f/64 scope link
         valid_lft forever preferred_lft forever
  57: fg-700fa25e-34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:9f:0f:bd brd ff:ff:ff:ff:ff:ff
      inet 192.168.2.56/24 brd 192.168.2.255 scope global fg-700fa25e-34
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:fe9f:fbd/64 scope link
         valid_lft forever preferred_lft forever
  [root@devstack ~]#

  You can see that the 'fpr-9e15faf3' device is stuck in fip-86f5bceb-a945-48c0-ad50-ae3e395fd21f (phys-net1) namespace
  while r3 should be connected to fip-a3704615-3e3e-4a03-a425-5851a381e702 namespace (phys-net2)

  L3 agent logs:

  Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.000
  Jul 27 14:30:40 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 1, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 0.001
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info [-] Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.process_external()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.connect_rtr_2_fip()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.fip_ns.create_rtr_2_fip_link(self)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     fip_2_rtr_dev.link.set_mtu(mtu)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     self.name, self._parent.namespace, mtu=mtu_size)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     r_call_timeout)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info     raise exc_type(*result[2])
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.router_info
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4: neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent Traceback (most recent call last):
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 819, in _process_routers_if_compatible
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 635, in _process_router_if_compatible
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 693, in _process_updated_router
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     ri.process()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 914, in process
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     super(DvrLocalRouter, self).process()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_router_base.py", line 29, in process
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     super(DvrRouterBase, self).process()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/common/utils.py", line 168, in call
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.logger(e)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.force_reraise()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     raise self.value
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/common/utils.py", line 165, in call
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1313, in process
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.process_external()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 729, in process_external
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.connect_rtr_2_fip()
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_local_router.py", line 744, in connect_rtr_2_fip
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.fip_ns.create_rtr_2_fip_link(self)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/l3/dvr_fip_ns.py", line 435, in create_rtr_2_fip_link
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     fip_2_rtr_dev.link.set_mtu(mtu)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 432, in set_mtu
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     self.name, self._parent.namespace, mtu=mtu_size)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 272, in _wrap
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     r_call_timeout)
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent   File "/usr/local/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 216, in remote_call
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.NetworkInterfaceNotFound: Network interface fpr-9e15faf3-8 not found in namespace fip-a3704615-3e3e-4a03-a425-5851a381e702.
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: ERROR neutron.agent.l3.agent
  Jul 27 14:30:41 devstack neutron-l3-agent[163379]: INFO neutron.agent.l3.agent [-] Starting processing update 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3, priority 2, update_id 0f60fc6f-0497-40bb-9c6b-c4ab5956b9d9. Wait time elapsed: 1.085
  <cut retries>
  Jul 27 14:30:46 devstack neutron-l3-agent[163379]: WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for 9e15faf3-8478-4b2a-83f1-ad2cc8cd9de4, action 3

  * Version:
  * devstack deployment (master branch) based on centos-stream OS

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1938191/+subscriptions