yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1943449] [NEW] VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Christian Rohmann <1943449@xxxxxxxxxxxxxxxxxx>
Date: Mon, 13 Sep 2021 12:46:58 -0000
Reply-to: Bug 1943449 <1943449@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

On OpenStack Ussuri (on Ubuntu bionic) running Neutron using the linux
bridge driver we observed an issue with VPNaaS ...


After an existing VPN setup was reconfigured by a user via terraform the
site connections remained in "DOWN" state:

```
openstack vpn ipsec site connection list
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
| ID                                   | Name                                     | Peer Address    | Authentication Algorithm | Status |
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
| b7634f92-bac8-4fed-aa28-e9181482176e | site-connection-1-REDACTED               | xxx.xxx.xxx.99  | psk                      | DOWN   |
| 543fa57d-e15e-444b-b5d2-20752196f57a | site-connection-2-REDACTED               | xxx.xxx.xxx.156 | psk                      | DOWN   |
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
```


First only the endpoint group and the connection was reconfigured, but after this causes the connection to remain DOWN the user also tried tearing down the connection, endpoints, vpn service, policies, ... only leaving the network and the router in place (which are actively used and hosting other resources such as instances).


2) Looking at the neutron logs on the active network node (HA router) we
saw tons of messages about duplicate IPtables, all of them for this very
setup and with pol "ipsec":


[...]
neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 192.168.0.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
[...]


We then simply restarted the neutron-l3-agent.service which caused the
active router instance to switch to another node and things got back in
working order quite quickly:

```
openstack vpn ipsec site connection list
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
| ID                                   | Name                                     | Peer Address    | Authentication Algorithm | Status |
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
| b7634f92-bac8-4fed-aa28-e9181482176e | site-connection-1-REDACTED               | xxx.xxx.xxx.99  | psk                      | ACTIVE |
| 543fa57d-e15e-444b-b5d2-20752196f57a | site-connection-2-REDACTED               | xxx.xxx.xxx.156 | psk                      | ACTIVE |
+--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
```


But the messages about duplicate iptables rules were thrown again on the restarted, now inactive network / router node, so there must be some clean up / rules generation issue and I believe this issue will return when the active router instance is switched back to the previous master node.


I tried to find an existing bug and found
 * https://bugs.launchpad.net/neutron/+bug/1447651
 * https://bugs.launchpad.net/neutron/+bug/1845145

to be somewhat related (duplicate iptables rules).

** Affects: neutron
     Importance: Undecided
         Status: New

** Summary changed:

- VPNaaS reconfiguration causes duplicate IPtable rules causes the VPN connection to remain DOWN
+ VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1943449

Title:
  VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN
  connection to remain DOWN

Status in neutron:
  New

Bug description:
  On OpenStack Ussuri (on Ubuntu bionic) running Neutron using the linux
  bridge driver we observed an issue with VPNaaS ...


  
  After an existing VPN setup was reconfigured by a user via terraform the site connections remained in "DOWN" state:

  ```
  openstack vpn ipsec site connection list
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  | ID                                   | Name                                     | Peer Address    | Authentication Algorithm | Status |
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  | b7634f92-bac8-4fed-aa28-e9181482176e | site-connection-1-REDACTED               | xxx.xxx.xxx.99  | psk                      | DOWN   |
  | 543fa57d-e15e-444b-b5d2-20752196f57a | site-connection-2-REDACTED               | xxx.xxx.xxx.156 | psk                      | DOWN   |
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  ```

  
  First only the endpoint group and the connection was reconfigured, but after this causes the connection to remain DOWN the user also tried tearing down the connection, endpoints, vpn service, policies, ... only leaving the network and the router in place (which are actively used and hosting other resources such as instances).


  2) Looking at the neutron logs on the active network node (HA router)
  we saw tons of messages about duplicate IPtables, all of them for this
  very setup and with pol "ipsec":

  
  [...]
  neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
  neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.382 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.383 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
  neutron-l3-agent.log:2021-09-13 10:46:24.384 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.30.122.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
  neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 192.168.0.0/24 -m policy --dir out --pol ipsec -j ACCEPT 
  neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.385 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.10.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.16.0/24 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.8.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  neutron-l3-agent.log:2021-09-13 10:46:24.386 4087474 WARNING neutron.agent.linux.iptables_manager [req-a169b585-589e-4d62-95d0-2eb6c2156ec2 df1d997e986b41d0b273945def7df72d 08cf58d22b314283b77bfa68a8611001 - - -] Duplicate iptables rule detected. This may indicate a bug in the ipt
  ables rule generation code. Line: -A neutron-l3-agent-POSTROUTING -s 10.0.0.0/21 -d 10.96.0.0/16 -m policy --dir out --pol ipsec -j ACCEPT
  [...]


  
  We then simply restarted the neutron-l3-agent.service which caused the active router instance to switch to another node and things got back in working order quite quickly:

  ```
  openstack vpn ipsec site connection list
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  | ID                                   | Name                                     | Peer Address    | Authentication Algorithm | Status |
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  | b7634f92-bac8-4fed-aa28-e9181482176e | site-connection-1-REDACTED               | xxx.xxx.xxx.99  | psk                      | ACTIVE |
  | 543fa57d-e15e-444b-b5d2-20752196f57a | site-connection-2-REDACTED               | xxx.xxx.xxx.156 | psk                      | ACTIVE |
  +--------------------------------------+------------------------------------------+-----------------+--------------------------+--------+
  ```

  
  But the messages about duplicate iptables rules were thrown again on the restarted, now inactive network / router node, so there must be some clean up / rules generation issue and I believe this issue will return when the active router instance is switched back to the previous master node.



  I tried to find an existing bug and found
   * https://bugs.launchpad.net/neutron/+bug/1447651
   * https://bugs.launchpad.net/neutron/+bug/1845145

  to be somewhat related (duplicate iptables rules).

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1943449/+subscriptions