yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1983600] [NEW] Neutron should clean up ACLs in OVN NB DB when a remote security group is deleted

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miro Tomaska <1983600@xxxxxxxxxxxxxxxxxx>
Date: Thu, 04 Aug 2022 15:30:30 -0000
Reply-to: Bug 1983600 <1983600@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Neutron does not clean up ACLs in OVN NB DB when a remote security group
is deleted in Neutron. This causes some warning in OVN logs. This issue
does not impact functionality but it would be great if we could clean
this up and make those logs go away. I think this BZ can be marked as
low priority and low hanging fruit.

Reproducing steps:

1. A security group rule which has a rule that uses a remote security
group

openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group                |
+--------------------------------------+-------------+----------+------------+--------------------------------------+
| 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp        | None     |            | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                                 |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                                 |
+--------------------------------------+-------------+----------+------------+--------------------------------------+

Here is a view in ovn nb db

docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
  to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

2. now remove the remote security group and check the rules in the first
security group

openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5

openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
+--------------------------------------+-------------+----------+------------+-----------------------+
| ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+----------+------------+-----------------------+
| 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                  |
| b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                  |
+--------------------------------------+-------------+----------+------------+-----------------------+

So from neutron the security group rule is removed

3. check the acl in ovn again and see they are still there

docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
  to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

this causes warnings to be generated in ovn

2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.

What that is happening:
Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use.

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  Neutron does not clean up ACLs in OVN NB DB when a remote security group
  is deleted in Neutron. This causes some warning in OVN logs. This issue
  does not impact functionality but it would be great if we could clean
- this up and make those logs go away.
+ this up and make those logs go away. I think this BZ can be marked as
+ low priority and low hanging fruit.
  
  Reproducing steps:
  
  1. A security group rule which has a rule that uses a remote security
  group
  
  openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
  +--------------------------------------+-------------+----------+------------+--------------------------------------+
  | ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group                |
  +--------------------------------------+-------------+----------+------------+--------------------------------------+
  | 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp        | None     |            | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
  | 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                                 |
  | b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                                 |
  +--------------------------------------+-------------+----------+------------+--------------------------------------+
  
  Here is a view in ovn nb db
  
  docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
-   to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
+   to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
  
  2. now remove the remote security group and check the rules in the first
  security group
  
  openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5
  
  openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
  +--------------------------------------+-------------+----------+------------+-----------------------+
  | ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group |
  +--------------------------------------+-------------+----------+------------+-----------------------+
  | 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                  |
  | b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                  |
  +--------------------------------------+-------------+----------+------------+-----------------------+
  
  So from neutron the security group rule is removed
  
  3. check the acl in ovn again and see they are still there
  
  docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
-   to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
+   to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related
  
  this causes warnings to be generated in ovn
  
  2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
  2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.
  
- 
  What that is happening:
  Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1983600

Title:
  Neutron should clean up ACLs in OVN  NB DB when a remote security
  group is deleted

Status in neutron:
  New

Bug description:
  Neutron does not clean up ACLs in OVN NB DB when a remote security
  group is deleted in Neutron. This causes some warning in OVN logs.
  This issue does not impact functionality but it would be great if we
  could clean this up and make those logs go away. I think this BZ can
  be marked as low priority and low hanging fruit.

  Reproducing steps:

  1. A security group rule which has a rule that uses a remote security
  group

  openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
  +--------------------------------------+-------------+----------+------------+--------------------------------------+
  | ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group                |
  +--------------------------------------+-------------+----------+------------+--------------------------------------+
  | 1155ffc7-9a99-41c7-bd65-fac33b719973 | icmp        | None     |            | 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5 |
  | 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                                 |
  | b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                                 |
  +--------------------------------------+-------------+----------+------------+--------------------------------------+

  Here is a view in ovn nb db

  docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
    to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

  2. now remove the remote security group and check the rules in the
  first security group

  openstack security group delete 4b0f872b-22ae-4ac2-b54e-ea1678a88dd5

  openstack security group rule list 44038bce-38fd-4219-ae22-bafaad9bbde9
  +--------------------------------------+-------------+----------+------------+-----------------------+
  | ID                                   | IP Protocol | IP Range | Port Range | Remote Security Group |
  +--------------------------------------+-------------+----------+------------+-----------------------+
  | 8e4cdd63-c3e8-411f-8d72-c5cced5d4fc8 | None        | None     |            | None                  |
  | b72b29df-d5f1-4815-92ae-71b1cdc35f97 | None        | None     |            | None                  |
  +--------------------------------------+-------------+----------+------------+-----------------------+

  So from neutron the security group rule is removed

  3. check the acl in ovn again and see they are still there

  docker exec -it ovn-dbs-bundle-docker-1 ovn-nbctl acl-list pg_44038bce_38fd_4219_ae22_bafaad9bbde9
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4) allow-related
  from-lport  1002 (inport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip6) allow-related
    to-lport  1002 (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4) allow-related

  this causes warnings to be generated in ovn

  2020-11-17T14:04:59Z|00708|lflow|WARN|Dropped 1 log messages in last 917 seconds (most recently, 917 seconds ago) due to excessive rate
  2020-11-17T14:04:59Z|00709|lflow|WARN|error parsing match "((ct.new && !ct.est) || (!ct.new && ct.est && !ct.rpl && ct_label.blocked == 1)) && (outport == @pg_44038bce_38fd_4219_ae22_bafaad9bbde9 && ip4 && ip4.src == $pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4 && icmp4)": Syntax error at `$pg_4b0f872b_22ae_4ac2_b54e_ea1678a88dd5_ip4' expecting address set name.

  What that is happening:
  Only security groups that are not in use by other ports can be deleted. If a security group referenced as remote by other group is removed, the security group rule is deleted in DB because of the references in DB tables. It means there is no code triggered in the Neutron code. The corresponding port group is deleted in OVN but OVN doesn't have any on delete cascade functionality like SQL has. It means ACL remains in the OVN DB using port group that no longer exists. It has no impact on the traffic because the deleted security group was not in use.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1983600/+subscriptions
Follow ups

[Bug 1983600] Re: Neutron should clean up ACLs in OVN NB DB when a remote security group is deleted
From: OpenStack Infra, 2022-08-24