yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2019449] [NEW] Neutron server log file full of tracebacks in neutron-tempest-plugin-ovn job

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Miguel Lavalle <2019449@xxxxxxxxxxxxxxxxxx>
Date: Fri, 12 May 2023 23:25:59 -0000
Reply-to: Bug 2019449 <2019449@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Since we merged https://review.opendev.org/c/openstack/neutron/+/876716,
the Neutron server log in the neutron-tempest-plugin-ovn job is full of
two types of tracebacks. The first type is

May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager [None req-f256cbd3-ea20-49e8-a9a0-0299fc7f34f3 tempest-SecGroupRulesQuotaTest-217177324 tempest-SecGroupRulesQuotaTest-217177324-project-member] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_after_del_notification-316503 security_group_rule, after_delete: KeyError: 'rule'
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 463, in _process_sg_rule_after_del_notification
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     sg_rule = payload.metadata['rule']
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager KeyError: 'rule'
May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager 

This happens because:

1) The patch mentioned above now publishes AFTER_DELETE events for
security group rules that include the rule deleted:
https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/db/securitygroups_db.py#293.
The same patch added an event handler that expects to find the deleted
security group rule in the payload:
https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#459

2) However, previous to this patch, we were already publishing
AFTER_DELETE events for security group rules that don't include the
security group rule:
https://github.com/openstack/neutron/blob/47d4ec4e99d5aae62656c88206eb6a77f70d4a8b/neutron/db/securitygroups_db.py#L901-L908.
As a consequence, the new handler added by
https://review.opendev.org/c/openstack/neutron/+/876716 produces the
KeyError exception shown above when trying to access the security group
rule in the payload when processing the preexisting event.


The second type of traceback is:

May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager [None req-4f41bf6e-d3c6-42a8-a987-b062a9a6b375 admin admin] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_after_del_notification-316503 security_group_rule, after_delete: neutron.extensions.securitygroup.SecurityGroupNotFound: Security group ae8107d1-0399-4491-8535-4aee2c9de5d0 does not exist
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group_rule(context, sg_rule)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2413, in delete_security_group_rule
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self._process_security_group_rule(rule, is_add_acl=False)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2401, in _process_security_group_rule
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ovn_acl.update_acls_for_security_group(
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     sg = plugin.get_security_group(admin_context, security_group_id)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 226, in wrapped
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f_with_retry(*args, **kwargs,
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     setattr(e, '_RETRY_EXCEEDED', True)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ectxt.value = e.inner_exc
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     context_reference.session.rollback()
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*dup_args, **dup_kwargs)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 209, in get_security_group
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ret = self._make_security_group_dict(self._get_security_group(
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 226, in _get_security_group
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise ext_sg.SecurityGroupNotFound(id=id)
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group ae8107d1-0399-4491-8535-4aee2c9de5d0 does not exist
May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager

This happens because the default security groups that we create for
every project include two rules that point to the security group they
belong to as the remote security group. As a consequence, when the event
handler added by https://review.opendev.org/c/openstack/neutron/+/876716
tries to delete these type of rules, it doesn't find their security
group, because it was deleted right before the event being handled was
published:
https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/db/securitygroups_db.py#281.

For now, I have pushed a test patch that fixes both tracebacks:
https://review.opendev.org/c/openstack/neutron/+/883047. I will report
the result of zuul in a follow up comment.

However, in revisiting
https://review.opendev.org/c/openstack/neutron/+/876716 and the
associated bug https://bugs.launchpad.net/neutron/+bug/2008712, a few
questions came to my mind:

1) In https://bugs.launchpad.net/neutron/+bug/2008712, it is reported
that the bug affected the OVS mechanism driver and not OVN's. Why did we
end up fixing OVN's? In fact, PS1 attempted to fix the bug for OVS here
https://review.opendev.org/c/openstack/neutron/+/876716/1/neutron/api/rpc/handlers/securitygroups_rpc.py.
Why was this approach abandoned?

2) It seems to me that in
https://review.opendev.org/c/openstack/neutron/+/876716 we violate the
semantics of the AFTER_DELETE event for security group rules. We delete
a security group and yet we publish a security group rule deletion
event. Even more, the event carries in its payload the deleted rule,
when this type of event normally doesn't carry the deleted resource,
because it was already deleted. In fact,
https://review.opendev.org/c/openstack/neutron/+/876716/1/neutron/api/rpc/handlers/securitygroups_rpc.py
tried to fix the bug processing a security group AFTER_DELETE event,
which seems logically consistent to me. Again, why did we abandon this
approach?

** Affects: neutron
     Importance: High
     Assignee: Miguel Lavalle (minsel)
         Status: Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2019449

Title:
  Neutron server log file full of tracebacks in neutron-tempest-plugin-
  ovn job

Status in neutron:
  Confirmed

Bug description:
  Since we merged
  https://review.opendev.org/c/openstack/neutron/+/876716, the Neutron
  server log in the neutron-tempest-plugin-ovn job is full of two types
  of tracebacks. The first type is

  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager [None req-f256cbd3-ea20-49e8-a9a0-0299fc7f34f3 tempest-SecGroupRulesQuotaTest-217177324 tempest-SecGroupRulesQuotaTest-217177324-project-member] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_after_del_notification-316503 security_group_rule, after_delete: KeyError: 'rule'
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 463, in _process_sg_rule_after_del_notification
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     sg_rule = payload.metadata['rule']
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager KeyError: 'rule'
  May 02 11:29:20.790754 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager 

  This happens because:

  1) The patch mentioned above now publishes AFTER_DELETE events for
  security group rules that include the rule deleted:
  https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/db/securitygroups_db.py#293.
  The same patch added an event handler that expects to find the deleted
  security group rule in the payload:
  https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py#459

  2) However, previous to this patch, we were already publishing
  AFTER_DELETE events for security group rules that don't include the
  security group rule:
  https://github.com/openstack/neutron/blob/47d4ec4e99d5aae62656c88206eb6a77f70d4a8b/neutron/db/securitygroups_db.py#L901-L908.
  As a consequence, the new handler added by
  https://review.opendev.org/c/openstack/neutron/+/876716 produces the
  KeyError exception shown above when trying to access the security
  group rule in the payload when processing the preexisting event.

  
  The second type of traceback is:

  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager [None req-4f41bf6e-d3c6-42a8-a987-b062a9a6b375 admin admin] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._process_sg_rule_after_del_notification-316503 security_group_rule, after_delete: neutron.extensions.securitygroup.SecurityGroupNotFound: Security group ae8107d1-0399-4491-8535-4aee2c9de5d0 does not exist
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group_rule(context, sg_rule)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2413, in delete_security_group_rule
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self._process_security_group_rule(rule, is_add_acl=False)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2401, in _process_security_group_rule
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ovn_acl.update_acls_for_security_group(
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     sg = plugin.get_security_group(admin_context, security_group_id)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 226, in wrapped
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f_with_retry(*args, **kwargs,
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     setattr(e, '_RETRY_EXCEEDED', True)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ectxt.value = e.inner_exc
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     context_reference.session.rollback()
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     return f(*dup_args, **dup_kwargs)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 209, in get_security_group
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     ret = self._make_security_group_dict(self._get_security_group(
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 226, in _get_security_group
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager     raise ext_sg.SecurityGroupNotFound(id=id)
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group ae8107d1-0399-4491-8535-4aee2c9de5d0 does not exist
  May 02 11:29:23.889104 np0033912622 neutron-server[100796]: ERROR neutron_lib.callbacks.manager

  This happens because the default security groups that we create for
  every project include two rules that point to the security group they
  belong to as the remote security group. As a consequence, when the
  event handler added by
  https://review.opendev.org/c/openstack/neutron/+/876716 tries to
  delete these type of rules, it doesn't find their security group,
  because it was deleted right before the event being handled was
  published:
  https://review.opendev.org/c/openstack/neutron/+/876716/5/neutron/db/securitygroups_db.py#281.

  For now, I have pushed a test patch that fixes both tracebacks:
  https://review.opendev.org/c/openstack/neutron/+/883047. I will report
  the result of zuul in a follow up comment.

  However, in revisiting
  https://review.opendev.org/c/openstack/neutron/+/876716 and the
  associated bug https://bugs.launchpad.net/neutron/+bug/2008712, a few
  questions came to my mind:

  1) In https://bugs.launchpad.net/neutron/+bug/2008712, it is reported
  that the bug affected the OVS mechanism driver and not OVN's. Why did
  we end up fixing OVN's? In fact, PS1 attempted to fix the bug for OVS
  here
  https://review.opendev.org/c/openstack/neutron/+/876716/1/neutron/api/rpc/handlers/securitygroups_rpc.py.
  Why was this approach abandoned?

  2) It seems to me that in
  https://review.opendev.org/c/openstack/neutron/+/876716 we violate the
  semantics of the AFTER_DELETE event for security group rules. We
  delete a security group and yet we publish a security group rule
  deletion event. Even more, the event carries in its payload the
  deleted rule, when this type of event normally doesn't carry the
  deleted resource, because it was already deleted. In fact,
  https://review.opendev.org/c/openstack/neutron/+/876716/1/neutron/api/rpc/handlers/securitygroups_rpc.py
  tried to fix the bug processing a security group AFTER_DELETE event,
  which seems logically consistent to me. Again, why did we abandon this
  approach?

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2019449/+subscriptions
Follow ups

[Bug 2019449] Re: Neutron server log file full of tracebacks in neutron-tempest-plugin-ovn job
From: OpenStack Infra, 2023-05-24