← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #92401

[Bug 2020437] [NEW] Deletion of SG caiuses stacktrace in the neutron server logs

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

When ML2/OVN backend is used there is a lot of errors like:

May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group_rule(context, sg_rule)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2414, in delete_security_group_rule
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self._process_security_group_rule(rule, is_add_acl=False)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2402, in _process_security_group_rule
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     ovn_acl.update_acls_for_security_group(
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     sg = plugin.get_security_group(admin_context, security_group_id)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f_with_retry(*args, **kwargs,
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception():
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception() as ectxt:
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception():
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*dup_args, **dup_kwargs)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 214, in get_security_group
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     ret = self._make_security_group_dict(self._get_security_group(
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 231, in _get_security_group
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise ext_sg.SecurityGroupNotFound(id=id)
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group caeb2144-af8c-4965-aa3b-3c93576f67a6 does not exist
May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager 
May 18 17:48:46.908377 np0034069340 neutron-server[68119]: DEBUG neutron_lib.callbacks.manager [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] Publish callbacks [] for security_group (None), before_response {{(pid=68119) _notify_loop /usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py:176}}
May 18 17:48:46.909953 np0034069340 neutron-server[68119]: INFO neutron.wsgi [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] 10.209.39.11,10.209.39.11 "DELETE /networking/v2.0/security-groups/caeb2144-af8c-4965-aa3b-3c93576f67a6 HTTP/1.1" status: 204  len: 173 time: 0.1885200

In the logs. For example in
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5b3/883246/5/check/tempest-
integrated-networking/5b39a98/controller/logs/screen-q-svc.txt but it's
also in other jobs as well.

** Affects: neutron
     Importance: Low
         Status: Confirmed


** Tags: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2020437

Title:
  Deletion of SG caiuses stacktrace in the neutron server logs

Status in neutron:
  Confirmed

Bug description:
  When ML2/OVN backend is used there is a lot of errors like:

  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 464, in _process_sg_rule_after_del_notification
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self._ovn_client.delete_security_group_rule(context, sg_rule)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2414, in delete_security_group_rule
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self._process_security_group_rule(rule, is_add_acl=False)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2402, in _process_security_group_rule
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     ovn_acl.update_acls_for_security_group(
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/common/ovn/acl.py", line 294, in update_acls_for_security_group
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     sg = plugin.get_security_group(admin_context, security_group_id)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 223, in wrapped
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f_with_retry(*args, **kwargs,
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 137, in wrapped
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception():
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 144, in wrapper
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception() as ectxt:
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*args, **kwargs)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     with excutils.save_and_reraise_exception():
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     self.force_reraise()
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise self.value
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 181, in wrapped
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     return f(*dup_args, **dup_kwargs)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 214, in get_security_group
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     ret = self._make_security_group_dict(self._get_security_group(
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/db/securitygroups_db.py", line 231, in _get_security_group
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager     raise ext_sg.SecurityGroupNotFound(id=id)
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager neutron.extensions.securitygroup.SecurityGroupNotFound: Security group caeb2144-af8c-4965-aa3b-3c93576f67a6 does not exist
  May 18 17:48:46.904556 np0034069340 neutron-server[68119]: ERROR neutron_lib.callbacks.manager 
  May 18 17:48:46.908377 np0034069340 neutron-server[68119]: DEBUG neutron_lib.callbacks.manager [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] Publish callbacks [] for security_group (None), before_response {{(pid=68119) _notify_loop /usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py:176}}
  May 18 17:48:46.909953 np0034069340 neutron-server[68119]: INFO neutron.wsgi [None req-f2eab262-f911-4632-b8be-f3914ead40b1 admin admin] 10.209.39.11,10.209.39.11 "DELETE /networking/v2.0/security-groups/caeb2144-af8c-4965-aa3b-3c93576f67a6 HTTP/1.1" status: 204  len: 173 time: 0.1885200

  In the logs. For example in
  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5b3/883246/5/check/tempest-
  integrated-networking/5b39a98/controller/logs/screen-q-svc.txt but
  it's also in other jobs as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2020437/+subscriptions
  Thread Previous • Date Previous • • Thread Next