yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1971569] Re: [neutron][api] test_log_deleted_with_corresponding_security_group failing randomly

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1971569@xxxxxxxxxxxxxxxxxx>
Date: Tue, 17 May 2022 13:33:19 -0000
Reply-to: Bug 1971569 <1971569@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/neutron/+/841094
Committed: https://opendev.org/openstack/neutron/commit/1471f531b86eef8928756d5e927f710a5683e6ad
Submitter: "Zuul (22348)"
Branch:    master

commit 1471f531b86eef8928756d5e927f710a5683e6ad
Author: yatinkarel <ykarel@xxxxxxxxxx>
Date:   Mon May 9 15:34:49 2022 +0530

    [OVN] Handle missing acls during log removal
    
    During log delete we fetch all the acls for
    all the pgs if there is only one log object
    and do clear log of all these acls, but if
    one or more acls of any of the pgs is removed
    concurrently, db_set fails as acl is not found.
    
    This patch proposes to only do log clear of those
    acls which are available and add log message for the
    acls which were deleted concurrently.
    
    Also add a unit test for this case where one of
    the acl get's missing.
    
    Closes-Bug: #1971569
    Change-Id: I58487024c8d0352776307f0185f0812bb00036ae


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1971569

Title:
  [neutron][api] test_log_deleted_with_corresponding_security_group
  failing randomly

Status in neutron:
  Fix Released

Bug description:
  test_log_deleted_with_corresponding_security_group api tests randomly
  failing with below Trace:-

  Traceback (most recent call last):
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/api/admin/test_logging.py", line 99, in test_log_deleted_with_corresponding_security_group
      self.assertRaises(exceptions.NotFound,
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 467, in assertRaises
      self.assertThat(our_callable, matcher)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/testtools/testcase.py", line 480, in assertThat
      raise mismatch_error
  testtools.matchers._impl.MismatchError: <function NetworkClientJSON._shower.<locals>._show at 0x7f784dd443a0> returned {'log': {'id': '4360d06d-7bd2-42e8-9468-2e760b1b246a', 'name': 'tempest-test-log-1596006254', 'resource_type': 'security_group', 'resource_id': '1df2df30-4901-4e41-b58f-bbef3b0c48ee', 'target_id': None, 'event': 'ALL', 'enabled': True, 'revision_number': 0, 'description': '', 'created_at': '2022-05-04T10:17:44Z', 'updated_at': '2022-05-04T10:17:44Z'}}

  
  Example logs:-
      https://c1d005e52a45f1c42de5-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf1.rackcdn.com/839477/1/check/neutron-tempest-plugin-api/825f39d/testr_results.html
      https://9a2b40b9abe499183777-8788f9c43324a469e03ac2bb48dfb234.ssl.cf2.rackcdn.com/837301/1/check/neutron-tempest-plugin-api-yoga/f0c7ad9/testr_results.html
      https://5fb6cfbc0c5c1a5be079-29e820dbac3fa779e4aa716d6c5c5850.ssl.cf1.rackcdn.com/839477/2/gate/neutron-tempest-plugin-api/929ff45/testr_results.html


  From neutron api:

  May 04 10:17:44.536866 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-69db86bc-5bbd-4b78-a6bc-00e84d0f5e88 None None] Running txn n=1 command(idx=14): DbSetCommand(table=ACL, record=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8, col_values=(('log', False), ('meter', []), ('name', []), ('severity', []))) {{(pid=98372) do_commit /usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py:90}}
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Traceback (most recent call last):
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:     txn.results.put(txn.do_commit())
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:     command.run_idl(txn)
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 139, in run_idl
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:     record = self.api.lookup(self.table, self.record)
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:     return self._lookup(table, record)
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 200, in _lookup
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]:     raise idlutils.RowNotFound(table=table, col='uuid',
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find ACL with uuid=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8
  May 04 10:17:44.541908 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: 
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Extension driver 'ovn' failed in delete_log: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find ACL with uuid=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager Traceback (most recent call last):
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/opt/stack/neutron/neutron/services/logapi/drivers/manager.py", line 116, in call
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     getattr(driver, method_name)(*args, **kwargs)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/opt/stack/neutron/neutron/services/logapi/drivers/ovn/driver.py", line 331, in delete_log
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     ovn_txn.add(self.ovn_nb.meter_del(self.meter_name,
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     next(self.gen)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 269, in transaction
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     yield t
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     next(self.gen)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/api.py", line 110, in transaction
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     del self._nested_txns_map[cur_thread_id]
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/api.py", line 61, in __exit__
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     self.result = self.commit()
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     raise result.ex
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     txn.results.put(txn.do_commit())
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     command.run_idl(txn)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 139, in run_idl
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     record = self.api.lookup(self.table, self.record)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 181, in lookup
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     return self._lookup(table, record)
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager   File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 200, in _lookup
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager     raise idlutils.RowNotFound(table=table, col='uuid',
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find ACL with uuid=8f71b6eb-d4cf-4e64-8bbb-709c99dc3ad8
  May 04 10:17:44.544980 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron.services.logapi.drivers.manager 
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Error during notification for neutron.services.logapi.logging_plugin.LoggingPlugin._clean_logs_by_resource_id-70942 security_group, after_delete: neutron_lib.exceptions.DriverCallError
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, payload=payload)
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/services/logapi/logging_plugin.py", line 63, in _clean_logs_by_resource_id
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager     self._clean_logs(payload.context.elevated(), sg_id=payload.resource_id)
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/services/logapi/logging_plugin.py", line 58, in _clean_logs
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager     self.delete_log(context, log['id'])
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/services/logapi/logging_plugin.py", line 136, in delete_log
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager     self.driver_manager.call(
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/services/logapi/drivers/manager.py", line 125, in call
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager     raise exceptions.DriverCallError(exc_list=exc_list)
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager neutron_lib.exceptions.DriverCallError
  May 04 10:17:44.550840 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: ERROR neutron_lib.callbacks.manager 
  May 04 10:17:44.552939 ubuntu-focal-rax-ord-0029540090 neutron-server[98372]: DEBUG neutron_lib.callbacks.manager [None req-d1699144-a926-431c-8202-470b285baa8a tempest-LoggingTestJSON-1873065189 tempest-LoggingTestJSON-1873065189-project] Publish callbacks [] for security_group (None), before_response {{(pid=98372) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}}

  Fails at:-
  https://opendev.org/openstack/neutron/src/branch/master/neutron/services/logapi/drivers/ovn/driver.py#L330

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1971569/+subscriptions
References

[Bug 1971569] [NEW] [neutron][api] test_log_deleted_with_corresponding_security_group failing randomly
From: yatin, 2022-05-04