yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1927077] [NEW] [OVN] Missing lock check in check_for_mcast_flood_reports

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lucas Alvares Gomes <1927077@xxxxxxxxxxxxxxxxxx>
Date: Tue, 04 May 2021 10:32:24 -0000
Reply-to: Bug 1927077 <1927077@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

The maintenance method check_for_mcast_flood_reports() is not checking
if that instance of the maintenance task has a proper lock before trying
to execute a transaction. If more than one controller is deployed only 1
instance will hold a valid lock the rest will fail with:

2021-05-02 14:23:29.193 599 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics KeyError: 139844066981736
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics During handling of the above exception, another exception occurred:
2021-05-02 14:23:29.193 599 ERROR futurist.periodics
2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     work()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return self.callback(*self.args, **self.kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return f(*args, **kwargs)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     for port in self._nb_idl.lsp_list().execute(check_error=True):
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     t.add(self)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 196, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield t
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     del self._nested_txns_map[cur_thread_id]
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     self.result = self.commit()
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise result.ex
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     txn.results.put(txn.do_commit())
2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise RuntimeError(msg)
2021-05-02 14:23:29.193 599 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2021-05-02 14:23:29.193 599 ERROR futurist.periodics

** Affects: neutron
     Importance: Medium
     Assignee: Lucas Alvares Gomes (lucasagomes)
         Status: In Progress


** Tags: ovn

** Changed in: neutron
     Assignee: (unassigned) => Lucas Alvares Gomes (lucasagomes)

** Changed in: neutron
   Importance: Undecided => Medium

** Changed in: neutron
       Status: New => Confirmed

** Tags added: ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1927077

Title:
  [OVN] Missing lock check in check_for_mcast_flood_reports

Status in neutron:
  In Progress

Bug description:
  The maintenance method check_for_mcast_flood_reports() is not checking
  if that instance of the maintenance task has a proper lock before
  trying to execute a transaction. If more than one controller is
  deployed only 1 instance will hold a valid lock the rest will fail
  with:

  2021-05-02 14:23:29.193 599 ERROR futurist.periodics [-] Failed to call periodic 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_mcast_flood_reports' (it runs every 600.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield self._nested_txns_map[cur_thread_id]
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics KeyError: 139844066981736
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics During handling of the above exception, another exception occurred:
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics Traceback (most recent call last):
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 290, in run
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     work()
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 64, in __call__
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return self.callback(*self.args, **self.kwargs)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/futurist/periodics.py", line 178, in decorator
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     return f(*args, **kwargs)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 667, in check_for_mcast_flood_reports
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     for port in self._nb_idl.lsp_list().execute(check_error=True):
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     t.add(self)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 196, in transaction
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     yield t
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     next(self.gen)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     del self._nested_txns_map[cur_thread_id]
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     self.result = self.commit()
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise result.ex
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     txn.results.put(txn.do_commit())
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics     raise RuntimeError(msg)
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
  2021-05-02 14:23:29.193 599 ERROR futurist.periodics

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1927077/+subscriptions
Follow ups

[Bug 1927077] Re: [OVN] Missing lock check in check_for_mcast_flood_reports
From: OpenStack Infra, 2021-05-05