yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1963921] [NEW] Load balancer creation often failing due to logical switch not found

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Fernando Royo <1963921@xxxxxxxxxxxxxxxxxx>
Date: Mon, 07 Mar 2022 14:19:02 -0000
Reply-to: Bug 1963921 <1963921@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

It seems possible that a load-balancer creation was triggered while
multiple Subnets were being deleted, causing an exception of logical
switch not found and moving the load-balancer to ERROR state.

2022-02-10 14:48:49.115 16 ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last):
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver [-] Exception occurred during creation of loadbalancer: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield self._nested_txns_map[cur_thread_id]
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver KeyError: 139860903978752
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1033, in lb_create
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self._execute_commands(commands)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 626, in _execute_commands
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.add(command)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 252, in transaction
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield t
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     del self._nested_txns_map[cur_thread_id]
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self.result = self.commit()
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise result.ex
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.results.put(txn.do_commit())
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     command.run_idl(txn)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1159, in run_idl
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     ls = self.api.lookup('Logical_Switch', self.switch)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 172, in lookup
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     return self._lookup(table, record)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 215, in _lookup
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     row = idlutils.row_by_value(self, rl.table, rl.column, record)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 130, in row_by_value
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise RowNotFound(table=table, col=column, match=match)
2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a

** Affects: neutron
     Importance: Undecided
     Assignee: Fernando Royo (froyoredhat)
         Status: In Progress

** Changed in: neutron
     Assignee: (unassigned) => Fernando Royo (froyoredhat)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1963921

Title:
  Load balancer creation often failing due to logical switch not found

Status in neutron:
  In Progress

Bug description:
  It seems possible that a load-balancer creation was triggered while
  multiple Subnets were being deleted, causing an exception of logical
  switch not found and moving the load-balancer to ERROR state.

  2022-02-10 14:48:49.115 16 ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver [-] Exception occurred during creation of loadbalancer: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield self._nested_txns_map[cur_thread_id]
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver KeyError: 139860903978752
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1033, in lb_create
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self._execute_commands(commands)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 626, in _execute_commands
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.add(command)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 252, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield t
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     del self._nested_txns_map[cur_thread_id]
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self.result = self.commit()
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise result.ex
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.results.put(txn.do_commit())
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     command.run_idl(txn)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1159, in run_idl
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     ls = self.api.lookup('Logical_Switch', self.switch)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 172, in lookup
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     return self._lookup(table, record)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 215, in _lookup
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 130, in row_by_value
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise RowNotFound(table=table, col=column, match=match)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1963921/+subscriptions
Follow ups

[Bug 1963921] Re: Load balancer creation often failing due to logical switch not found
From: OpenStack Infra, 2022-03-15