yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1963921] Re: Load balancer creation often failing due to logical switch not found

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1963921@xxxxxxxxxxxxxxxxxx>
Date: Tue, 15 Mar 2022 11:21:12 -0000
Reply-to: Bug 1963921 <1963921@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/ovn-octavia-provider/+/829126
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/1a62902b5601c764da659f13242131828b4078ca
Submitter: "Zuul (22348)"
Branch:    master

commit 1a62902b5601c764da659f13242131828b4078ca
Author: Fernando Royo <froyo@xxxxxxxxxx>
Date:   Mon Feb 14 19:56:20 2022 +0100

    Retry logical switch associations to load balancers
    
    On load-balancer creation, or other operations related to association
    of the load balancer to the logical router, all logical switches
    associated with the logical router in the target network are also
    associated with the load-balancer.  When the topology includes multiple
    subnets, it may happen that the operation over the load-balancer match
    in time with the removal of some of the subnets.
    
    The creation of the load-balancer is a transactional atomic
    process. This patch is splitting such transactions and retrying
    in case of the above error. In case the attempts are exhausted and the
    error remains, we evaluate command by command, in case LsLbAdd or
    LsLbDel of an Ls associated to the Lr, we can omit the error and go
    forward.
    
    Closes-Bug: #1963921
    
    Change-Id: I2c7e0c677f0687990bbd71b3f8d511051dbe0359


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1963921

Title:
  Load balancer creation often failing due to logical switch not found

Status in neutron:
  Fix Released

Bug description:
  It seems possible that a load-balancer creation was triggered while
  multiple Subnets were being deleted, causing an exception of logical
  switch not found and moving the load-balancer to ERROR state.

  2022-02-10 14:48:49.115 16 ERROR ovsdbapp.backend.ovs_idl.transaction [-] Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver [-] Exception occurred during creation of loadbalancer: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield self._nested_txns_map[cur_thread_id]
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver KeyError: 139860903978752
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver During handling of the above exception, another exception occurred:
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver 
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver Traceback (most recent call last):
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 1033, in lb_create
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self._execute_commands(commands)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/octavia/ovn_driver.py", line 626, in _execute_commands
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.add(command)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/networking_ovn/ovsdb/impl_idl_ovn.py", line 252, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     yield t
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     next(self.gen)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     del self._nested_txns_map[cur_thread_id]
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     self.result = self.commit()
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise result.ex
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 128, in run
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     txn.results.put(txn.do_commit())
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 86, in do_commit
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     command.run_idl(txn)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 1159, in run_idl
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     ls = self.api.lookup('Logical_Switch', self.switch)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 172, in lookup
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     return self._lookup(table, record)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 215, in _lookup
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     row = idlutils.row_by_value(self, rl.table, rl.column, record)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver   File "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 130, in row_by_value
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver     raise RowNotFound(table=table, col=column, match=match)
  2022-02-10 14:48:49.115 16 ERROR networking_ovn.octavia.ovn_driver ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Switch with name=neutron-6fa06cae-0145-4571-9919-0541f0bea93a

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1963921/+subscriptions
References

[Bug 1963921] [NEW] Load balancer creation often failing due to logical switch not found
From: Fernando Royo, 2022-03-07