yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1894136] Re: [OVN Octavia Provider] OVN provider fails during listener delete

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1894136@xxxxxxxxxxxxxxxxxx>
Date: Mon, 19 Oct 2020 09:24:21 -0000
Reply-to: Bug 1894136 <1894136@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/758472
Committed: https://git.openstack.org/cgit/openstack/ovn-octavia-provider/commit/?id=5b0715d96700d1c288f601b3ecb2cad507c2c134
Submitter: Zuul
Branch:    master

commit 5b0715d96700d1c288f601b3ecb2cad507c2c134
Author: Terry Wilson <twilson@xxxxxxxxxx>
Date:   Thu Oct 15 12:53:36 2020 -0500

    Fix leaked file descriptors by cleaning up objects
    
    A reference to the ovsdbapp Connection object is stored on the the
    API impl class and it is causing the object to stick around and
    its pipe fds to never close. Ultimately, python-ovs's call to
    select() is limited to 1024 file descriptors and will eventually
    error out. This deletes the connection and api objects explicitly
    when we are done with them.
    
    Change-Id: I97b27eaa293fb161724d34da88b4398f8b590c33
    Co-authored-by: Brian Haley <bhaley@xxxxxxxxxx>
    Closes-Bug: #1894136


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1894136

Title:
   [OVN Octavia Provider] OVN provider fails during listener delete

Status in neutron:
  Fix Released

Bug description:
  The OVN provider is consistently failing during a listener delete as
  part of the member API tempest test tear down with a 'filedescriptor
  out of range in select()' error.

  o-api logs snippet:

  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last):
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     ovs_poller.block()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     events = self.poll.poll(self.timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn 
  Sep 03 15:44:05.172746 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR octavia.api.drivers.driver_factory [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR wsme.api [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Server-side error: "Provider 'ovn' was not found.". Detail:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovsdb_connection.start()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     idlutils.wait_for_change(self.idl, self.timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     ovs_poller.block()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     events = self.poll.poll(self.timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 44, in get_driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     invoke_on_load=True).driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/driver.py", line 61, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     warn_on_missing_entrypoint=warn_on_missing_entrypoint
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 81, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 233, in _load_plugins
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._on_load_failure_callback(self, ep, err)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 225, in _load_plugins
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 257, in _load_one_plugin
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     obj = plugin(*invoke_args, **invoke_kwds)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/driver.py", line 42, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._ovn_helper = ovn_helper.OvnProviderHelper()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/helper.py", line 60, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovn_nbdb_api = self.ovn_nbdb.start()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 174, in start
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     return impl_idl_ovn.OvsdbNbOvnIdl(self.conn)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 118, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(OvsdbNbOvnIdl, self).__init__(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 57, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(Backend, self).__init__(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 35, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.start_connection(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 66, in start_connection
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise connection_exception
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/wsmeext/pecan.py", line 84, in callfunction
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     result = f(self, *args, **kwargs)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/v2/controllers/listener.py", line 632, in delete
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     driver = driver_factory.get_driver(provider)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 49, in get_driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise exceptions.ProviderNotFound(prov=provider)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
  Sep 03 15:44:05.178879 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: [pid: 10089|app: 0|req: 1109/2281] 198.72.124.16 () {56 vars in 1234 bytes} [Thu Sep  3 15:44:04 2020] DELETE /load-balancer/v2.0/lbaas/listeners/3537f411-5ab3-48c8-9669-c9fa1f40fee9 => generated 90 bytes in 447 msecs (HTTP/1.1 501) 4 headers in 169 bytes (1 switches on core 0)

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1894136/+subscriptions
References

[Bug 1894136] [NEW] [OVN Octavia Provider] OVN provider fails during listener delete
From: Michael Johnson, 2020-09-03