yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1894136@xxxxxxxxxxxxxxxxxx>]

You have been subscribed to a public bug:

The OVN provider is consistently failing during a listener delete as
part of the member API tempest test tear down with a 'filedescriptor out
of range in select()' error.

o-api logs snippet:

Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ValueError: filedescriptor out of range in select()
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last):
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     ovs_poller.block()
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     events = self.poll.poll(self.timeout)
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     timeout)
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ValueError: filedescriptor out of range in select()
Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn 
Sep 03 15:44:05.172746 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR octavia.api.drivers.driver_factory [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR wsme.api [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Server-side error: "Provider 'ovn' was not found.". Detail:
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovsdb_connection.start()
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     idlutils.wait_for_change(self.idl, self.timeout)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     ovs_poller.block()
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     events = self.poll.poll(self.timeout)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     timeout)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ValueError: filedescriptor out of range in select()
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 44, in get_driver
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     invoke_on_load=True).driver
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/driver.py", line 61, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     warn_on_missing_entrypoint=warn_on_missing_entrypoint
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 81, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 233, in _load_plugins
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._on_load_failure_callback(self, ep, err)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 225, in _load_plugins
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 257, in _load_one_plugin
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     obj = plugin(*invoke_args, **invoke_kwds)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/driver.py", line 42, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._ovn_helper = ovn_helper.OvnProviderHelper()
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/helper.py", line 60, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovn_nbdb_api = self.ovn_nbdb.start()
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 174, in start
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     return impl_idl_ovn.OvsdbNbOvnIdl(self.conn)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 118, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(OvsdbNbOvnIdl, self).__init__(connection)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 57, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(Backend, self).__init__(connection)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 35, in __init__
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.start_connection(connection)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 66, in start_connection
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise connection_exception
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/wsmeext/pecan.py", line 84, in callfunction
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     result = f(self, *args, **kwargs)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/v2/controllers/listener.py", line 632, in delete
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     driver = driver_factory.get_driver(provider)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 49, in get_driver
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise exceptions.ProviderNotFound(prov=provider)
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
Sep 03 15:44:05.178879 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: [pid: 10089|app: 0|req: 1109/2281] 198.72.124.16 () {56 vars in 1234 bytes} [Thu Sep  3 15:44:04 2020] DELETE /load-balancer/v2.0/lbaas/listeners/3537f411-5ab3-48c8-9669-c9fa1f40fee9 => generated 90 bytes in 447 msecs (HTTP/1.1 501) 4 headers in 169 bytes (1 switches on core 0)

** Affects: neutron
     Importance: High
     Assignee: Brian Haley (brian-haley)
         Status: In Progress


** Tags: ovn-octavia-provider
-- 
 [OVN Octavia Provider] OVN provider fails during listener delete
https://bugs.launchpad.net/bugs/1894136
You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron.