yahoo-eng-team team mailing list archive

[Terry Wilson <1894136@xxxxxxxxxxxxxxxxxx>]

** Project changed: neutron => ovsdbapp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1894136

Title:
   [OVN Octavia Provider] OVN provider fails during listener delete

Status in ovsdbapp:
  In Progress

Bug description:
  The OVN provider is consistently failing during a listener delete as
  part of the member API tempest test tear down with a 'filedescriptor
  out of range in select()' error.

  o-api logs snippet:

  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last):
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     ovs_poller.block()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     events = self.poll.poll(self.timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     timeout)
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.171297 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn 
  Sep 03 15:44:05.172746 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR octavia.api.drivers.driver_factory [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ERROR wsme.api [None req-9201aee8-9a5b-460c-bf8b-c6408d20aec7 tempest-MemberAPITest-903346660 tempest-MemberAPITest-903346660] Server-side error: "Provider 'ovn' was not found.". Detail:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 61, in start_connection
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovsdb_connection.start()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     idlutils.wait_for_change(self.idl, self.timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 201, in wait_for_change
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     ovs_poller.block()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 231, in block
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     events = self.poll.poll(self.timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovs/poller.py", line 140, in poll
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     timeout)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ValueError: filedescriptor out of range in select()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 44, in get_driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     invoke_on_load=True).driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/driver.py", line 61, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     warn_on_missing_entrypoint=warn_on_missing_entrypoint
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 81, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 233, in _load_plugins
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._on_load_failure_callback(self, ep, err)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 225, in _load_plugins
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/named.py", line 158, in _load_one_plugin
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     verify_requirements,
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/stevedore/extension.py", line 257, in _load_one_plugin
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     obj = plugin(*invoke_args, **invoke_kwds)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/driver.py", line 42, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self._ovn_helper = ovn_helper.OvnProviderHelper()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/helper.py", line 60, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.ovn_nbdb_api = self.ovn_nbdb.start()
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 174, in start
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     return impl_idl_ovn.OvsdbNbOvnIdl(self.conn)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 118, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(OvsdbNbOvnIdl, self).__init__(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 57, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     super(Backend, self).__init__(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 35, in __init__
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     self.start_connection(connection)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 66, in start_connection
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise connection_exception
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: During handling of the above exception, another exception occurred:
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: Traceback (most recent call last):
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/usr/local/lib/python3.6/dist-packages/wsmeext/pecan.py", line 84, in callfunction
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     result = f(self, *args, **kwargs)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/v2/controllers/listener.py", line 632, in delete
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     driver = driver_factory.get_driver(provider)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:   File "/opt/stack/octavia/octavia/api/drivers/driver_factory.py", line 49, in get_driver
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]:     raise exceptions.ProviderNotFound(prov=provider)
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
  Sep 03 15:44:05.175074 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
  Sep 03 15:44:05.178879 ubuntu-bionic-inap-mtl01-0019624710 devstack@o-api.service[10087]: [pid: 10089|app: 0|req: 1109/2281] 198.72.124.16 () {56 vars in 1234 bytes} [Thu Sep  3 15:44:04 2020] DELETE /load-balancer/v2.0/lbaas/listeners/3537f411-5ab3-48c8-9669-c9fa1f40fee9 => generated 90 bytes in 447 msecs (HTTP/1.1 501) 4 headers in 169 bytes (1 switches on core 0)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ovsdbapp/+bug/1894136/+subscriptions