yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2063043] Re: octavia/ovn: filedescriptor out of range in select()

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gregory Thiemonge <2063043@xxxxxxxxxxxxxxxxxx>
Date: Mon, 22 Apr 2024 08:17:01 -0000
Reply-to: Bug 2063043 <2063043@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
moving to neutron, the ovn-octavia-provider is a neutron project

** Project changed: octavia => neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2063043

Title:
  octavia/ovn: filedescriptor out of range in select()

Status in neutron:
  New

Bug description:
  Running octavia-api in a container (kolla-ansible), where a health monitoring process creates OVN provider loadbalancers with listeners, pools, members, health-mons and tests them and cleans them up again.
  Upon creation of resources, the octavia-api process creates `[eventpoll]` file descriptors (with `epoll_create1()` system call which it never closes. Once it hits 1024, it errors out with
  ```
   2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [None req-eefd1e9d-8bfe-473d-9ff1-5a8b0d4ab5d2 - 2767ef0256804b92ae0e51e3a99f809a - - 93db18cb391748009639058c52577527 93db18cb391748009639058c52577527] OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ValueError: filedescriptor out of range in select()
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last):
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.10/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 65, in start_connection
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 252, in wait_for_change
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     ovs_poller.block()
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/poller.py", line 231, in block
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     events = self.poll.poll(self.timeout)
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovs/poller.py", line 137, in poll
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     rlist, wlist, xlist = select.select(self.rlist,
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn ValueError: filedescriptor out of range in select()
  2024-04-09 03:00:12.130 732 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
  2024-04-09 03:00:12.132 732 ERROR octavia.api.drivers.driver_factory [None req-eefd1e9d-8bfe-473d-9ff1-5a8b0d4ab5d2 - 2767ef0256804b92ae0e51e3a99f809a - - 93db18cb391748009639058c52577527 93db18cb391748009639058c52577527] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'filedescriptor out of range in select()'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2024-04-09 03:00:12.134 732 ERROR wsme.api [None req-eefd1e9d-8bfe-473d-9ff1-5a8b0d4ab5d2 - 2767ef0256804b92ae0e51e3a99f809a - - 93db18cb391748009639058c52577527 93db18cb391748009639058c52577527] Server-side error: "Provider 'ovn' was not found.". Detail:
  Traceback (most recent call last):
  [...]
  ```
  Subsequently, the ovn provider no longer is registered from an octavia-api perspective and the container goes into an unhealthy state and needs to be restarted.

  This was observed on octavia from OpenStack 2023.2 (Bobcat) installed
  via kolla-ansible (OSISM).

  Original bug report at
  https://github.com/osism/issues/issues/959

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2063043/+subscriptions