yahoo-eng-team team mailing list archive

[OpenStack Infra <1961874@xxxxxxxxxxxxxxxxxx>]

This issue was fixed in the openstack/kolla 12.2.0  release.

** Changed in: kolla/wallaby
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1961874

Title:
  [ovn-octavia-provider] OVN NB DB timeouts

Status in kolla:
  Fix Released
Status in kolla victoria series:
  Fix Released
Status in kolla wallaby series:
  Fix Released
Status in kolla xena series:
  Fix Released
Status in kolla yoga series:
  Fix Released
Status in neutron:
  Invalid

Bug description:
  Hello,

  During adding members to an LB/pool - following error shows up after
  180 seconds (ovsdb_timeout)

  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Traceback (most recent call last):
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 62, in start_connection
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 84, in start
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn   File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 244, in wait_for_change
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn Exception: Timeout
  2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
  2022-02-22 17:00:22.792 24 ERROR octavia.api.drivers.driver_factory [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Unable to load provider driver ovn due to: OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2022-02-22 17:00:22.794 24 ERROR wsme.api [req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 - d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Server-side error: "Provider 'ovn' was not found.". Detail:
  Traceback (most recent call last):

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 62, in start_connection
      self.ovsdb_connection.start()

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 84, in start
      idlutils.wait_for_change(self.idl, self.timeout)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 244, in wait_for_change
      raise Exception("Timeout")  # TODO(twilson) use TimeoutException?

  Exception: Timeout

  The above exception was the direct cause of the following exception:

  Traceback (most recent call last):

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py", line 44, in get_driver
      invoke_on_load=True).driver

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/driver.py", line 62, in __init__
      warn_on_missing_entrypoint=warn_on_missing_entrypoint

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py", line 81, in __init__
      verify_requirements)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 233, in _load_plugins
      self._on_load_failure_callback(self, ep, err)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 225, in _load_plugins
      verify_requirements,

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py", line 158, in _load_one_plugin
      verify_requirements,

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line 257, in _load_one_plugin
      obj = plugin(*invoke_args, **invoke_kwds)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/driver.py", line 42, in __init__
      self._ovn_helper = ovn_helper.OvnProviderHelper()

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/helper.py", line 60, in __init__
      self.ovn_nbdb_api = self.ovn_nbdb.start()

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 176, in start
      return impl_idl_ovn.OvsdbNbOvnIdl(self.conn)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 119, in __init__
      super().__init__(connection)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 58, in __init__
      super().__init__(connection)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 60, in __init__
      self.start_connection(connection)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 67, in start_connection
      raise connection_exception from e

  ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable:
  OVS database connection to OVN_Northbound failed with error:
  'Timeout'. Verify that the OVS and OVN services are available and that
  the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options
  are correct.

  During handling of the above exception, another exception occurred:

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/wsmeext/pecan.py", line 85, in callfunction
      result = f(self, *args, **kwargs)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/v2/controllers/pool.py", line 250, in post
      driver = driver_factory.get_driver(provider)

    File "/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py", line 49, in get_driver
      raise exceptions.ProviderNotFound(prov=provider)

  octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.
  : octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.

  In the same moment - Neutron works fine (creating networks, ports,
  etc)

  OVN version: 21.06 (ovn-2021-21.06.0-29.el8s.x86_64)
  OVS version: 2.16 (openvswitch2.16-2.16.0-31.el8s.x86_64)
  octavia-ovn-provider version: Wallaby (1.0.1dev16)

To manage notifications about this bug go to:
https://bugs.launchpad.net/kolla/+bug/1961874/+subscriptions