yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1928031] Re: neutron-ovn-metadata-agent AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: James Page <1928031@xxxxxxxxxxxxxxxxxx>
Date: Fri, 07 Jan 2022 10:36:37 -0000
Reply-to: Bug 1928031 <1928031@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Changed in: openvswitch (Ubuntu)
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1928031

Title:
  neutron-ovn-metadata-agent AttributeError: 'MetadataProxyHandler'
  object has no attribute 'sb_idl'

Status in charm-ovn-chassis:
  Invalid
Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive ussuri series:
  New
Status in Ubuntu Cloud Archive wallaby series:
  New
Status in neutron:
  Fix Released
Status in neutron package in Ubuntu:
  Fix Released
Status in openvswitch package in Ubuntu:
  Fix Released
Status in neutron source package in Focal:
  New
Status in openvswitch source package in Focal:
  New

Bug description:
  neutron-ovn-metadata-agent not able to handle any metadata requests
  from the instances.

  Scenario:
  * Initially there is some intermittent connectivity issues that are descirbed in LP #1907686
    https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9

  * The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket
    Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one

  * neutron-ovn-metadata-agent on one of the compute nodes not able to
  handle any metadata requests after restart. ( Please note the problem
  happened with only one ovn-metadata agent and rest of the agents are
  good on other compute nodes, so this is some race condition in IDL)

  Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL
  connection and hence sb_idl is never initialized.

  Stacktrace of Attribute error:
  ------------------------------
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last):
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server     instance_id, project_id = self._get_instance_and_project_id(req)
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server     ports = self.sb_idl.get_network_port_bindings_by_ip(network_id,
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'
  2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server

  Stacktrace at the restart of neutron-ovn-metadata-agent process:
  ----------------------------------------------------------------
  2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0
  2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting...
  2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected
  2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead.
  2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers
  2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
  2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
  2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
  2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
  2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
  2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
  2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock']
  2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
  2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting
  2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
  2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none
  2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
  2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     self.ovsdb_connection.start()
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     idlutils.wait_for_change(self.idl, self.timeout)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     tables=('Port_Binding', 'Datapath_Binding')).start()
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     return impl_idl_ovn.OvsdbSbOvnIdl(conn)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(OvsdbSbOvnIdl, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(OvnSbApiIdlImpl, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(Backend, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     self.start_connection(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     raise connection_exception
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     tables=('Port_Binding', 'Datapath_Binding')).start()
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     return impl_idl_ovn.OvsdbSbOvnIdl(conn)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(OvsdbSbOvnIdl, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(OvnSbApiIdlImpl, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     super(Backend, self).__init__(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     self.start_connection(connection)
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     raise connection_exception
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
  2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     self.ovsdb_connection.start()
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     idlutils.wait_for_change(self.idl, self.timeout)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     tables=('Port_Binding', 'Datapath_Binding')).start()
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     return impl_idl_ovn.OvsdbSbOvnIdl(conn)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     super(OvsdbSbOvnIdl, self).__init__(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     super(OvnSbApiIdlImpl, self).__init__(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     super(OvsdbSbOvnIdl, self).__init__(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     super(OvnSbApiIdlImpl, self).__init__(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     super(Backend, self).__init__(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     self.start_connection(connection)
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager     raise connection_exception
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
  2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
  2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy

  * However netstat shows the TCP connection to SB IDL is established
  for both the workers. So this could be some problem in data at IDL.

  $ grep 10.216.241.118 sosreport-*-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa
  tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0)
  tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0)
  tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0)

  It will be good if sb_idl is not initialised, the worker thread can be
  closed or neutron-ovn-metadata-agent should be stopped so that the
  problem will be notified to operator at the earliest.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1928031/+subscriptions