yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #88527
[Bug 1928031] Re: neutron-ovn-metadata-agent AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'
** Changed in: neutron (Ubuntu Hirsute)
Status: New => Won't Fix
** Changed in: openvswitch (Ubuntu Hirsute)
Status: New => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1928031
Title:
neutron-ovn-metadata-agent AttributeError: 'MetadataProxyHandler'
object has no attribute 'sb_idl'
Status in charm-ovn-chassis:
Invalid
Status in Ubuntu Cloud Archive:
New
Status in Ubuntu Cloud Archive ussuri series:
New
Status in Ubuntu Cloud Archive wallaby series:
New
Status in neutron:
Fix Released
Status in neutron package in Ubuntu:
Fix Released
Status in openvswitch package in Ubuntu:
Fix Released
Status in neutron source package in Focal:
New
Status in openvswitch source package in Focal:
New
Status in neutron source package in Hirsute:
Won't Fix
Status in openvswitch source package in Hirsute:
Won't Fix
Bug description:
neutron-ovn-metadata-agent not able to handle any metadata requests
from the instances.
Scenario:
* Initially there is some intermittent connectivity issues that are descirbed in LP #1907686
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/comments/9
* The fix for the above is available in python3-openvswitch package in ussuri-proposed pocket
Installed the fix on all neutron-server and compute and restarted neutron-ovn-metadata-agent one by one
* neutron-ovn-metadata-agent on one of the compute nodes not able to
handle any metadata requests after restart. ( Please note the problem
happened with only one ovn-metadata agent and rest of the agents are
good on other compute nodes, so this is some race condition in IDL)
Stacktrace shows both the workers 69188/69189 timed out on OVNSB IDL
connection and hence sb_idl is never initialized.
Stacktrace of Attribute error:
------------------------------
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server Traceback (most recent call last):
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 67, in __call__
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server instance_id, project_id = self._get_instance_and_project_id(req)
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 84, in _get_instance_and_project_id
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server ports = self.sb_idl.get_network_port_bindings_by_ip(network_id,
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'
2021-04-27 08:51:01.340 69188 ERROR neutron.agent.ovn.metadata.server
Stacktrace at the restart of neutron-ovn-metadata-agent process:
----------------------------------------------------------------
2021-04-15 22:27:03.803 69124 INFO neutron.common.config [-] /usr/bin/neutron-ovn-metadata-agent version 16.2.0
2021-04-15 22:27:03.832 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting...
2021-04-15 22:27:03.833 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected
2021-04-15 22:27:03.949 69124 WARNING neutron.agent.ovn.metadata.agent [-] Can't read ovn-bridge external-id from OVSDB. Using br-int instead.
2021-04-15 22:27:03.950 69124 INFO oslo_service.service [-] Starting 2 workers
2021-04-15 22:27:03.985 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
2021-04-15 22:27:03.986 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
2021-04-15 22:27:04.005 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connecting...
2021-04-15 22:27:04.006 69188 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
2021-04-15 22:27:04.033 69189 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
2021-04-15 22:27:04.061 69124 INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.216.241.118:6642: connected
2021-04-15 22:27:06.129 69124 INFO oslo.privsep.daemon [-] Running privsep helper: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/neutron_ovn_metadata_agent.ini', '--privsep_context', 'neutron.privileged.default', '--privsep_sock_path', '/tmp/tmpgncr2rq7/privsep.sock']
2021-04-15 22:27:06.757 69124 INFO oslo.privsep.daemon [-] Spawned new privsep daemon via rootwrap
2021-04-15 22:27:06.676 69211 INFO oslo.privsep.daemon [-] privsep daemon starting
2021-04-15 22:27:06.678 69211 INFO oslo.privsep.daemon [-] privsep process running with uid/gid: 0/0
2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN|CAP_SYS_PTRACE/none
2021-04-15 22:27:06.680 69211 INFO oslo.privsep.daemon [-] privsep daemon running as pid 69211
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager self.start_connection(connection)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager raise connection_exception
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.546 69188 INFO eventlet.wsgi.server [-] (69188) wsgi starting up on http:/var/lib/neutron/metadata_proxy
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
2021-04-15 22:30:08.550 69189 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.ovsdb_connection.start()
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Exception: Timeout
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager During handling of the above exception, another exception occurred:
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/server.py", line 60, in post_fork_initialize
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager tables=('Port_Binding', 'Datapath_Binding')).start()
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/ovsdb.py", line 57, in start
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager return impl_idl_ovn.OvsdbSbOvnIdl(conn)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 724, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvsdbSbOvnIdl, self).__init__(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_southbound/impl_idl.py", line 26, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(OvnSbApiIdlImpl, self).__init__(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 63, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager super(Backend, self).__init__(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 32, in __init__
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager self.start_connection(connection)
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 72, in start_connection
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager raise connection_exception
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.552 69189 ERROR neutron_lib.callbacks.manager
2021-04-15 22:30:08.555 69189 INFO eventlet.wsgi.server [-] (69189) wsgi starting up on http:/var/lib/neutron/metadata_proxy
* However netstat shows the TCP connection to SB IDL is established
for both the workers. So this could be some problem in data at IDL.
$ grep 10.216.241.118 sosreport-*-2021-04-27-edidnbl/sos_commands/networking/netstat_-W_-neopa
tcp 0 0 10.216.241.244:53284 10.216.241.118:6642 ESTABLISHED 0 407222779 69124/neutron-ovn-m off (0.00/0/0)
tcp 1490260 0 10.216.241.244:53282 10.216.241.118:6642 ESTABLISHED 0 407288937 69189/neutron-ovn-m off (0.00/0/0)
tcp 1490260 0 10.216.241.244:53280 10.216.241.118:6642 ESTABLISHED 0 407156544 69188/neutron-ovn-m off (0.00/0/0)
It will be good if sb_idl is not initialised, the worker thread can be
closed or neutron-ovn-metadata-agent should be stopped so that the
problem will be notified to operator at the earliest.
To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1928031/+subscriptions
