yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1911128] [NEW] Neutron with ovn driver failed to start on Fedora

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1911128@xxxxxxxxxxxxxxxxxx>
Date: Tue, 12 Jan 2021 09:02:35 -0000
Reply-to: Bug 1911128 <1911128@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Our periodic job neutron-ovn-tempest-ovs-master-fedora is failing since
some time 100% of times. It fails due to failed neutron-server start.
Error in the neutron logs is like below:

Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship QosNetworkPolicyBinding.port to refer to viewonly relationship Port.qos_network_policy_binding should include sync_backref=False set on the QosNetworkPolicyBinding.port relationship.  (this warning may be suppressed after 10 occurrences)
Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]:   util.warn_limited(
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     self._wait_for_pg_drop_event()
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     idl = ovsdb_monitor.OvnInitPGNbIdl.from_server(
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     helper = idlutils.get_schema_helper(connection_string, schema_name)
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     raise Exception("Could not retrieve schema from %s" % connection)
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641
Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager 
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     self._wait_for_pg_drop_event()
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     idl = ovsdb_monitor.OvnInitPGNbIdl.from_server(
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     helper = idlutils.get_schema_helper(connection_string, schema_name)
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     raise Exception("Could not retrieve schema from %s" % connection)
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641
Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager 
Jan 11 06:30:42.033666 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi starting up on http://0.0.0.0:19696
Jan 11 06:30:42.034612 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi starting up on http://0.0.0.0:19696
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: CRITICAL neutron [None req-4c1185cb-214e-4848-91b8-ea3b529f1d30 None None] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641"
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Traceback (most recent call last):
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/bin/neutron-server", line 10, in <module>
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     sys.exit(main())
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19, in main
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/server/__init__.py", line 68, in boot_server
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     server_func()
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/service.py", line 94, in serve_wsgi
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service)
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     _get_callback_manager().publish(resource, event, trigger, payload=payload)
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     return self.notify(resource, event, trigger, payload=payload)
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 109, in _wrapped
Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise db_exc.RetryRequest(e)
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     self.force_reraise()
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     six.reraise(self.type_, self.value, self.tb)
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/six.py", line 703, in reraise
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise value
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     return function(*args, **kwargs)
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 174, in notify
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise exceptions.CallbackFailure(errors=errors)
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641"
Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron 
Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship.  (this warning may be suppressed after 10 occurrences)
Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]:   util.warn_limited(
Jan 11 06:30:42.267854 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
Jan 11 06:30:42.267794 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi exited, is_accepting=True
Jan 11 06:30:42.267851 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Failed with result 'exit-code'.
Jan 11 06:30:42.269078 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi exited, is_accepting=True
Jan 11 06:30:42.268577 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Consumed 4.684s CPU time.

Link to logs of the failed job:
https://zuul.openstack.org/build/4c2d09b13a8c4e9391433aa6382d344a/logs

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure ovn

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1911128

Title:
  Neutron with ovn driver failed to start on Fedora

Status in neutron:
  Confirmed

Bug description:
  Our periodic job neutron-ovn-tempest-ovs-master-fedora is failing
  since some time 100% of times. It fails due to failed neutron-server
  start. Error in the neutron logs is like below:

  Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship QosNetworkPolicyBinding.port to refer to viewonly relationship Port.qos_network_policy_binding should include sync_backref=False set on the QosNetworkPolicyBinding.port relationship.  (this warning may be suppressed after 10 occurrences)
  Jan 11 06:30:42.031451 fedora-32-limestone-regionone-0022468210 neutron-server[88725]:   util.warn_limited(
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     self._wait_for_pg_drop_event()
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     idl = ovsdb_monitor.OvnInitPGNbIdl.from_server(
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     helper = idlutils.get_schema_helper(connection_string, schema_name)
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager     raise Exception("Could not retrieve schema from %s" % connection)
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641
  Jan 11 06:30:42.032081 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: ERROR neutron_lib.callbacks.manager 
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-627086 process, after_init: Exception: Could not retrieve schema from ssl:10.4.70.225:6641
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 197, in _notify_loop
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     callback(resource, event, trigger, **kwargs)
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 272, in post_fork_initialize
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     self._wait_for_pg_drop_event()
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 340, in _wait_for_pg_drop_event
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     idl = ovsdb_monitor.OvnInitPGNbIdl.from_server(
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py", line 582, in from_server
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     helper = idlutils.get_schema_helper(connection_string, schema_name)
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager   File "/usr/local/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 204, in get_schema_helper
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager     raise Exception("Could not retrieve schema from %s" % connection)
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager Exception: Could not retrieve schema from ssl:10.4.70.225:6641
  Jan 11 06:30:42.033155 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: ERROR neutron_lib.callbacks.manager 
  Jan 11 06:30:42.033666 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi starting up on http://0.0.0.0:19696
  Jan 11 06:30:42.034612 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi starting up on http://0.0.0.0:19696
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: CRITICAL neutron [None req-4c1185cb-214e-4848-91b8-ea3b529f1d30 None None] Unhandled error: neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641"
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron Traceback (most recent call last):
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/bin/neutron-server", line 10, in <module>
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     sys.exit(main())
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/cmd/eventlet/server/__init__.py", line 19, in main
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     server.boot_server(wsgi_eventlet.eventlet_wsgi_server)
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/server/__init__.py", line 68, in boot_server
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     server_func()
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/server/wsgi_eventlet.py", line 24, in eventlet_wsgi_server
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/opt/stack/neutron/neutron/service.py", line 94, in serve_wsgi
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     registry.publish(resources.PROCESS, events.BEFORE_SPAWN, service)
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/registry.py", line 60, in publish
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     _get_callback_manager().publish(resource, event, trigger, payload=payload)
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 149, in publish
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     return self.notify(resource, event, trigger, payload=payload)
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 109, in _wrapped
  Jan 11 06:30:42.037436 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise db_exc.RetryRequest(e)
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     self.force_reraise()
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/six.py", line 703, in reraise
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise value
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/db/utils.py", line 104, in _wrapped
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     return function(*args, **kwargs)
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron   File "/usr/local/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 174, in notify
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron     raise exceptions.CallbackFailure(errors=errors)
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron neutron_lib.callbacks.exceptions.CallbackFailure: Callback neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.pre_fork_initialize-627113 failed with "Could not retrieve schema from ssl:10.4.70.225:6641"
  Jan 11 06:30:42.038157 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: ERROR neutron 
  Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]: /usr/local/lib64/python3.8/site-packages/sqlalchemy/orm/relationships.py:1994: SAWarning: Setting backref / back_populates on relationship Tag.standard_attr to refer to viewonly relationship StandardAttribute.tags should include sync_backref=False set on the Tag.standard_attr relationship.  (this warning may be suppressed after 10 occurrences)
  Jan 11 06:30:42.075747 fedora-32-limestone-regionone-0022468210 neutron-server[88725]:   util.warn_limited(
  Jan 11 06:30:42.267854 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
  Jan 11 06:30:42.267794 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Main process exited, code=exited, status=1/FAILURE
  Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
  Jan 11 06:30:42.268621 fedora-32-limestone-regionone-0022468210 neutron-server[89302]: INFO neutron.wsgi [-] (89302) wsgi exited, is_accepting=True
  Jan 11 06:30:42.267851 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Failed with result 'exit-code'.
  Jan 11 06:30:42.269078 fedora-32-limestone-regionone-0022468210 neutron-server[89301]: INFO neutron.wsgi [-] (89301) wsgi exited, is_accepting=True
  Jan 11 06:30:42.268577 fedora-32-limestone-regionone-0022468210 systemd[1]: devstack@q-svc.service: Consumed 4.684s CPU time.

  Link to logs of the failed job:
  https://zuul.openstack.org/build/4c2d09b13a8c4e9391433aa6382d344a/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1911128/+subscriptions
Follow ups

[Bug 1911128] Re: Neutron with ovn driver failed to start on Fedora
From: Bernard Cafarelli, 2021-03-12