yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2058018] [NEW] AssertionError: do not call blocking functions from the mainloop

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Will Szumski <2058018@xxxxxxxxxxxxxxxxxx>
Date: Fri, 15 Mar 2024 09:29:50 -0000
Reply-to: Bug 2058018 <2058018@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Release: 2023.1
Driver: OVN

I'm seeing the following error in my logs. I don't know the exact
trigger but seemed to occur during a tempest run:

Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dialect.do_execute(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     cursor.execute(statement, parameters)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self._query(query)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn.query(q)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result.read()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     first_packet = self.connection._read_packet()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     packet_header = self._read_bytes(4)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     data = self._rfile.read(num_bytes)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/usr/lib/python3.10/socket.py", line 705, in readinto
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._sock.recv_into(b)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._read_trampoline()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._trampoline(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return trampoline(fd, read=read, write=write, timeout=timeout,
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '')
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception:
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with util.safe_reraise(warn_only=True):
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     compat.raise_(value, with_traceback=traceback)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._autorollback()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._rollback_impl()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(e, None, None, None, None)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     util.raise_(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '')
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     action_func(x)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda>
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     lambda x: x.wait(),
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.thread.wait()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._exit_event.wait()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = hub.switch()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.greenlet.switch()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.wait(sleep_time)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     presult = self.do_poll(seconds)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.poll.poll(seconds)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f_with_retry(*args, **kwargs,
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception() as ectxt:
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*dup_args, **dup_kwargs)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_models.OVNHashRing.node_uuid == node_uuid).delete()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self.session.execute(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = self._connection_for_bind(bind)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._transaction._connection_for_bind(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = bind.connect()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._connection_cls(self, close_with_result=close_with_result)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dispatch.engine_connect(self, _branch_from is not None)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.scalar(select(1))
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.execute(object_, *multiparams, **params).scalar()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return connection._execute_clauseelement(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ret = self._execute_context(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.pool._invalidate(dbapi_conn_wrapper, e)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.invalidate(exception)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._checkin()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     _finalize_fairy(
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection_record.checkin()
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     pool.dispatch.checkin(connection, self)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     time.sleep(0)
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 


Sorry for the long backtrace. The important bit seems to be:

2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
    2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(

** Affects: neutron
     Importance: Undecided
         Status: New

** Description changed:

  Release: 2023.1
  Driver: OVN
  
  I'm seeing the following error in my logs. I don't know the exact
  trigger but seemed to occur during a tempest run:
  
-     Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dialect.do_execute(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     cursor.execute(statement, parameters)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self._query(query)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn.query(q)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result.read()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     first_packet = self.connection._read_packet()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     packet_header = self._read_bytes(4)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     data = self._rfile.read(num_bytes)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/usr/lib/python3.10/socket.py", line 705, in readinto
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._sock.recv_into(b)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._read_trampoline()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._trampoline(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return trampoline(fd, read=read, write=write, timeout=timeout,
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '')
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception:
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with util.safe_reraise(warn_only=True):
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     compat.raise_(value, with_traceback=traceback)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._autorollback()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._rollback_impl()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(e, None, None, None, None)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     util.raise_(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '')
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     action_func(x)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda>
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     lambda x: x.wait(),
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.thread.wait()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._exit_event.wait()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = hub.switch()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.greenlet.switch()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.wait(sleep_time)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     presult = self.do_poll(seconds)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.poll.poll(seconds)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f_with_retry(*args, **kwargs,
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception() as ectxt:
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*dup_args, **dup_kwargs)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_models.OVNHashRing.node_uuid == node_uuid).delete()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self.session.execute(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = self._connection_for_bind(bind)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._transaction._connection_for_bind(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = bind.connect()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._connection_cls(self, close_with_result=close_with_result)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dispatch.engine_connect(self, _branch_from is not None)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.scalar(select(1))
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.execute(object_, *multiparams, **params).scalar()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return connection._execute_clauseelement(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ret = self._execute_context(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.pool._invalidate(dbapi_conn_wrapper, e)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.invalidate(exception)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._checkin()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     _finalize_fairy(
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection_record.checkin()
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     pool.dispatch.checkin(connection, self)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     time.sleep(0)
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dialect.do_execute(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     cursor.execute(statement, parameters)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self._query(query)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn.query(q)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result.read()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     first_packet = self.connection._read_packet()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     packet_header = self._read_bytes(4)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     data = self._rfile.read(num_bytes)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/usr/lib/python3.10/socket.py", line 705, in readinto
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._sock.recv_into(b)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._read_trampoline()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._trampoline(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return trampoline(fd, read=read, write=write, timeout=timeout,
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '')
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception:
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with util.safe_reraise(warn_only=True):
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     compat.raise_(value, with_traceback=traceback)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._autorollback()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._rollback_impl()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(e, None, None, None, None)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     util.raise_(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '')
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     action_func(x)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda>
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     lambda x: x.wait(),
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.thread.wait()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._exit_event.wait()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = hub.switch()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.greenlet.switch()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.wait(sleep_time)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     presult = self.do_poll(seconds)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.poll.poll(seconds)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f_with_retry(*args, **kwargs,
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception() as ectxt:
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*dup_args, **dup_kwargs)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_models.OVNHashRing.node_uuid == node_uuid).delete()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self.session.execute(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = self._connection_for_bind(bind)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._transaction._connection_for_bind(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = bind.connect()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._connection_cls(self, close_with_result=close_with_result)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dispatch.engine_connect(self, _branch_from is not None)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.scalar(select(1))
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.execute(object_, *multiparams, **params).scalar()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return connection._execute_clauseelement(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ret = self._execute_context(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.pool._invalidate(dbapi_conn_wrapper, e)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.invalidate(exception)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._checkin()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     _finalize_fairy(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection_record.checkin()
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     pool.dispatch.checkin(connection, self)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     time.sleep(0)
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  
  
  Sorry for the long backtrace. The important bit seems to be:
  
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
-     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(
+ 2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
+     2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2058018

Title:
  AssertionError: do not call blocking functions from the mainloop

Status in neutron:
  New

Bug description:
  Release: 2023.1
  Driver: OVN

  I'm seeing the following error in my logs. I don't know the exact
  trigger but seemed to occur during a tempest run:

  Error waiting on thread.: AssertionError: do not call blocking functions from the mainloop
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dialect.do_execute(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     cursor.execute(statement, parameters)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 148, in execute
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self._query(query)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/cursors.py", line 310, in _query
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn.query(q)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 548, in query
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 775, in _read_query_result
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result.read()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 1156, in read
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     first_packet = self.connection._read_packet()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 692, in _read_packet
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     packet_header = self._read_bytes(4)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 732, in _read_bytes
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     data = self._rfile.read(num_bytes)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/usr/lib/python3.10/socket.py", line 705, in readinto
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._sock.recv_into(b)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 376, in recv_into
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._recv_loop(self.fd.recv_into, 0, buffer, nbytes, flags)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 364, in _recv_loop
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._read_trampoline()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 332, in _read_trampoline
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._trampoline(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenio/base.py", line 211, in _trampoline
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return trampoline(fd, read=read, write=write, timeout=timeout,
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/__init__.py", line 141, in trampoline
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup pymysql.err.InterfaceError: (0, '')
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup The above exception was the direct cause of the following exception:
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2118, in _handle_dbapi_exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with util.safe_reraise(warn_only=True):
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 84, in __exit__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     compat.raise_(value, with_traceback=traceback)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _handle_dbapi_exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._autorollback()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1187, in _autorollback
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._rollback_impl()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1059, in _rollback_impl
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(e, None, None, None, None)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     util.raise_(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1057, in _rollback_impl
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.dialect.do_rollback(self.connection)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 683, in do_rollback
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     dbapi_connection.rollback()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 479, in rollback
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/pymysql/connections.py", line 793, in _execute_command
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise err.InterfaceError(0, "")
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup sqlalchemy.exc.InterfaceError: (pymysql.err.InterfaceError) (0, '')
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup (Background on this error at: https://sqlalche.me/e/14/rvf5)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup During handling of the above exception, another exception occurred:
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup Traceback (most recent call last):
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 268, in _perform_action_on_threads
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     action_func(x)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 342, in <lambda>
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     lambda x: x.wait(),
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/threadgroup.py", line 61, in wait
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.thread.wait()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._exit_event.wait()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/event.py", line 125, in wait
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = hub.switch()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 313, in switch
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.greenlet.switch()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 365, in run
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.wait(sleep_time)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/poll.py", line 80, in wait
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     presult = self.do_poll(seconds)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.poll.poll(seconds)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 226, in wrapped
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f_with_retry(*args, **kwargs,
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 140, in wrapped
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 138, in wrapped
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception() as ectxt:
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*args, **kwargs)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 186, in wrapped
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     with excutils.save_and_reraise_exception():
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.force_reraise()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     raise self.value
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron_lib/db/api.py", line 184, in wrapped
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return f(*dup_args, **dup_kwargs)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/db/ovn_hash_ring_db.py", line 59, in remove_node_by_uuid
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_models.OVNHashRing.node_uuid == node_uuid).delete()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 3213, in delete
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     result = self.session.execute(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1711, in execute
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = self._connection_for_bind(bind)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1552, in _connection_for_bind
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._transaction._connection_for_bind(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 747, in _connection_for_bind
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     conn = bind.connect()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3315, in connect
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self._connection_cls(self, close_with_result=close_with_result)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 120, in __init__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.dispatch.engine_connect(self, _branch_from is not None)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 74, in _connect_ping_listener
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.scalar(select(1))
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1262, in scalar
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return self.execute(object_, *multiparams, **params).scalar()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     return connection._execute_clauseelement(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ret = self._execute_context(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._handle_dbapi_exception(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2137, in _handle_dbapi_exception
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self.engine.pool._invalidate(dbapi_conn_wrapper, e)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 283, in _invalidate
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection.invalidate(exception)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1090, in invalidate
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     self._checkin()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 982, in _checkin
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     _finalize_fairy(
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 790, in _finalize_fairy
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     connection_record.checkin()
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 531, in checkin
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     pool.dispatch.checkin(connection, self)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/sqlalchemy/event/attr.py", line 334, in __call__
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     fn(*args, **kw)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py", line 52, in _thread_yield
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     time.sleep(0)
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 33, in sleep
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup AssertionError: do not call blocking functions from the mainloop
  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup 

  
  Sorry for the long backtrace. The important bit seems to be:

  2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup   File "/var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 292, in _remove_node_from_hash_ring
      2024-03-14 15:31:34.443 1018 ERROR oslo_service.threadgroup     ovn_hash_ring_db.remove_node_by_uuid(

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2058018/+subscriptions