yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2009263] Re: 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <2009263@xxxxxxxxxxxxxxxxxx>
Date: Fri, 17 Mar 2023 09:03:00 -0000
Reply-to: Bug 2009263 <2009263@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
This looks to me that the servicegroup API wasn't able to query the DB
to find the compute state value. For some reason, the conductor can't
connect to the DB.

Anyway, closing this one as this is not related to a project development
bug.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2009263

Title:
  'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted
  interval by 7230.58 sec

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  nova-conductor service to access the DB service to delay 7230.58s

  The more information is：
  2023-03-02 18:39:53.726 22 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.727 19 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.729 23 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
  (Background on this error at: http://sqlalche.me/e/e3q8)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 812, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exc.InvalidatePoolError()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db sqlalchemy.exc.InvalidatePoolError: ()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db During handling of the above exception, another exception occurred:
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db The above exception was the direct cause of the following exception:
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py", line 93, in _report_state
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service.service_ref.save()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn(self, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/service.py", line 436, in save
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     db_service = db.service_update(self._context, self.id, updates)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 182, in service_update
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ectxt.value = e.inner_exc
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.force_reraise()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     six.reraise(self.type_, self.value, self.tb)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise value
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 223, in wrapped
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 552, in service_update
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service_ref = service_get(context, service_id)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 238, in wrapped
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 413, in service_get
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     result = query.first()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3300, in first
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ret = list(self[0:1])
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3078, in __getitem__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return list(res)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self._execute_and_instances(context)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3427, in _execute_and_instances
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     querycontext, self._connection_from_session, close_with_result=True
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3442, in _get_bind_args
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     mapper=self._bind_mapper(), clause=querycontext.statement, **kw
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3420, in _connection_from_session
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = self.session.connection(**kw)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1133, in connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     execution_options=execution_options,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1139, in _connection_for_bind
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     engine, execution_options
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = bind._contextual_connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._wrap_pool_connect(self.pool.connect, None),
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2289, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     e, dialect, self
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1552, in _handle_dbapi_exception_noconnection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db (Background on this error at: http://sqlalche.me/e/e3q8)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.737 20 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec
  2023-03-02 18:39:53.741 21 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.745 20 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2009263/+subscriptions
References

[Bug 2009263] [NEW] 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec
From: Jorhson Deng, 2023-03-04