yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2009263] [NEW] 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jorhson Deng <2009263@xxxxxxxxxxxxxxxxxx>
Date: Sat, 04 Mar 2023 12:24:59 -0000
Reply-to: Bug 2009263 <2009263@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

nova-conductor service to access the DB service to delay 7230.58s

The more information is：
2023-03-02 18:39:53.726 22 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-03-02 18:39:53.727 19 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-03-02 18:39:53.729 23 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
(Background on this error at: http://sqlalche.me/e/e3q8)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 812, in _checkout
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exc.InvalidatePoolError()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db sqlalchemy.exc.InvalidatePoolError: ()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db During handling of the above exception, another exception occurred:
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db The above exception was the direct cause of the following exception:
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py", line 93, in _report_state
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service.service_ref.save()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn(self, *args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/service.py", line 436, in save
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     db_service = db.service_update(self._context, self.id, updates)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 182, in service_update
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ectxt.value = e.inner_exc
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.force_reraise()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     six.reraise(self.type_, self.value, self.tb)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise value
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(*args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 223, in wrapped
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 552, in service_update
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service_ref = service_get(context, service_id)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 238, in wrapped
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 413, in service_get
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     result = query.first()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3300, in first
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ret = list(self[0:1])
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3078, in __getitem__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return list(res)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3427, in _execute_and_instances
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     querycontext, self._connection_from_session, close_with_result=True
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3442, in _get_bind_args
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     mapper=self._bind_mapper(), clause=querycontext.statement, **kw
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3420, in _connection_from_session
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = self.session.connection(**kw)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1133, in connection
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     execution_options=execution_options,
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1139, in _connection_for_bind
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     engine, execution_options
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = bind._contextual_connect()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._wrap_pool_connect(self.pool.connect, None),
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2289, in _wrap_pool_connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     e, dialect, self
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1552, in _handle_dbapi_exception_noconnection
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db (Background on this error at: http://sqlalche.me/e/e3q8)
2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
2023-03-02 18:39:53.737 20 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec
2023-03-02 18:39:53.741 21 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2023-03-02 18:39:53.745 20 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2009263

Title:
  'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted
  interval by 7230.58 sec

Status in OpenStack Compute (nova):
  New

Bug description:
  nova-conductor service to access the DB service to delay 7230.58s

  The more information is：
  2023-03-02 18:39:53.726 22 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.727 19 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.729 23 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
  (Background on this error at: http://sqlalche.me/e/e3q8)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 812, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exc.InvalidatePoolError()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db sqlalchemy.exc.InvalidatePoolError: ()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db During handling of the above exception, another exception occurred:
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query')
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db The above exception was the direct cause of the following exception:
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/servicegroup/drivers/db.py", line 93, in _report_state
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service.service_ref.save()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn(self, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/objects/service.py", line 436, in save
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     db_service = db.service_update(self._context, self.id, updates)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/api.py", line 182, in service_update
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ectxt.value = e.inner_exc
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.force_reraise()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     six.reraise(self.type_, self.value, self.tb)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise value
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 223, in wrapped
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 552, in service_update
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     service_ref = service_get(context, service_id)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 238, in wrapped
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return f(context, *args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/db/sqlalchemy/api.py", line 413, in service_get
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     result = query.first()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3300, in first
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     ret = list(self[0:1])
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3078, in __getitem__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return list(res)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3405, in __iter__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self._execute_and_instances(context)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3427, in _execute_and_instances
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     querycontext, self._connection_from_session, close_with_result=True
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3442, in _get_bind_args
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     mapper=self._bind_mapper(), clause=querycontext.statement, **kw
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3420, in _connection_from_session
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = self.session.connection(**kw)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1133, in connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     execution_options=execution_options,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 1139, in _connection_for_bind
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     engine, execution_options
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 432, in _connection_for_bind
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     conn = bind._contextual_connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2251, in _contextual_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._wrap_pool_connect(self.pool.connect, None),
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2289, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     e, dialect, self
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1552, in _handle_dbapi_exception_noconnection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     util.raise_(newraise, with_traceback=exc_info[2], from_=e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 2285, in _wrap_pool_connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return fn()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 363, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return _ConnectionFairy._checkout(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 842, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record._checkin_failed(err)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 838, in _checkout
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     fairy._connection_record.get_connection()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 606, in get_connection
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.__connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 657, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     pool.logger.debug("Error on connect(): %s", e)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     exc_value, with_traceback=exc_tb,
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     raise exception
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/pool/base.py", line 652, in __connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     connection = pool._invoke_creator(self)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/__init__.py", line 94, in Connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 325, in __init__
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self.connect()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 598, in connect
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     self._get_server_information()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 975, in _get_server_information
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet = self._read_packet()
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 657, in _read_packet
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     packet_header = self._read_bytes(4)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db   File "/var/lib/kolla/venv/lib/python3.6/site-packages/pymysql/connections.py", line 707, in _read_bytes
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db (Background on this error at: http://sqlalche.me/e/e3q8)
  2023-03-02 18:39:53.732 20 ERROR nova.servicegroup.drivers.db
  2023-03-02 18:39:53.737 20 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec
  2023-03-02 18:39:53.741 21 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
  2023-03-02 18:39:53.745 20 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2009263/+subscriptions
Follow ups

[Bug 2009263] Re: 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 7230.58 sec
From: Sylvain Bauza, 2023-03-17