yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1836029] Re: Got some db errors when call list servers

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Mon, 22 Jul 2019 15:57:26 -0000
Reply-to: Bug 1836029 <1836029@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I'm going to mark this as Invalid given the old release (newton) and the
fact it might already be fixed if you can upgrade.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836029

Title:
  Got some db errors when call list servers

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Got some db errors when call list servers and nova-api return http
  code 500

  2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exception in API method
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 208, in detail
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 359, in _get_servers
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1325, in fill_faults
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance_fault.py", line 108, in get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1906, in instance_fault_get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5999, in instance_fault_get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return list(self)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     close_with_result=True)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kw)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     execution_options=execution_options)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     engine, execution_options)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kwargs)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     fn(*args, **kw)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._autorollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     exc_info
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.engine.dialect.do_rollback(self.connection)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     dbapi_connection.rollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._read_ok_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     pkt = self._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Exception during reset or similar
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     fairy._reset(pool)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 776, in _reset
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pool._dialect.do_rollback(self)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     dbapi_connection.rollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     self._read_ok_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pkt = self._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool RuntimeError: reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters [req-179abec8-6e3e-4ee2-a732-43709f25aa91 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 972, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     recv_data = self._read_bytes(bytes_to_read)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 986, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._sock.settimeout(self._read_timeout)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'NoneType' object has no attribute 'settimeout'
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.788 16 INFO nova.api.openstack.wsgi [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | <class 'sqlalchemy.exc.DBAPIError'>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.789 16 INFO nova.osapi_compute.wsgi.server [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?host=node-4.domain.tld&project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 500 len: 568 time: 0.1924100

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1836029/+subscriptions
References

[Bug 1836029] [NEW] Got some db errors when call list servers
From: zhangyujun, 2019-07-10