yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1836029] [NEW] Got some db errors when call list servers

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: zhangyujun <1836029@xxxxxxxxxxxxxxxxxx>
Date: Wed, 10 Jul 2019 09:21:21 -0000
Reply-to: Bug 1836029 <1836029@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Got some db errors when call list servers and nova-api return http code
500

2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     packet_header = self._read_bytes(4)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     data = self._rfile.read(num_bytes)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23>
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exception in API method
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 208, in detail
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 359, in _get_servers
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1325, in fill_faults
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance_fault.py", line 108, in get_by_instance_uuids
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1906, in instance_fault_get_by_instance_uuids
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5999, in instance_fault_get_by_instance_uuids
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return list(self)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     close_with_result=True)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kw)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     execution_options=execution_options)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     engine, execution_options)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kwargs)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     fn(*args, **kw)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     context)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._autorollback()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     exc_info
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.engine.dialect.do_rollback(self.connection)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     dbapi_connection.rollback()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._read_ok_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     pkt = self._read_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     packet_header = self._read_bytes(4)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     data = self._rfile.read(num_bytes)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=23>
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions 
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Exception during reset or similar
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last):
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     fairy._reset(pool)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 776, in _reset
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pool._dialect.do_rollback(self)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     dbapi_connection.rollback()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     self._read_ok_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pkt = self._read_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     packet_header = self._read_bytes(4)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     data = self._rfile.read(num_bytes)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool RuntimeError: reentrant call inside <_io.BufferedReader name=23>
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool 
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters [req-179abec8-6e3e-4ee2-a732-43709f25aa91 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 972, in _read_packet
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     recv_data = self._read_bytes(bytes_to_read)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 986, in _read_bytes
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._sock.settimeout(self._read_timeout)
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'NoneType' object has no attribute 'settimeout'
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters 
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.788 16 INFO nova.api.openstack.wsgi [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | <class 'sqlalchemy.exc.DBAPIError'>
2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.789 16 INFO nova.osapi_compute.wsgi.server [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?host=node-4.domain.tld&project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 500 len: 568 time: 0.1924100

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1836029

Title:
  Got some db errors when call list servers

Status in OpenStack Compute (nova):
  New

Bug description:
  Got some db errors when call list servers and nova-api return http
  code 500

  2019-07-02 10:08:39 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:39.736 16 INFO nova.osapi_compute.wsgi.server [req-5804b3dd-9968-481d-b03f-ed36192abb2f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 200 len: 42583 time: 0.2910399
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters RuntimeError: reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.784 16 ERROR oslo_db.sqlalchemy.exc_filters 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Unexpected exception in API method
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 208, in detail
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 359, in _get_servers
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1325, in fill_faults
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/instance_fault.py", line 108, in get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1906, in instance_fault_get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 5999, in instance_fault_get_by_instance_uuids
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2613, in all
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return list(self)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     close_with_result=True)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kw)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     execution_options=execution_options)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     engine, execution_options)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     **kwargs)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     fn(*args, **kw)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._autorollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     exc_info
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self.engine.dialect.do_rollback(self.connection)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     dbapi_connection.rollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     self._read_ok_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     pkt = self._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions DBAPIError: (exceptions.RuntimeError) reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.785 16 ERROR nova.api.openstack.extensions 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] Exception during reset or similar
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     fairy._reset(pool)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 776, in _reset
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pool._dialect.do_rollback(self)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2542, in do_rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     dbapi_connection.rollback()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in rollback
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     self._read_ok_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 747, in _read_ok_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     pkt = self._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 962, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     packet_header = self._read_bytes(4)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 989, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool     data = self._rfile.read(num_bytes)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool RuntimeError: reentrant call inside <_io.BufferedReader name=23>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR sqlalchemy.pool.QueuePool 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters [req-179abec8-6e3e-4ee2-a732-43709f25aa91 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] DB exception wrapped.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     context)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 972, in _read_packet
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     recv_data = self._read_bytes(bytes_to_read)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 986, in _read_bytes
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters     self._sock.settimeout(self._read_timeout)
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters AttributeError: 'NoneType' object has no attribute 'settimeout'
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.787 16 ERROR oslo_db.sqlalchemy.exc_filters 
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.788 16 INFO nova.api.openstack.wsgi [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | <class 'sqlalchemy.exc.DBAPIError'>
  2019-07-02 10:08:42 +0800 | nova-api-osapi-5475cd965-9wbl2 | 2019-07-02 10:08:42.789 16 INFO nova.osapi_compute.wsgi.server [req-edd4d5e5-05ee-4a94-974b-c127fc4ce86f 1c827fbc129a4025bcb2a2d8cacc6b3d d8ac61ac9f1d4e5e9aa9c4313c668834 - default default] 10.233.66.23 "GET /v2.1/d8ac61ac9f1d4e5e9aa9c4313c668834/servers/detail?host=node-4.domain.tld&project_id=d8ac61ac9f1d4e5e9aa9c4313c668834&redirect=detail_x HTTP/1.1" status: 500 len: 568 time: 0.1924100

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1836029/+subscriptions
Follow ups

[Bug 1836029] Re: Got some db errors when call list servers
From: Tytus Kurek, 2019-07-24
[Bug 1836029] Re: Got some db errors when call list servers
From: Matt Riedemann, 2019-07-22