yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1813885] [NEW] ERROR nova.api.openstack.wsgi

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Franky Yustanto <battossai@xxxxxxxxx>
Date: Wed, 30 Jan 2019 08:30:00 -0000
Reply-to: Bug 1813885 <1813885@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

2019-01-30 15:28:10.476 24371 INFO nova.api.openstack.wsgi [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2019-01-30 15:28:10.477 24371 DEBUG nova.api.openstack.wsgi [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077
2019-01-30 15:28:10.478 24371 INFO nova.osapi_compute.wsgi.server [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] 10.254.254.1 "GET /v2.1/os-simple-tenant-usage/d451c7c67f6a4e15856b591b79aa6df7?start=2019-01-29T00:00:00&end=2019-01-30T23:59:59 HTTP/1.1" status: 500 len: 640 time: 0.0175700
2019-01-30 15:28:10.490 24371 DEBUG nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Calling method '<function version_select at 0x7f7eb6a65050>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:618
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/limits.py", line 51, in index
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._index(req)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/limits.py", line 79, in _index
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     usages=False)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/quota.py", line 1106, in get_project_quotas
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     remains=remains)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/quota.py", line 285, in get_project_quotas
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     context, project_id)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/quotas.py", line 419, in get_all_by_project
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     project_id)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 992, in wrapper
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     with self._transaction_scope(context):
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.gen.next()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     context=context) as resource:
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.gen.next()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 645, in _session
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     bind=self.connection, mode=self.mode)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 409, in _create_session
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._start()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 496, in _start
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     engine_args, maker_args)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 520, in _setup_for_connection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     sql_connection=sql_connection, **engine_kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return wrapped(*args, **kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 202, in create_engine
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     test_conn = _test_connection(engine, max_retries, retry_interval)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 380, in _test_connection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return engine.connect()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2102, in connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._connection_cls(self, **kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     if connection is not None else engine.raw_connection()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.pool.unique_connection, _connection)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     e, dialect, self)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     util.raise_from_cause(newraise, exc_info)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return fn()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return _ConnectionFairy._checkout(self)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     fairy = _ConnectionRecord.checkout(pool)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     rec = pool._do_get()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._dec_overflow()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     compat.reraise(exc_type, exc_value, exc_tb)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._create_connection()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return _ConnectionRecord(self)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.__connect(first_connect_check=True)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     connection = pool._invoke_creator(self)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return dialect.connect(*cargs, **cparams)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.dbapi.connect(*cargs, **cparams)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return Connection(*args, **kwargs)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 327, in __init__
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.connect()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 598, in connect
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._request_authentication()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     auth_packet = self._process_auth(plugin_name, auth_packet)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 933, in _process_auth
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     pkt = self._read_packet()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     packet.check_error()
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi 
2019-01-30 15:28:10.500 24371 INFO nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2019-01-30 15:28:10.500 24371 DEBUG nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077
2019-01-30 15:28:10.502 24371 INFO nova.osapi_compute.wsgi.server [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] 10.254.254.1 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 500 len: 638 time: 0.0152109
2019-01-30 15:28:37.231 24363 DEBUG nova.osapi_compute.wsgi.server [req-808dd2f8-2bbe-4b5f-8b56-1ef84b847160 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24363) timed out ('10.254.254.1', 55138) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
2019-01-30 15:28:37.236 24398 DEBUG nova.osapi_compute.wsgi.server [req-feee6b38-93d7-48f3-b3c5-d74788eb7c0b 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24398) timed out ('10.254.254.1', 55132) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
2019-01-30 15:28:37.241 24376 DEBUG nova.osapi_compute.wsgi.server [req-be332107-4d44-4388-89a6-e54d0dc5ad50 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24376) timed out ('10.254.254.1', 55136) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
2019-01-30 15:28:37.278 24354 DEBUG nova.osapi_compute.wsgi.server [req-e161ab75-409e-43ed-b25c-d250800f3b2b 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24354) timed out ('10.254.254.1', 55144) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1813885

Title:
  ERROR nova.api.openstack.wsgi

Status in OpenStack Compute (nova):
  New

Bug description:
  2019-01-30 15:28:10.476 24371 INFO nova.api.openstack.wsgi [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2019-01-30 15:28:10.477 24371 DEBUG nova.api.openstack.wsgi [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077
  2019-01-30 15:28:10.478 24371 INFO nova.osapi_compute.wsgi.server [req-4b93e227-220b-4758-a100-7ed779da9cc2 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] 10.254.254.1 "GET /v2.1/os-simple-tenant-usage/d451c7c67f6a4e15856b591b79aa6df7?start=2019-01-29T00:00:00&end=2019-01-30T23:59:59 HTTP/1.1" status: 500 len: 640 time: 0.0175700
  2019-01-30 15:28:10.490 24371 DEBUG nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Calling method '<function version_select at 0x7f7eb6a65050>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:618
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 192, in wrapper
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return func(*args, **kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/limits.py", line 51, in index
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._index(req)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/limits.py", line 79, in _index
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     usages=False)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/quota.py", line 1106, in get_project_quotas
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     remains=remains)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/quota.py", line 285, in get_project_quotas
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     context, project_id)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/quotas.py", line 419, in get_all_by_project
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     project_id)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 992, in wrapper
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     with self._transaction_scope(context):
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.gen.next()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     context=context) as resource:
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.gen.next()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 645, in _session
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     bind=self.connection, mode=self.mode)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 409, in _create_session
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._start()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 496, in _start
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     engine_args, maker_args)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 520, in _setup_for_connection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     sql_connection=sql_connection, **engine_kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return wrapped(*args, **kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 202, in create_engine
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     test_conn = _test_connection(engine, max_retries, retry_interval)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 380, in _test_connection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return engine.connect()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2102, in connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._connection_cls(self, **kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     if connection is not None else engine.raw_connection()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.pool.unique_connection, _connection)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     e, dialect, self)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     util.raise_from_cause(newraise, exc_info)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return fn()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return _ConnectionFairy._checkout(self)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 788, in _checkout
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     fairy = _ConnectionRecord.checkout(pool)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     rec = pool._do_get()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1193, in _do_get
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._dec_overflow()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     compat.reraise(exc_type, exc_value, exc_tb)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1190, in _do_get
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self._create_connection()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return _ConnectionRecord(self)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.__connect(first_connect_check=True)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 671, in __connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     connection = pool._invoke_creator(self)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return dialect.connect(*cargs, **cparams)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return self.dbapi.connect(*cargs, **cparams)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 94, in Connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     return Connection(*args, **kwargs)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 327, in __init__
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self.connect()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 598, in connect
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     self._request_authentication()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 862, in _request_authentication
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     auth_packet = self._process_auth(plugin_name, auth_packet)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 933, in _process_auth
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     pkt = self._read_packet()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     packet.check_error()
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
  2019-01-30 15:28:10.497 24371 ERROR nova.api.openstack.wsgi 
  2019-01-30 15:28:10.500 24371 INFO nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2019-01-30 15:28:10.500 24371 DEBUG nova.api.openstack.wsgi [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1077
  2019-01-30 15:28:10.502 24371 INFO nova.osapi_compute.wsgi.server [req-183f3b97-9394-41df-a3a4-3a14263c3672 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] 10.254.254.1 "GET /v2.1/limits?reserved=1 HTTP/1.1" status: 500 len: 638 time: 0.0152109
  2019-01-30 15:28:37.231 24363 DEBUG nova.osapi_compute.wsgi.server [req-808dd2f8-2bbe-4b5f-8b56-1ef84b847160 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24363) timed out ('10.254.254.1', 55138) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
  2019-01-30 15:28:37.236 24398 DEBUG nova.osapi_compute.wsgi.server [req-feee6b38-93d7-48f3-b3c5-d74788eb7c0b 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24398) timed out ('10.254.254.1', 55132) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
  2019-01-30 15:28:37.241 24376 DEBUG nova.osapi_compute.wsgi.server [req-be332107-4d44-4388-89a6-e54d0dc5ad50 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24376) timed out ('10.254.254.1', 55136) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739
  2019-01-30 15:28:37.278 24354 DEBUG nova.osapi_compute.wsgi.server [req-e161ab75-409e-43ed-b25c-d250800f3b2b 308e45a6e9904a81a148c76917460727 d451c7c67f6a4e15856b591b79aa6df7 - default default] (24354) timed out ('10.254.254.1', 55144) process_request /usr/lib/python2.7/site-packages/eventlet/wsgi.py:739

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1813885/+subscriptions
Follow ups

[Bug 1813885] Re: ERROR nova.api.openstack.wsgi
From: Matt Riedemann, 2019-02-01