yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1712703] Re: <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8c522feb-528b-4cb9-8dc6-9f39dac4c45b)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Aizuddin Zali <1712703@xxxxxxxxxxxxxxxxxx>
Date: Thu, 24 Aug 2017 14:08:57 -0000
Reply-to: Bug 1712703 <1712703@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This was caused by the underlying mysql database issue.

2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions
OperationalError: (pymysql.err.OperationalError) (1040, u'Too many
connections')

This stackoverflow should get you going back on track.
https://stackoverflow.com/questions/14331032/mysql-error-1040-too-many-connection


 Which deployment method being used to deploy? Packstack , TripleO, Devstack or etc?


** Changed in: nova
     Assignee: (unassigned) => Aizuddin Zali (mymzbe)

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1712703

Title:
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-
  8c522feb-528b-4cb9-8dc6-9f39dac4c45b)

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Right in the middle of normal operation, my openstack CLI interface
  gives me this error while trying to execute any openstack command.

  ~]# openstack server show demo-test
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8c522feb-528b-4cb9-8dc6-9f39dac4c45b)


  Here are my logs from /var/log/nova/nova-api.log:

  <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1039
  2017-08-23 16:37:40.884 28847 INFO nova.osapi_compute.wsgi.server [req-ea37152e-9b41-4ca4-aeba-5a6a94c0139b 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] 10.32.121.140 "GET /v2.1/a7f3a817a96c492992c1ed4f9fbdeee9/servers?name=dfd HTTP/1.1" status: 500 len: 572 time: 0.0153840
  2017-08-23 16:37:40.892 28847 DEBUG nova.api.openstack.wsgi [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x7154f90>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:626
  2017-08-23 16:37:40.896 28847 DEBUG nova.compute.api [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] Searching by: {'deleted': False, 'project_id': u'a7f3a817a96c492992c1ed4f9fbdeee9'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2384
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] Unexpected exception in API method
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 214, in detail
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 357, in _get_servers
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2452, in get_all
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     sort_dirs=sort_dirs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 342, in get_by_filters
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     build_requests = cls.get_all(context)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 257, in get_all
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     db_build_reqs = cls._get_all_from_db(context)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 893, in wrapper
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     with self._transaction_scope(context):
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return self.gen.next()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 944, in _transaction_scope
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     allow_async=self._allow_async) as resource:
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return self.gen.next()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 558, in _session
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     bind=self.connection, mode=self.mode)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _create_session
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self._start()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _start
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     engine_args, maker_args)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 427, in _setup_for_connection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     sql_connection=sql_connection, **engine_kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     test_conn = _test_connection(engine, max_retries, retry_interval)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return engine.connect()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2018, in connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return self._connection_cls(self, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in __init__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     if connection is not None else engine.raw_connection()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     e, dialect, self)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     util.raise_from_cause(newraise, exc_info)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return fn()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     rec = pool._do_get()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self._dec_overflow()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return self._create_connection()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return _ConnectionRecord(self)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in __init__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self.connection = self.__connect()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self.connect()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     self._request_authentication()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     auth_packet = self._read_packet()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     packet.check_error()
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
  2017-08-23 16:37:40.898 28847 ERROR nova.api.openstack.extensions
  2017-08-23 16:37:40.900 28847 INFO nova.api.openstack.wsgi [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2017-08-23 16:37:40.900 28847 DEBUG nova.api.openstack.wsgi [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1039
  2017-08-23 16:37:40.901 28847 INFO nova.osapi_compute.wsgi.server [req-4a3167b9-9129-4f73-a2a7-fc67aeb75cf7 8de5b578ddfe420aa28a9088ccf57403 a7f3a817a96c492992c1ed4f9fbdeee9 - default default] 10.32.121.140 "GET /v2.1/a7f3a817a96c492992c1ed4f9fbdeee9/servers/detail HTTP/1.1" status: 500 len: 572 time: 0.0152760

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1712703/+subscriptions
References

[Bug 1712703] [NEW] <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8c522feb-528b-4cb9-8dc6-9f39dac4c45b)
From: raminder, 2017-08-23