yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1790027] Re: Error retrieving instances list

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Sun, 23 Sep 2018 23:09:52 -0000
Reply-to: Bug 1790027 <1790027@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This doesn't seem like a nova problem - there is a problem with the DB
connection configuration (maybe not scaled properly for your
environment?).

2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi
OperationalError: (pymysql.err.OperationalError) (1040, u'Too many
connections') (Background on this error at: http://sqlalche.me/e/e3q8)

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1790027

Title:
  Error retrieving instances list

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Queens release
  (After restart works ok)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi [req-73c5fdb2-822a-416b-b91c-b7e6fa36fb67 a700160233594eddb2621493b1a1d8d2 08847efa215b4fa6bbe6d1437e38194b - default default] Unexpected exception in API method: OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 439, in show
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     instance = self._get_server(context, req, id, is_detail=True)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 344, in _get_server
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     expected_attrs=expected_attrs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/api/openstack/common.py", line 479, in get_instance
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     expected_attrs=expected_attrs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2348, in get
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     expected_attrs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2308, in _get_instance
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     context, instance_uuid, expected_attrs=expected_attrs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     result = fn(cls, context, *args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 475, in get_by_uuid
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     use_slave=use_slave)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 467, in _db_instance_get_by_uuid
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     columns_to_join=columns_to_join)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 764, in instance_get_by_uuid
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return IMPL.instance_get_by_uuid(context, uuid, columns_to_join)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 169, in wrapper
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return f(*args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return f(context, *args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1872, in instance_get_by_uuid
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     columns_to_join=columns_to_join)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 1877, in _instance_get_by_uuid
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     filter_by(uuid=uuid).\
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     ret = list(self[0:1])
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return list(res)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return self._execute_and_instances(context)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     close_with_result=True)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     **kw
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     conn = self.session.connection(**kw)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     execution_options=execution_options)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     engine, execution_options)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     conn = bind.contextual_connect()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     self._wrap_pool_connect(self.pool.connect, None),
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     e, dialect, self)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     util.raise_from_cause(newraise, exc_info)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return fn()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return _ConnectionFairy._checkout(self)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     fairy = _ConnectionRecord.checkout(pool)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     rec = pool._do_get()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     self._dec_overflow()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     compat.reraise(exc_type, exc_value, exc_tb)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return self._create_connection()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return _ConnectionRecord(self)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     self.__connect(first_connect_check=True)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     connection = pool._invoke_creator(self)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return dialect.connect(*cargs, **cparams)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return self.dbapi.connect(*cargs, **cparams)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     return Connection(*args, **kwargs)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     self.connect()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 932, in connect
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     self._request_authentication()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1152, in _request_authentication
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     auth_packet = self._read_packet()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     packet.check_error()
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     err.raise_mysql_exception(self._data)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi     raise errorclass(errno, errval)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections') (Background on this error at: http://sqlalche.me/e/e3q8)
  2018-08-31 04:04:11.328 58009 ERROR nova.api.openstack.wsgi
  2018-08-31 04:04:11.339 58009 INFO nova.api.openstack.wsgi [req-73c5fdb2-822a-416b-b91c-b7e6fa36fb67 a700160233594eddb2621493b1a1d8d2 08847efa215b4fa6bbe6d1437e38194b - default default] HTTP exception thrown: Неожиданная ошибка API. Сообщите об этом в http://bugs.launchpad.net/nova/ и прикрепите протокол API Nova, если возможно.
  <class 'sqlalchemy.exc.OperationalError'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1790027/+subscriptions
References

[Bug 1790027] [NEW] Error retrieving instances list
From: Alexander, 2018-08-31