← Back to team overview

openstack team mailing list archive

Re: Keystone Too Many Connections Problem

 

Thanks! I was able to reproduce and I've proposed a fix to both master and
grizzly.

master- https://review.openstack.org/#/c/25831/
grizzly- https://review.openstack.org/#/c/25833/

If you can verify this patch on your system, that would be really helpful.


-Dolph


On Mon, Apr 1, 2013 at 1:57 PM, Miller, Mark M (EB SW Cloud - R&D -
Corvallis) <mark.m.miller@xxxxxx> wrote:

>  Dolph,****
>
> ** **
>
> [token]****
>
> driver = keystone.token.backends.sql.Token****
>
> ** **
>
> # Amount of time a token should remain valid (in seconds)****
>
> expiration = 999986400****
>
> ** **
>
> I get the same results for either UUID or PKI tokens.****
>
> ** **
>
> Mark****
>
> ** **
>
> *From:* Dolph Mathews [mailto:dolph.mathews@xxxxxxxxx]
> *Sent:* Monday, April 01, 2013 11:56 AM
>
> *To:* Miller, Mark M (EB SW Cloud - R&D - Corvallis)
> *Cc:* openstack@xxxxxxxxxxxxxxxxxxx
> *Subject:* Re: [Openstack] Keystone Too Many Connections Problem****
>
>  ** **
>
> Mark,****
>
> ** **
>
> Which token driver are you using?****
>
> ** **
>
> -Dolph****
>
> ** **
>
> On Mon, Apr 1, 2013 at 10:49 AM, Dolph Mathews <dolph.mathews@xxxxxxxxx>
> wrote:****
>
> I also opened bug 1162857, which produces a slightly different symptom
> with a different configuration (sqlite), but may share the same underlying
> cause.****
>
>
> ****
>
> ** **
>
> -Dolph****
>
> ** **
>
> On Mon, Apr 1, 2013 at 10:26 AM, Dolph Mathews <dolph.mathews@xxxxxxxxx>
> wrote:****
>
> I'm attempting to reproduce this now (haven't been successful with PKI +
> sqlite on disk), but in the meantime I opened a bug on this issue for
> tracking:****
>
> ** **
>
>   https://bugs.launchpad.net/keystone/+bug/1162845****
>
>
> ****
>
> ** **
>
> -Dolph****
>
> ** **
>
> On Fri, Mar 29, 2013 at 1:35 PM, Miller, Mark M (EB SW Cloud - R&D -
> Corvallis) <mark.m.miller@xxxxxx> wrote:****
>
>   Hi,****
>
>  ****
>
> I have a loop that gets PKI tokens from Keystone (RC2). After about 116
> loops I get the following error and have to reset Keystone to continue. Has
> anyone else run into this?****
>
>  ****
>
> Thanks,****
>
>  ****
>
> Mark****
>
>  ****
>
> (keystone.auth.controllers): 2013-03-29 11:14:01,274 ERROR
> (OperationalError) (1040, 'Too many connections') None None****
>
> Traceback (most recent call last):****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py",
> line 286, in authenticate_for_token****
>
>     self.authenticate(context, auth_info, auth_context)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/controllers.py",
> line 352, in authenticate****
>
>     auth_context)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py",
> line 108, in authenticate****
>
>     user_info = UserAuthInfo(context, auth_payload)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py",
> line 36, in __init__****
>
>     self._validate_and_normalize_auth_data(auth_payload)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py",
> line 87, in _validate_and_normalize_auth_data****
>
>     domain_ref = self._lookup_domain(user_info['domain'])****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/auth/plugins/password.py",
> line 60, in _lookup_domain****
>
>     context=self.context, domain_name=domain_name)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/manager.py",
> line 47, in _wrapper****
>
>     return f(*args, **kw)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/common/sql/core.py",
> line 272, in wrapper****
>
>     return method(*args, **kwargs)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/keystone-2013.1.rc2-py2.7.egg/keystone/identity/backends/sql.py",
> line 589, in get_domain_by_name****
>
>     ref = session.query(Domain).filter_by(name=domain_name).one()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2184, in one****
>
>     ret = list(self)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2227, in __iter__****
>
>     return self._execute_and_instances(context)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2240, in _execute_and_instances****
>
>     close_with_result=True)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2231, in _connection_from_session****
>
>     **kw)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 777, in connection****
>
>     close_with_result=close_with_result)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/orm/session.py",
> line 783, in _connection_for_bind****
>
>     return engine.contextual_connect(**kwargs)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py",
> line 2489, in contextual_connect****
>
>     self.pool.connect(),****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 236, in connect****
>
>     return _ConnectionFairy(self).checkout()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 401, in __init__****
>
>     rec = self._connection_record = pool._do_get()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 746, in _do_get****
>
>     con = self._create_connection()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 189, in _create_connection****
>
>     return _ConnectionRecord(self)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 282, in __init__****
>
>     self.connection = self.__connect()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/pool.py",
> line 344, in __connect****
>
>     connection = self.__pool._creator()****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/strategies.py",
> line 80, in connect****
>
>     return dialect.connect(*cargs, **cparams)****
>
>   File
> "/usr/local/lib/python2.7/dist-packages/SQLAlchemy-0.7.9-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py",
> line 281, in connect****
>
>     return self.dbapi.connect(*cargs, **cparams)****
>
>   File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in
> Connect****
>
>     return Connection(*args, **kwargs)****
>
>   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line
> 187, in __init__****
>
>     super(Connection, self).__init__(*args, **kwargs2)****
>
> OperationalError: (OperationalError) (1040, 'Too many connections') None
> None****
>
> (access): 2013-03-29 11:14:01,278 INFO 15.253.58.148 - -
> [29/Mar/2013:18:14:01 +0000] "POST
> http://15.253.58.165:35357/v3/auth/tokens HTTP/1.0" 401 125****
>
> (eventlet.wsgi.server): 2013-03-29 11:14:01,279 DEBUG 15.253.58.148 - -
> [29/Mar/2013 11:14:01] "POST /v3/auth/tokens HTTP/1.1" 401 290 0.017322***
> *
>
>  ****
>
> ** **
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp****
>
>  ** **
>
> ** **
>
> ** **
>

References