yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1648542] [NEW] keystone does not retry on deadlock Transactions [500 Error]

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Young <1648542@xxxxxxxxxxxxxxxxxx>
Date: Thu, 08 Dec 2016 16:29:38 -0000
Reply-to: Bug 1648542 <1648542@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description of problem:
DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

The above error is retry-able error, but no evidence for keystone would
really did a retry before throwing a 500.

2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi [req-c7dba8f5-9269-4c6f-b3e2-9a6c43b6cf0b a8377c6c4d05430d92ac661a2319cc95 c24adc59b2d0490c930d0270a1faecb5 - default default] (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     result = method(req, **params)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_log/versionutils.py", line 174, in wrapped
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return func_or_cls(*args, **kwargs)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/controllers.py", line 164, in delete_user
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.identity_api.delete_user(user_id, initiator)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 416, in wrapper
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 426, in wrapper
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 990, in delete_user
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     driver.delete_user(entity_id)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 277, in delete_user
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     session.delete(ref)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.gen.next()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 875, in _transaction_scope
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     yield resource
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.gen.next()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 522, in _session
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._end_session_transaction(self.session)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 543, in _end_session_transaction
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     session.commit()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.transaction.commit()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     t[1].commit()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._do_commit()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.connection._commit_impl()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._handle_dbapi_exception(e, None, None, None, None)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     util.raise_from_cause(newraise, exc_info)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     reraise(type(exception), exception, tb=exc_tb)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.engine.dialect.do_commit(self.connection)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     dbapi_connection.commit()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 719, in commit
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._read_ok_packet()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 698, in _read_ok_packet
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     pkt = self._read_packet()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     packet.check_error()
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     err.raise_mysql_exception(self._data)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     _check_mysql_exception(errinfo)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     raise InternalError(errno, errorvalue)
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')
2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1648542

Title:
  keystone does not retry on deadlock Transactions [500 Error]

Status in OpenStack Identity (keystone):
  New

Bug description:
  Description of problem:
  DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

  The above error is retry-able error, but no evidence for keystone
  would really did a retry before throwing a 500.

  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi [req-c7dba8f5-9269-4c6f-b3e2-9a6c43b6cf0b a8377c6c4d05430d92ac661a2319cc95 c24adc59b2d0490c930d0270a1faecb5 - default default] (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi Traceback (most recent call last):
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     result = method(req, **params)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_log/versionutils.py", line 174, in wrapped
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return func_or_cls(*args, **kwargs)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/controllers.py", line 164, in delete_user
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.identity_api.delete_user(user_id, initiator)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 416, in wrapper
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 426, in wrapper
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/core.py", line 990, in delete_user
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     driver.delete_user(entity_id)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/identity/backends/sql.py", line 277, in delete_user
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     session.delete(ref)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.gen.next()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 875, in _transaction_scope
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     yield resource
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.gen.next()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 522, in _session
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._end_session_transaction(self.session)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 543, in _end_session_transaction
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     session.commit()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.transaction.commit()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     t[1].commit()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._do_commit()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.connection._commit_impl()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._handle_dbapi_exception(e, None, None, None, None)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     util.raise_from_cause(newraise, exc_info)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     reraise(type(exception), exception, tb=exc_tb)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self.engine.dialect.do_commit(self.connection)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     dbapi_connection.commit()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 719, in commit
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     self._read_ok_packet()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 698, in _read_ok_packet
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     pkt = self._read_packet()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     packet.check_error()
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     err.raise_mysql_exception(self._data)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     _check_mysql_exception(errinfo)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi     raise InternalError(errno, errorvalue)
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')
  2016-11-12 08:55:10.995 13952 ERROR keystone.common.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1648542/+subscriptions
Follow ups

[Bug 1648542] Re: keystone does not retry on deadlock Transactions [500 Error] in identity backend
From: OpenStack Infra, 2017-01-07
[Bug 1648542] Re: keystone does not retry on deadlock Transactions [500 Error]
From: Boris Bobrov, 2016-12-08
[Bug 1648542] Re: keystone does not retry on deadlock Transactions [500 Error]
From: Adam Young, 2016-12-08