yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1583512] Re: DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ivan Lozgachev <ilozgachev@xxxxxxxxxxxx>
Date: Mon, 23 May 2016 09:03:47 -0000
Reply-to: Bug 1583512 <1583512@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Project changed: keystone => mos

** Changed in: mos
    Milestone: None => 9.0

** Changed in: mos
     Assignee: (unassigned) => MOS Keystone (mos-keystone)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1583512

Title:
  DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user

Status in Mirantis OpenStack:
  New

Bug description:
  The error happened at the end of Rally test when Rally started to
  remove users

  Found on MOS 9.0 build 308

  Environment:
    1 controller
    2 compute
    1 cinder

  Steps to reproduce:
    Run Sahara Rally test create_and_list_nodegroup_templates.yaml
    Wait until iterations passed and users context started to remove users

  Expected result:
    Keystone removed user by Rally request without errors

  Actual result:
    Keystone API raised 500 error

  From Keystone admin log:
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi [req-23d801cf-baa4-410d-9d46-6eb6ca7dfa34 21fe895155b54960b09290ef179ce15e 56b54d0fb50e40b5bb7ccdb46c8d21a7 - default default] (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 6, 57, 44, 270231),)]
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi Traceback (most recent call last):
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     result = method(context, **params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/oslo_log/versionutils.py", line 165, in wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return func_or_cls(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 174, in delete_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self.identity_api.delete_user(user_id, initiator)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 433, in wrapper
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 443, in wrapper
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 986, in delete_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     notifications.Audit.deleted(self._USER, user_id, initiator)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 166, in deleted
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     public)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 137, in _emit
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     public=public)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 421, in _send_notification
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     notify_event_callbacks(SERVICE, resource_type, operation, payload)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/notifications.py", line 316, in notify_event_callbacks
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     cb(service, resource_type, operation, payload)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 84, in _user_callback
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self.revoke_by_user(payload['resource_info'])
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 151, in revoke_by_user
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return self.revoke(revoke_model.RevokeEvent(user_id=user_id))
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 124, in wrapped
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/revoke/core.py", line 229, in revoke
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self.driver.revoke(event)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 100, in revoke
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self._prune_expired_events()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/keystone/revoke/backends/sql.py", line 77, in _prune_expired_events
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     query.delete(synchronize_session=False)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     delete_op.exec_()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1127, in exec_
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self._do_exec()
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1311, in _do_exec
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     mapper=self.mapper)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1034, in execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     bind, close_with_result=True).execute(clause, params or {})
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return meth(self, multiparams, params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     return connection._execute_clauseelement(self, multiparams, params)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     compiled_sql, distilled_params
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     context)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     util.raise_from_cause(newraise, exc_info)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     context)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     cursor.execute(statement, parameters)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     self.errorhandler(self, exc, value)
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi     raise errorvalue
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM revocation_event WHERE revocation_event.revoked_at < %s'] [parameters: (datetime.datetime(2016, 5, 19, 6, 57, 44, 270231),)]
  2016-05-19 08:27:44.302 21563 ERROR keystone.common.wsgi

To manage notifications about this bug go to:
https://bugs.launchpad.net/mos/+bug/1583512/+subscriptions
References

[Bug 1583512] [NEW] DBDeadlock: (_mysql_exceptions.OperationalError) while deleting user
From: Ivan Lozgachev, 2016-05-19