yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1355928] [NEW] Deadlock in reservation commit

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matthew Booth <mbooth@xxxxxxxxxx>
Date: Tue, 12 Aug 2014 16:28:48 -0000
Reply-to: Bug 1355928 <1355928@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Details in http://logs.openstack.org/46/104146/15/check/check-tempest-
dsvm-full/d235389/, specifically in n-cond logs:

2014-08-12 14:58:57.099 ERROR nova.quota [req-7efe48be-f5b4-4343-898a-5b4b32694530 AggregatesAdminTestJSON-719157131 AggregatesAdminTestJSON-1908648657] Failed to commit reservations [u'5bdde344-b26f-4e0a-9aa7-d91d775b6df0', u'5f757426-8f4e-454f-aedb-1186771f85fd', u'819aeaf6-9faf-4da5-a16d-ce1c571c4975']
2014-08-12 14:58:57.099 21994 TRACE nova.quota Traceback (most recent call last):
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/quota.py", line 1326, in commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/quota.py", line 569, in commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/api.py", line 1148, in reservation_commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
2014-08-12 14:58:57.099 21994 TRACE nova.quota     return f(*args, **kwargs)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 205, in wrapped
2014-08-12 14:58:57.099 21994 TRACE nova.quota     return f(*args, **kwargs)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 3302, in reservation_commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota     for reservation in reservation_query.all():
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-08-12 14:58:57.099 21994 TRACE nova.quota     return list(self)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-08-12 14:58:57.099 21994 TRACE nova.quota     return self._execute_and_instances(context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-08-12 14:58:57.099 21994 TRACE nova.quota     result = conn.execute(querycontext.statement, self._params)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota     params)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-08-12 14:58:57.099 21994 TRACE nova.quota     compiled_sql, distilled_params
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-08-12 14:58:57.099 21994 TRACE nova.quota     context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-08-12 14:58:57.099 21994 TRACE nova.quota     exc_info
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
2014-08-12 14:58:57.099 21994 TRACE nova.quota     reraise(type(exception), exception, tb=exc_tb)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-08-12 14:58:57.099 21994 TRACE nova.quota     context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota     cursor.execute(statement, parameters)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota     self.errorhandler(self, exc, value)
2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-08-12 14:58:57.099 21994 TRACE nova.quota     raise errorclass, errorvalue
2014-08-12 14:58:57.099 21994 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.user_id AS reservations_user_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bdde344-b26f-4e0a-9aa7-d91d775b6df0', '5f757426-8f4e-454f-aedb-1186771f85fd', '819aeaf6-9faf-4da5-a16d-ce1c571c4975')
2014-08-12 14:58:57.099 21994 TRACE nova.quota

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1355928

Title:
  Deadlock in reservation commit

Status in OpenStack Compute (Nova):
  New

Bug description:
  Details in http://logs.openstack.org/46/104146/15/check/check-tempest-
  dsvm-full/d235389/, specifically in n-cond logs:

  2014-08-12 14:58:57.099 ERROR nova.quota [req-7efe48be-f5b4-4343-898a-5b4b32694530 AggregatesAdminTestJSON-719157131 AggregatesAdminTestJSON-1908648657] Failed to commit reservations [u'5bdde344-b26f-4e0a-9aa7-d91d775b6df0', u'5f757426-8f4e-454f-aedb-1186771f85fd', u'819aeaf6-9faf-4da5-a16d-ce1c571c4975']
  2014-08-12 14:58:57.099 21994 TRACE nova.quota Traceback (most recent call last):
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/quota.py", line 1326, in commit
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/quota.py", line 569, in commit
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/api.py", line 1148, in reservation_commit
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     user_id=user_id)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     return f(*args, **kwargs)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 205, in wrapped
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     return f(*args, **kwargs)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 3302, in reservation_commit
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     for reservation in reservation_query.all():
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2241, in all
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     return list(self)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     return self._execute_and_instances(context)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     result = conn.execute(querycontext.statement, self._params)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     params)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     compiled_sql, distilled_params
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     context)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     exc_info
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     reraise(type(exception), exception, tb=exc_tb)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     context)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     cursor.execute(statement, parameters)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     self.errorhandler(self, exc, value)
  2014-08-12 14:58:57.099 21994 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2014-08-12 14:58:57.099 21994 TRACE nova.quota     raise errorclass, errorvalue
  2014-08-12 14:58:57.099 21994 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.user_id AS reservations_user_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bdde344-b26f-4e0a-9aa7-d91d775b6df0', '5f757426-8f4e-454f-aedb-1186771f85fd', '819aeaf6-9faf-4da5-a16d-ce1c571c4975')
  2014-08-12 14:58:57.099 21994 TRACE nova.quota

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1355928/+subscriptions
Follow ups

[Bug 1355928] [NEW] Deadlock in reservation commit
From: Matthew Booth, 2014-08-12
References

[Bug 1355928] [NEW] Deadlock in reservation commit
From: Matthew Booth, 2014-08-12