← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #09143

[Bug 1274341] [NEW] reservation_commit can lead to deadlock

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

reservation_commit can deadlock under load:

2014-01-29 18:51:00.470 ERROR nova.quota [req-659db2df-6124-4e6a-b86b-700aae1de805 183137d95c584efb84e773a21f2ef7a1 d8d5b06deffc45e7b258eb65ea04017c] Failed to commit reservations ['5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb']
2014-01-29 18:51:00.470 4380 TRACE nova.quota Traceback (most recent call last):
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 982, in commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota     self._driver.commit(context, reservations, project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 370, in commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota     db.reservation_commit(context, reservations, project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 970, in reservation_commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota     project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
2014-01-29 18:51:00.470 4380 TRACE nova.quota     return f(*args, **kwargs)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2786, in reservation_commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota     for reservation in reservation_query.all():
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
2014-01-29 18:51:00.470 4380 TRACE nova.quota     return list(self)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-01-29 18:51:00.470 4380 TRACE nova.quota     return self._execute_and_instances(context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-01-29 18:51:00.470 4380 TRACE nova.quota     result = conn.execute(querycontext.statement, self._params)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota     params)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-01-29 18:51:00.470 4380 TRACE nova.quota     compiled_sql, distilled_params
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-01-29 18:51:00.470 4380 TRACE nova.quota     context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
2014-01-29 18:51:00.470 4380 TRACE nova.quota     context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota     cursor.execute(statement, parameters)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota     self.errorhandler(self, exc, value)
2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-01-29 18:51:00.470 4380 TRACE nova.quota     raise errorclass, errorvalue
2014-01-29 18:51:00.470 4380 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb')

This can be fixed with our @_retry_on_deadlock wrapper

** Affects: nova
     Importance: High
         Status: Triaged


** Tags: havana-backport-potential

** Changed in: nova
   Importance: Undecided => High

** Changed in: nova
       Status: New => Triaged

** Tags added: havana-backport-potential

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1274341

Title:
  reservation_commit can lead to deadlock

Status in OpenStack Compute (Nova):
  Triaged

Bug description:
  reservation_commit can deadlock under load:

  2014-01-29 18:51:00.470 ERROR nova.quota [req-659db2df-6124-4e6a-b86b-700aae1de805 183137d95c584efb84e773a21f2ef7a1 d8d5b06deffc45e7b258eb65ea04017c] Failed to commit reservations ['5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb']
  2014-01-29 18:51:00.470 4380 TRACE nova.quota Traceback (most recent call last):
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 982, in commit
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     self._driver.commit(context, reservations, project_id=project_id)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 370, in commit
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     db.reservation_commit(context, reservations, project_id=project_id)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 970, in reservation_commit
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     project_id=project_id)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     return f(*args, **kwargs)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2786, in reservation_commit
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     for reservation in reservation_query.all():
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     return list(self)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     return self._execute_and_instances(context)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     result = conn.execute(querycontext.statement, self._params)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     params)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     compiled_sql, distilled_params
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     context)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     context)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     cursor.execute(statement, parameters)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     self.errorhandler(self, exc, value)
  2014-01-29 18:51:00.470 4380 TRACE nova.quota   File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2014-01-29 18:51:00.470 4380 TRACE nova.quota     raise errorclass, errorvalue
  2014-01-29 18:51:00.470 4380 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb')

  This can be fixed with our @_retry_on_deadlock wrapper

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1274341/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next