← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #17425

[Bug 1344657] [NEW] Deadlock found when trying to get lock; try restarting transaction (image_properties)

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Still seeing deadlocks occur when updating image properties:

2014-07-19 04:10:55.484 20377 INFO glance.wsgi.server [858953c7-3f21-470e-af68-bfa35cfd38d1 62f09a8e53d44a8d841eabdce3d8f61e a25b4b81a3e54551bf168ef3a27d33aa - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 559, in __call__
    return self._app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/new/glance/glance/common/wsgi.py", line 381, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/new/glance/glance/common/wsgi.py", line 647, in __call__
    request, **action_args)
  File "/opt/stack/new/glance/glance/common/wsgi.py", line 671, in dispatch
    return method(*args, **kwargs)
  File "/opt/stack/new/glance/glance/common/utils.py", line 428, in wrapped
    return func(self, req, *args, **kwargs)
  File "/opt/stack/new/glance/glance/registry/api/v1/images.py", line 364, in delete
    deleted_image = self.db_api.image_destroy(req.context, id)
  File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 135, in image_destroy
    _image_property_delete_all(context, image_id, delete_time, session)
  File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 964, in _image_property_delete_all
    session)
  File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 925, in _image_child_entry_delete_all
    count = query.update({"deleted": True, "deleted_at": delete_time})
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2690, in update
    update_op.exec_()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 816, in exec_
    self._do_exec()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 913, in _do_exec
    update_stmt, params=self.query._params)
  File "/opt/stack/new/glance/glance/openstack/common/db/sqlalchemy/session.py", line 444, in _wrap
    _raise_if_deadlock_error(e, self.bind.dialect.name)
  File "/opt/stack/new/glance/glance/openstack/common/db/sqlalchemy/session.py", line 427, in _raise_if_deadlock_error
    raise exception.DBDeadlock(operational_error)
DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE image_properties SET updated_at=%s, deleted_at=%s, deleted=%s WHERE image_properties.image_id = %s AND image_properties.deleted = false' (datetime.datetime(2014, 7, 19, 4, 10, 55, 320775), datetime.datetime(2014, 7, 19, 4, 10, 55, 310447), 1, '98a1f0c1-fbd9-4cd0-83fc-bc2fc547b4ab')


http://logs.openstack.org/75/108175/2/check/check-tempest-dsvm-full/5f85a7e/logs/screen-g-reg.txt.gz#_2014-07-19_04_10_55_477

** Affects: glance
     Importance: Undecided
         Status: New

** Summary changed:

- eadlock found when trying to get lock; try restarting transaction
+ Deadlock found when trying to get lock; try restarting transaction (image_properties)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1344657

Title:
  Deadlock found when trying to get lock; try restarting transaction
  (image_properties)

Status in OpenStack Image Registry and Delivery Service (Glance):
  New

Bug description:
  Still seeing deadlocks occur when updating image properties:

  2014-07-19 04:10:55.484 20377 INFO glance.wsgi.server [858953c7-3f21-470e-af68-bfa35cfd38d1 62f09a8e53d44a8d841eabdce3d8f61e a25b4b81a3e54551bf168ef3a27d33aa - - -] Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
      result = self.application(self.environ, start_response)
    File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 559, in __call__
      return self._app(env, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/new/glance/glance/common/wsgi.py", line 381, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/new/glance/glance/common/wsgi.py", line 647, in __call__
      request, **action_args)
    File "/opt/stack/new/glance/glance/common/wsgi.py", line 671, in dispatch
      return method(*args, **kwargs)
    File "/opt/stack/new/glance/glance/common/utils.py", line 428, in wrapped
      return func(self, req, *args, **kwargs)
    File "/opt/stack/new/glance/glance/registry/api/v1/images.py", line 364, in delete
      deleted_image = self.db_api.image_destroy(req.context, id)
    File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 135, in image_destroy
      _image_property_delete_all(context, image_id, delete_time, session)
    File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 964, in _image_property_delete_all
      session)
    File "/opt/stack/new/glance/glance/db/sqlalchemy/api.py", line 925, in _image_child_entry_delete_all
      count = query.update({"deleted": True, "deleted_at": delete_time})
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2690, in update
      update_op.exec_()
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 816, in exec_
      self._do_exec()
    File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 913, in _do_exec
      update_stmt, params=self.query._params)
    File "/opt/stack/new/glance/glance/openstack/common/db/sqlalchemy/session.py", line 444, in _wrap
      _raise_if_deadlock_error(e, self.bind.dialect.name)
    File "/opt/stack/new/glance/glance/openstack/common/db/sqlalchemy/session.py", line 427, in _raise_if_deadlock_error
      raise exception.DBDeadlock(operational_error)
  DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE image_properties SET updated_at=%s, deleted_at=%s, deleted=%s WHERE image_properties.image_id = %s AND image_properties.deleted = false' (datetime.datetime(2014, 7, 19, 4, 10, 55, 320775), datetime.datetime(2014, 7, 19, 4, 10, 55, 310447), 1, '98a1f0c1-fbd9-4cd0-83fc-bc2fc547b4ab')

  
  http://logs.openstack.org/75/108175/2/check/check-tempest-dsvm-full/5f85a7e/logs/screen-g-reg.txt.gz#_2014-07-19_04_10_55_477

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1344657/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next