yahoo-eng-team team mailing list archive

[Mike Bayer <mike_mp@xxxxxxxxxxxxxxxx>]

Public bug reported:

continuing from the same pattern as that of
https://bugs.launchpad.net/nova/+bug/1370191, we are also observing
unhandled deadlocks on derivatives of _instance_update(), such as the
stacktrace below.  As _instance_update() is a point of transaction
demarcation based on its use of get_session(), the @_retry_on_deadlock
should be added to this method.

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\
    incoming.message))\
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\
    return self._do_dispatch(endpoint, method, ctxt, args)\
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\
    result = getattr(endpoint, method)(ctxt, **new_args)\
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 887, in instance_update\
    service)\
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\
    return func(*args, **kwargs)\
File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 130, in instance_update\
    context, instance_uuid, updates)\
File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 742, in instance_update_and_get_original\
 columns_to_join=columns_to_join)\
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper\
    return f(*args, **kwargs)\
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2208, in instance_update_and_get_original\
 columns_to_join=columns_to_join)\
File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2299, in _instance_update\
    session.add(instance_ref)\
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\
    self.rollback()\
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\
    compat.reraise(exc_type, exc_value, exc_tb)\
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\
    self.commit()\
File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 443, in _wrap\
    _raise_if_deadlock_error(e, self.bind.dialect.name)\
File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 427, in _raise_if_deadlock_error\
    raise exception.DBDeadlock(operational_error)\
DBDeadlock: (OperationalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\') None None\

** Affects: nova
     Importance: Undecided
     Assignee: Mike Bayer (zzzeek)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1375467

Title:
  db deadlock on _instance_update()

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  continuing from the same pattern as that of
  https://bugs.launchpad.net/nova/+bug/1370191, we are also observing
  unhandled deadlocks on derivatives of _instance_update(), such as the
  stacktrace below.  As _instance_update() is a point of transaction
  demarcation based on its use of get_session(), the @_retry_on_deadlock
  should be added to this method.

  Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\
      incoming.message))\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\
      return self._do_dispatch(endpoint, method, ctxt, args)\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\
      result = getattr(endpoint, method)(ctxt, **new_args)\
  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 887, in instance_update\
      service)\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\
      return func(*args, **kwargs)\
  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 130, in instance_update\
      context, instance_uuid, updates)\
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 742, in instance_update_and_get_original\
   columns_to_join=columns_to_join)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper\
      return f(*args, **kwargs)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2208, in instance_update_and_get_original\
   columns_to_join=columns_to_join)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2299, in _instance_update\
      session.add(instance_ref)\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\
      self.rollback()\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\
      compat.reraise(exc_type, exc_value, exc_tb)\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\
      self.commit()\
  File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 443, in _wrap\
      _raise_if_deadlock_error(e, self.bind.dialect.name)\
  File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 427, in _raise_if_deadlock_error\
      raise exception.DBDeadlock(operational_error)\
  DBDeadlock: (OperationalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\') None None\

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1375467/+subscriptions