yahoo-eng-team team mailing list archive

[Alan Pevec <1375467@xxxxxxxxxxxxxxxxxx>]

** Also affects: nova/icehouse
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1375467

Title:
  db deadlock on _instance_update()

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  New
Status in OpenStack Compute (nova) juno series:
  Fix Released

Bug description:
  continuing from the same pattern as that of
  https://bugs.launchpad.net/nova/+bug/1370191, we are also observing
  unhandled deadlocks on derivatives of _instance_update(), such as the
  stacktrace below.  As _instance_update() is a point of transaction
  demarcation based on its use of get_session(), the @_retry_on_deadlock
  should be added to this method.

  Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\
      incoming.message))\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\
      return self._do_dispatch(endpoint, method, ctxt, args)\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\
      result = getattr(endpoint, method)(ctxt, **new_args)\
  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 887, in instance_update\
      service)\
  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\
      return func(*args, **kwargs)\
  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 130, in instance_update\
      context, instance_uuid, updates)\
  File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 742, in instance_update_and_get_original\
   columns_to_join=columns_to_join)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 164, in wrapper\
      return f(*args, **kwargs)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2208, in instance_update_and_get_original\
   columns_to_join=columns_to_join)\
  File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 2299, in _instance_update\
      session.add(instance_ref)\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 447, in __exit__\
      self.rollback()\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__\
      compat.reraise(exc_type, exc_value, exc_tb)\
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 444, in __exit__\
      self.commit()\
  File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 443, in _wrap\
      _raise_if_deadlock_error(e, self.bind.dialect.name)\
  File "/usr/lib/python2.7/site-packages/nova/openstack/common/db/sqlalchemy/sessi on.py", line 427, in _raise_if_deadlock_error\
      raise exception.DBDeadlock(operational_error)\
  DBDeadlock: (OperationalError) (1213, \'Deadlock found when trying to get lock; try restarting transaction\') None None\

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1375467/+subscriptions