yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1370191] Re: db deadlock on service_update()

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 01 Oct 2014 07:44:42 -0000
Reply-to: Bug 1370191 <1370191@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => juno-rc1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1370191

Title:
  db deadlock on service_update()

Status in OpenStack Compute (Nova):
  Fix Released
Status in OpenStack Compute (nova) havana series:
  Fix Released
Status in OpenStack Compute (nova) icehouse series:
  Fix Committed

Bug description:
  Several methods in nova.db.sqlalchemy.api are decorated with
  @_retry_on_deadlock.  service_update() is not currently one of them,
  but it should be based on the following backtrace:

  4-09-15 15:40:22.574 34384 ERROR nova.servicegroup.drivers.db [-] model
  server went away
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db Traceback
  (most recent call last):
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line
  95, in _report_state
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  service.service_ref, state_catalog)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 218, in
  service_update
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  return self._manager.service_update(context, service, values)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/utils.py", line 967, in wrapper
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  return func(*args, **kwargs)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139,
  in inner
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  return func(*args, **kwargs)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 491, in
  service_update
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db     svc =
  self.db.service_update(context, service['id'], values)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/db/api.py", line 148, in
  service_update
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  return IMPL.service_update(context, service_id, values)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 146, in
  wrapper
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  return f(*args, **kwargs)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 533, in
  service_update
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  service_ref.update(values)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 447,
  in __exit__
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self.rollback()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line
  58, in __exit__
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  compat.reraise(exc_type, exc_value, exc_tb)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 444,
  in __exit__
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self.commit()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 358,
  in commit
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  t[1].commit()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1195,
  in commit
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self._do_commit()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1226,
  in _do_commit
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self.connection._commit_impl()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 491,
  in _commit_impl
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self._handle_dbapi_exception(e, None, None, None, None)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1024,
  in _handle_dbapi_exception
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  exc_info
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 196,
  in raise_from_cause
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  reraise(type(exception), exception, tb=exc_tb)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 488,
  in _commit_impl
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  self.engine.dialect.do_commit(self.connection)
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db   File
  "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
  line 2030, in do_commit
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  dbapi_connection.commit()
  2014-09-15 15:40:22.574 34384 TRACE nova.servicegroup.drivers.db
  OperationalError: (OperationalError) (1213, 'Deadlock found when trying to
  get lock; try restarting transaction') None None

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1370191/+subscriptions
References

[Bug 1370191] [NEW] db deadlock on service_update()
From: Russell Bryant, 2014-09-16