yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1791204] [NEW] nova conductor update database get deadlock

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Henry <1791204@xxxxxxxxxxxxxxxxxx>
Date: Fri, 07 Sep 2018 02:48:48 -0000
Reply-to: Bug 1791204 <1791204@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In order to test HA for controller node: shudown one controller node and
then vip switch to another node. Creating instance at this situation got
the Traceback in conductor:

2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server [req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371 0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in schedule_and_build_instances
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     instance.flavor, instance.uuid, block_device_mapping)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in _create_block_device_mapping
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     bdm.update_or_create()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return fn(self, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._create(self._context, update_or_create=True)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in _create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     context, updates, legacy=False)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in block_device_mapping_update_or_create
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return IMPL.block_device_mapping_update_or_create(context, values, legacy)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return f(context, *args, **kwargs)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 945, in _transaction_scope
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     yield resource
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.gen.next()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 562, in _session
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._end_session_transaction(self.session)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 583, in _end_session_transaction
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     session.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.transaction.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     t[1].commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._do_commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.connection._commit_impl()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._handle_dbapi_exception(e, None, None, None, None)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.engine.dialect.do_commit(self.connection)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     dbapi_connection.commit()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._read_ok_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in _read_ok_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     pkt = self._read_packet()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     packet.check_error()
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

Always got that in _create_block_device_mapping.

openstack version：ocata
nova --version
10.2.0

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791204

Title:
  nova conductor update database get deadlock

Status in OpenStack Compute (nova):
  New

Bug description:
  In order to test HA for controller node: shudown one controller node
  and then vip switch to another node. Creating instance at this
  situation got the Traceback in conductor:

  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server [req-503920a4-f2d8-47a8-a8d2-f9e4297be0ed a5be69ec54db4db2bf5aedc997da6371 0cdf4389dd09480098f4874e31585882 - - -] Exception during message handling
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 930, in schedule_and_build_instances
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     instance.flavor, instance.uuid, block_device_mapping)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 799, in _create_block_device_mapping
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     bdm.update_or_create()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return fn(self, *args, **kwargs)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 166, in update_or_create
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._create(self._context, update_or_create=True)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/objects/block_device.py", line 145, in _create
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     context, updates, legacy=False)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 1295, in block_device_mapping_update_or_create
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return IMPL.block_device_mapping_update_or_create(context, values, legacy)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 251, in wrapped
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     return f(context, *args, **kwargs)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.gen.next()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 945, in _transaction_scope
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     yield resource
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.gen.next()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 562, in _session
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._end_session_transaction(self.session)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 583, in _end_session_transaction
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     session.commit()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 813, in commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.transaction.commit()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 396, in commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     t[1].commit()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1574, in commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._do_commit()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1605, in _do_commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.connection._commit_impl()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 690, in _commit_impl
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._handle_dbapi_exception(e, None, None, None, None)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 688, in _commit_impl
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self.engine.dialect.do_commit(self.connection)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2514, in do_commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     dbapi_connection.commit()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 773, in commit
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     self._read_ok_packet()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 752, in _read_ok_packet
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     pkt = self._read_packet()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     packet.check_error()
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     err.raise_mysql_exception(self._data)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server     raise errorclass(errno, errval)
  2018-09-06 18:51:29.223 25 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

  Always got that in _create_block_device_mapping.

  openstack version：ocata
  nova --version
  10.2.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791204/+subscriptions