yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1617499] Re: DBDeadlock occurs when multiple neutron server starts at the same time

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1617499@xxxxxxxxxxxxxxxxxx>
Date: Tue, 22 Nov 2016 07:45:46 -0000
Reply-to: Bug 1617499 <1617499@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/361534
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=55b6c6a63a97c86ce06ff4dc87d6f253f0f6fea4
Submitter: Jenkins
Branch:    master

commit 55b6c6a63a97c86ce06ff4dc87d6f253f0f6fea4
Author: Kahou Lei <kahou82@xxxxxxxxx>
Date:   Fri Aug 26 17:43:38 2016 -0700

    Fixes DBDeadlock race condition during driver initialization.
    
    When multiple controllers are initially built, the ml2_vlan_allocations
    will be empty and the neutron servers under different controllers will
    try to populate the entry at the same time. This will cause this
    DBDeadlock error as they try to access the db at the same time.
    
    This patch set is to add db_api.retry_db_error decorator in the
    _sync_vlan_allocations method to avoid this race condition.
    
    Change-Id: I7c0e3ae515f592a5852e6decf6820b103f146761
    Closes-bug: 1617499


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1617499

Title:
  DBDeadlock occurs when multiple neutron server starts at the same time

Status in neutron:
  Fix Released

Bug description:
  I have multiple controllers and each controllers are running neutron-
  server. When I start the neutron-server, some of the controllers
  occasionally crash with DBDeadlock exceptions:

  2016-08-27 00:32:11.937 193309 CRITICAL neutron [-] DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]
  2016-08-27 00:32:11.937 193309 ERROR neutron Traceback (most recent call last):
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/bin/neutron-server", line 10, in <module>
  2016-08-27 00:32:11.937 193309 ERROR neutron     sys.exit(main_wsgi_eventlet())
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main_wsgi_eventlet
  2016-08-27 00:32:11.937 193309 ERROR neutron     wsgi_eventlet.main()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 50, in main
  2016-08-27 00:32:11.937 193309 ERROR neutron     server.boot_server(_eventlet_wsgi_server)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 35, in boot_server
  2016-08-27 00:32:11.937 193309 ERROR neutron     server_func()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 27, in _eventlet_wsgi_server
  2016-08-27 00:32:11.937 193309 ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 117, in serve_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     LOG.exception(_LE('Unrecoverable error: please check log '
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 114, in serve_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     service.start()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in start
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.wsgi_app = _run_wsgi(self.app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 225, in _run_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = config.load_paste_app(app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 246, in load_paste_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = deploy.loadapp("config:%s" % config_path, name=app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-08-27 00:32:11.937 193309 ERROR neutron     return loadobj(APP, uri, name=name, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-08-27 00:32:11.937 193309 ERROR neutron     return context.create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(app_name, global_conf=global_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(pipeline[-1])
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 81, in factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     return cls(**local_config)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 85, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     plugin = manager.NeutronManager.get_plugin()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 248, in get_plugin
  2016-08-27 00:32:11.937 193309 ERROR neutron     return weakref.proxy(cls.get_instance().plugin)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 242, in get_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     cls._create_instance()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
  2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 228, in _create_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     cls._instance = cls()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 121, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     plugin_provider)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 161, in _get_plugin_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     return plugin_class()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper
  2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 146, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.type_manager.initialize()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 182, in initialize
  2016-08-27 00:32:11.937 193309 ERROR neutron     driver.obj.initialize()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 167, in initialize
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._sync_vlan_allocations()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 161, in _sync_vlan_allocations
  2016-08-27 00:32:11.937 193309 ERROR neutron     session.delete(alloc)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.rollback()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.commit()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._prepare_impl()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.session.flush()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._flush(objects)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     transaction.rollback(_capture_exception=True)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     flush_context.execute()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     rec.execute(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     uow
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
  2016-08-27 00:32:11.937 193309 ERROR neutron     mapper, table, insert)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748, in _emit_insert_statements
  2016-08-27 00:32:11.937 193309 ERROR neutron     execute(statement, multiparams)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     return meth(self, multiparams, params)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-08-27 00:32:11.937 193309 ERROR neutron     return connection._execute_clauseelement(self, multiparams, params)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-08-27 00:32:11.937 193309 ERROR neutron     compiled_sql, distilled_params
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-08-27 00:32:11.937 193309 ERROR neutron     context)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-08-27 00:32:11.937 193309 ERROR neutron     util.raise_from_cause(newraise, exc_info)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2016-08-27 00:32:11.937 193309 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
  2016-08-27 00:32:11.937 193309 ERROR neutron     context)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 95, in do_executemany
  2016-08-27 00:32:11.937 193309 ERROR neutron     rowcount = cursor.executemany(statement, parameters)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in executemany
  2016-08-27 00:32:11.937 193309 ERROR neutron     r = r + self.execute(query, a)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.errorhandler(self, exc, value)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2016-08-27 00:32:11.937 193309 ERROR neutron     raise errorclass, errorvalue
  2016-08-27 00:32:11.937 193309 ERROR neutron DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]

  The problem occurs more oftern under this following situation:

  1. network_vlan_ranges is large (My condition is from 1 to 4095)
  2. slow controller

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1617499/+subscriptions
References

[Bug 1617499] [NEW] DBDeadlock occurs when multiple neutron server starts at the same time
From: Kahou Lei, 2016-08-27