yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1617499] [NEW] DBDeadlock occurs when multiple neutron server starts at the same time

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Kahou Lei <kahou82@xxxxxxxxx>
Date: Sat, 27 Aug 2016 00:34:52 -0000
Reply-to: Bug 1617499 <1617499@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have multiple controllers and each controllers are running neutron-
server. When I start the neutron-server, some of the controllers
occasionally crash with DBDeadlock exceptions:

2016-08-27 00:32:11.937 193309 CRITICAL neutron [-] DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]
2016-08-27 00:32:11.937 193309 ERROR neutron Traceback (most recent call last):
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/bin/neutron-server", line 10, in <module>
2016-08-27 00:32:11.937 193309 ERROR neutron     sys.exit(main_wsgi_eventlet())
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main_wsgi_eventlet
2016-08-27 00:32:11.937 193309 ERROR neutron     wsgi_eventlet.main()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 50, in main
2016-08-27 00:32:11.937 193309 ERROR neutron     server.boot_server(_eventlet_wsgi_server)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 35, in boot_server
2016-08-27 00:32:11.937 193309 ERROR neutron     server_func()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 27, in _eventlet_wsgi_server
2016-08-27 00:32:11.937 193309 ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 117, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron     LOG.exception(_LE('Unrecoverable error: please check log '
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 114, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron     service.start()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in start
2016-08-27 00:32:11.937 193309 ERROR neutron     self.wsgi_app = _run_wsgi(self.app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 225, in _run_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron     app = config.load_paste_app(app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 246, in load_paste_app
2016-08-27 00:32:11.937 193309 ERROR neutron     app = deploy.loadapp("config:%s" % config_path, name=app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-08-27 00:32:11.937 193309 ERROR neutron     return loadobj(APP, uri, name=name, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-08-27 00:32:11.937 193309 ERROR neutron     return context.create()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(app_name, global_conf=global_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(pipeline[-1])
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron     return fix_call(context.object, context.global_conf, **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 81, in factory
2016-08-27 00:32:11.937 193309 ERROR neutron     return cls(**local_config)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 85, in __init__
2016-08-27 00:32:11.937 193309 ERROR neutron     plugin = manager.NeutronManager.get_plugin()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 248, in get_plugin
2016-08-27 00:32:11.937 193309 ERROR neutron     return weakref.proxy(cls.get_instance().plugin)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 242, in get_instance
2016-08-27 00:32:11.937 193309 ERROR neutron     cls._create_instance()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 228, in _create_instance
2016-08-27 00:32:11.937 193309 ERROR neutron     cls._instance = cls()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 121, in __init__
2016-08-27 00:32:11.937 193309 ERROR neutron     plugin_provider)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 161, in _get_plugin_instance
2016-08-27 00:32:11.937 193309 ERROR neutron     return plugin_class()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper
2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 146, in __init__
2016-08-27 00:32:11.937 193309 ERROR neutron     self.type_manager.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 182, in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron     driver.obj.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 167, in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron     self._sync_vlan_allocations()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 161, in _sync_vlan_allocations
2016-08-27 00:32:11.937 193309 ERROR neutron     session.delete(alloc)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron     self.rollback()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron     self.commit()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-08-27 00:32:11.937 193309 ERROR neutron     self._prepare_impl()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-08-27 00:32:11.937 193309 ERROR neutron     self.session.flush()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
2016-08-27 00:32:11.937 193309 ERROR neutron     self._flush(objects)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
2016-08-27 00:32:11.937 193309 ERROR neutron     transaction.rollback(_capture_exception=True)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
2016-08-27 00:32:11.937 193309 ERROR neutron     flush_context.execute()
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron     rec.execute(self)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron     uow
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
2016-08-27 00:32:11.937 193309 ERROR neutron     mapper, table, insert)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748, in _emit_insert_statements
2016-08-27 00:32:11.937 193309 ERROR neutron     execute(statement, multiparams)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron     return meth(self, multiparams, params)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-08-27 00:32:11.937 193309 ERROR neutron     return connection._execute_clauseelement(self, multiparams, params)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-08-27 00:32:11.937 193309 ERROR neutron     compiled_sql, distilled_params
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron     context)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-08-27 00:32:11.937 193309 ERROR neutron     util.raise_from_cause(newraise, exc_info)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2016-08-27 00:32:11.937 193309 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron     context)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 95, in do_executemany
2016-08-27 00:32:11.937 193309 ERROR neutron     rowcount = cursor.executemany(statement, parameters)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in executemany
2016-08-27 00:32:11.937 193309 ERROR neutron     r = r + self.execute(query, a)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron     self.errorhandler(self, exc, value)
2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-08-27 00:32:11.937 193309 ERROR neutron     raise errorclass, errorvalue
2016-08-27 00:32:11.937 193309 ERROR neutron DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]

The problem occurs more oftern under this following situation:

1. network_vlan_ranges is large (My condition is from 1 to 4095)
2. slow controller

** Affects: neutron
     Importance: Undecided
     Assignee: Kahou Lei (kahou82)
         Status: New

** Changed in: neutron
     Assignee: (unassigned) => Kahou Lei (kahou82)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1617499

Title:
  DBDeadlock occurs when multiple neutron server starts at the same time

Status in neutron:
  New

Bug description:
  I have multiple controllers and each controllers are running neutron-
  server. When I start the neutron-server, some of the controllers
  occasionally crash with DBDeadlock exceptions:

  2016-08-27 00:32:11.937 193309 CRITICAL neutron [-] DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]
  2016-08-27 00:32:11.937 193309 ERROR neutron Traceback (most recent call last):
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/bin/neutron-server", line 10, in <module>
  2016-08-27 00:32:11.937 193309 ERROR neutron     sys.exit(main_wsgi_eventlet())
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 19, in main_wsgi_eventlet
  2016-08-27 00:32:11.937 193309 ERROR neutron     wsgi_eventlet.main()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 50, in main
  2016-08-27 00:32:11.937 193309 ERROR neutron     server.boot_server(_eventlet_wsgi_server)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 35, in boot_server
  2016-08-27 00:32:11.937 193309 ERROR neutron     server_func()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 27, in _eventlet_wsgi_server
  2016-08-27 00:32:11.937 193309 ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 117, in serve_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     LOG.exception(_LE('Unrecoverable error: please check log '
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 114, in serve_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     service.start()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in start
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.wsgi_app = _run_wsgi(self.app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 225, in _run_wsgi
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = config.load_paste_app(app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 246, in load_paste_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = deploy.loadapp("config:%s" % config_path, name=app_name)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2016-08-27 00:32:11.937 193309 ERROR neutron     return loadobj(APP, uri, name=name, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2016-08-27 00:32:11.937 193309 ERROR neutron     return context.create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(app_name, global_conf=global_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     app = loader.get_app(pipeline[-1])
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2016-08-27 00:32:11.937 193309 ERROR neutron     name=name, global_conf=global_conf).create()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2016-08-27 00:32:11.937 193309 ERROR neutron     return self.object_type.invoke(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2016-08-27 00:32:11.937 193309 ERROR neutron     return fix_call(context.object, context.global_conf, **context.local_conf)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
  2016-08-27 00:32:11.937 193309 ERROR neutron     val = callable(*args, **kw)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 81, in factory
  2016-08-27 00:32:11.937 193309 ERROR neutron     return cls(**local_config)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 85, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     plugin = manager.NeutronManager.get_plugin()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 248, in get_plugin
  2016-08-27 00:32:11.937 193309 ERROR neutron     return weakref.proxy(cls.get_instance().plugin)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 242, in get_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     cls._create_instance()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in inner
  2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 228, in _create_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     cls._instance = cls()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 121, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     plugin_provider)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 161, in _get_plugin_instance
  2016-08-27 00:32:11.937 193309 ERROR neutron     return plugin_class()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper
  2016-08-27 00:32:11.937 193309 ERROR neutron     return f(*args, **kwargs)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 146, in __init__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.type_manager.initialize()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 182, in initialize
  2016-08-27 00:32:11.937 193309 ERROR neutron     driver.obj.initialize()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 167, in initialize
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._sync_vlan_allocations()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py", line 161, in _sync_vlan_allocations
  2016-08-27 00:32:11.937 193309 ERROR neutron     session.delete(alloc)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.rollback()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.commit()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._prepare_impl()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.session.flush()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     self._flush(objects)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in _flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     transaction.rollback(_capture_exception=True)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-08-27 00:32:11.937 193309 ERROR neutron     compat.reraise(exc_type, exc_value, exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in _flush
  2016-08-27 00:32:11.937 193309 ERROR neutron     flush_context.execute()
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     rec.execute(self)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     uow
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj
  2016-08-27 00:32:11.937 193309 ERROR neutron     mapper, table, insert)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748, in _emit_insert_statements
  2016-08-27 00:32:11.937 193309 ERROR neutron     execute(statement, multiparams)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     return meth(self, multiparams, params)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-08-27 00:32:11.937 193309 ERROR neutron     return connection._execute_clauseelement(self, multiparams, params)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-08-27 00:32:11.937 193309 ERROR neutron     compiled_sql, distilled_params
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-08-27 00:32:11.937 193309 ERROR neutron     context)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
  2016-08-27 00:32:11.937 193309 ERROR neutron     util.raise_from_cause(newraise, exc_info)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2016-08-27 00:32:11.937 193309 ERROR neutron     reraise(type(exception), exception, tb=exc_tb)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
  2016-08-27 00:32:11.937 193309 ERROR neutron     context)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 95, in do_executemany
  2016-08-27 00:32:11.937 193309 ERROR neutron     rowcount = cursor.executemany(statement, parameters)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in executemany
  2016-08-27 00:32:11.937 193309 ERROR neutron     r = r + self.execute(query, a)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
  2016-08-27 00:32:11.937 193309 ERROR neutron     self.errorhandler(self, exc, value)
  2016-08-27 00:32:11.937 193309 ERROR neutron   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2016-08-27 00:32:11.937 193309 ERROR neutron     raise errorclass, errorvalue
  2016-08-27 00:32:11.937 193309 ERROR neutron DBDeadlock: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations (physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters: (('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0', 5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0)  ... displaying 10 of 4095 total bound parameter sets ...  ('bond0', 4094, 0), ('bond0', 4095, 0))]

  The problem occurs more oftern under this following situation:

  1. network_vlan_ranges is large (My condition is from 1 to 4095)
  2. slow controller

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1617499/+subscriptions
Follow ups

[Bug 1617499] Re: DBDeadlock occurs when multiple neutron server starts at the same time
From: OpenStack Infra, 2016-11-22