yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1417560] Re: DB race when starting multiple neutron-servers with GRE

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Adam Gandelman <1417560@xxxxxxxxxxxxxxxxxx>
Date: Thu, 09 Apr 2015 18:39:46 -0000
Reply-to: Bug 1417560 <1417560@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron/juno
   Importance: Undecided
       Status: New

** Changed in: neutron/juno
       Status: New => Fix Committed

** Changed in: neutron/juno
    Milestone: None => 2014.2.3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1417560

Title:
  DB race when starting multiple neutron-servers with GRE

Status in OpenStack Neutron (virtual network service):
  Fix Released
Status in neutron juno series:
  Fix Committed

Bug description:
  If more neutron-servers are started at once, all of them are trying to
  make the same entries in ml2_gre_allocations. Before entries are
  created, neutron looks whether allocation is already created and then
  creates missing allocations. It can happen that allocation is created
  by second neutron server causing first neutron server to crash:

  2015-01-09 14:28:25.768 24538 INFO neutron.plugins.ml2.drivers.type_tunnel [-] gre ID ranges: [(10, 10000)]
  2015-01-09 14:28:39.072 24538 ERROR neutron.service [-] Unrecoverable error: please check log for details.
  2015-01-09 14:28:39.072 24538 TRACE neutron.service Traceback (most recent call last):
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 102, in serve_wsgi
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     service.start()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 73, in start
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.wsgi_app = _run_wsgi(self.app_name)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 168, in _run_wsgi
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     app = config.load_paste_app(app_name)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 183, in load_paste_app
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     app = deploy.loadapp("config:%s" % config_path, name=app_name)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return loadobj(APP, uri, name=name, **kw)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return context.create()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return self.object_type.invoke(self)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     **context.local_conf)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     val = callable(*args, **kw)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     app = loader.get_app(app_name, global_conf=global_conf)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     name=name, global_conf=global_conf).create()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return self.object_type.invoke(self)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     **context.local_conf)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     val = callable(*args, **kw)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     app = loader.get_app(pipeline[-1])
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     name=name, global_conf=global_conf).create()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return self.object_type.invoke(self)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return fix_call(context.object, context.global_conf, **context.local_conf)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 56, in fix_call
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     val = callable(*args, **kw)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return cls(**local_config)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     plugin = manager.NeutronManager.get_plugin()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 219, in get_plugin
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return weakref.proxy(cls.get_instance().plugin)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 213, in get_instance
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     cls._create_instance()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py", line 249, in inner
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return f(*args, **kwargs)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 199, in _create_instance
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     cls._instance = cls()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 114, in __init__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     plugin_provider)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 140, in _get_plugin_instance
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return plugin_class()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 124, in __init__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.type_manager.initialize()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 146, in initialize
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     driver.obj.initialize()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_gre.py", line 73, in initialize
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self._initialize(cfg.CONF.ml2_type_gre.tunnel_id_ranges)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_tunnel.py", line 65, in _initialize
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.sync_allocations()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_gre.py", line 110, in sync_allocations
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     session.add(alloc)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 470, in __exit__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.rollback()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     compat.reraise(exc_type, exc_value, exc_tb)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 467, in __exit__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.commit()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 377, in commit
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self._prepare_impl()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 357, in _prepare_impl
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.session.flush()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self._flush(objects)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     transaction.rollback(_capture_exception=True)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     compat.reraise(exc_type, exc_value, exc_tb)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     flush_context.execute()
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     rec.execute(self)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     uow
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 65, in save_obj
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     mapper, table, insert)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 570, in _emit_insert_statements
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     execute(statement, multiparams)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return meth(self, multiparams, params)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     return connection._execute_clauseelement(self, multiparams, params)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     compiled_sql, distilled_params
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     context)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     util.raise_from_cause(newraise, exc_info)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     reraise(type(exception), exception, tb=exc_tb)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     context)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/sqlalchemy/connectors/mysqldb.py", line 68, in do_executemany
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     rowcount = cursor.executemany(statement, parameters)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in executemany
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     r = r + self.execute(query, a)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     self.errorhandler(self, exc, value)
  2015-01-09 14:28:39.072 24538 TRACE neutron.service   File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
  2015-01-09 14:28:39.072 24538 TRACE neutron.service     raise errorclass, errorvalue
  2015-01-09 14:28:39.072 24538 TRACE neutron.service DBDuplicateEntry: (IntegrityError) (1062, "Duplicate entry '10' for key 'PRIMARY'") 'INSERT INTO ml2_gre_allocations (gre_id, allocated) VALUES (%s, %s)' ((10, 0), (11, 0), (12, 0), (13, 0), (14, 0), (15, 0), (16, 0), (17, 0)  ... displaying 10 of 9991 total bound parameter sets ...  (9999, 0), (10000, 0))

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1417560/+subscriptions
References

[Bug 1417560] [NEW] DB race when starting multiple neutron-servers with GRE
From: Jakub Libosvar, 2015-02-03