yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1546110] Re: DB error causes router rescheduling loop to fail

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dave Walker <email@xxxxxxxxxx>
Date: Mon, 09 May 2016 11:43:57 -0000
Reply-to: Bug 1546110 <1546110@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron/kilo
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1546110

Title:
  DB error causes router rescheduling loop to fail

Status in neutron:
  Fix Released
Status in neutron kilo series:
  New

Bug description:
  In router rescheduling looping task db call to get down bindings is
  done outside of try/except block which may cause task to fail (see
  traceback below). Need to put db operation inside try/except.

  2016-02-15T10:44:44.259995+00:00 err: 2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall [req-79bce4c3-2e81-446c-8b37-6d30e3a964e2 - - - - -] Fixed interval looping call 'neutron.services.l3_router.l3_router_plugin.L3RouterPlugin.reschedule_routers_from_down_agents' failed
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall Traceback (most recent call last):
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 101, in reschedule_routers_from_down_agents
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall down_bindings = self._get_down_bindings(context, cutoff)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/neutron/db/l3_dvrscheduler_db.py", line 460, in _get_down_bindings
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall context, cutoff)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/neutron/db/l3_agentschedulers_db.py", line 149, in _get_down_bindings
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return query.all()
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2399, in all
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return list(self)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return self._execute_and_instances(context)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall close_with_result=True)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall **kw)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall execution_options=execution_options)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall conn = engine.contextual_connect(**kw)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall self._wrap_pool_connect(self.pool.connect, None),
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall e, dialect, self)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall util.raise_from_cause(newraise, exc_info)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall reraise(type(exception), exception, tb=exc_tb)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return fn()
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return _ConnectionFairy._checkout(self)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall fairy = _ConnectionRecord.checkout(pool)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 485, in checkout
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall rec.checkin()
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall compat.reraise(exc_type, exc_value, exc_tb)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 482, in checkout
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall dbapi_connection = rec.get_connection()
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 594, in get_connection
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall self.connection = self.__connect()
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall connection = self.__pool._invoke_creator(self)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return dialect.connect(*cargs, **cparams)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return self.dbapi.connect(*cargs, **cparams)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall return Connection(*args, **kwargs)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 206, in __init__
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall super(Connection, self).__init__(*args, **kwargs2)
  2016-02-15 10:44:44.250 15419 ERROR oslo.service.loopingcall DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1546110/+subscriptions
References

[Bug 1546110] [NEW] DB error causes router rescheduling loop to fail
From: Oleg Bondarev, 2016-02-16