← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1518927] [NEW] nova service state stops updating after SQL connection lost

 

Public bug reported:

After nova-conductor loses mysql connection, it stops to update the service 'Updated_at' status. 
After logging the error:
ERROR oslo.service.loopingcall [-] Fixed interval looping call 'nova.servicegroup.drivers.db.DbDriver._report_state' failed
.
.
.
DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.97.36.127' ((2013, 'Lost connection to MySQL server during query'))") [SQL: u'SELECT 1']

After some tracing, it seems nova.servicegroup.drivers.db _report state
is never called again. Restarting the service correctly restarts
reporting, too. Probably the reporting thread got stuck?

Using Liberty from Ubuntu Cloud Archive: 2:12.0.0-0ubuntu2~cloud0

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1518927

Title:
  nova service state stops updating after SQL connection lost

Status in OpenStack Compute (nova):
  New

Bug description:
  After nova-conductor loses mysql connection, it stops to update the service 'Updated_at' status. 
  After logging the error:
  ERROR oslo.service.loopingcall [-] Fixed interval looping call 'nova.servicegroup.drivers.db.DbDriver._report_state' failed
  .
  .
  .
  DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.97.36.127' ((2013, 'Lost connection to MySQL server during query'))") [SQL: u'SELECT 1']

  After some tracing, it seems nova.servicegroup.drivers.db _report
  state is never called again. Restarting the service correctly restarts
  reporting, too. Probably the reporting thread got stuck?

  Using Liberty from Ubuntu Cloud Archive: 2:12.0.0-0ubuntu2~cloud0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1518927/+subscriptions