yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1241665

Title:
  Mysql losing connection can result in running instance stuck in BUILD

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  I ran into a situation where an instance starting up and getting
  through the whole spawning process, was actually left in BUILD and
  never recovered:

  2013-10-18 14:28:27.781 2566 DEBUG nova.virt.libvirt.driver [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Instance is running spawn /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:2072
  2013-10-18 14:28:27.787 2566 INFO nova.virt.libvirt.driver [-] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Instance spawned successfully.
  2013-10-18 14:28:27.794 2566 DEBUG nova.compute.manager [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:801
  2013-10-18 14:28:27.826 2566 WARNING nova.openstack.common.db.sqlalchemy.session [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')

  It looks like this query was never retried and the result is
  inconsistent. The instance itself is running, getting all the traffic,
  disk files are periodically checked by nova-compute, but the state is
  not changing. Restarting nova-compute doesn't correct the status
  either:

  2013-10-18 15:13:51.635 21590 DEBUG nova.compute.manager [req-d438f27e-13a8-4a59-b238-66b2227e4415 None None] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:801
  2013-10-18 15:13:51.642 21590 DEBUG nova.compute.manager [req-d438f27e-13a8-4a59-b238-66b2227e4415 None None] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Current state is 1, state in DB is 1. _init_instance /usr/lib/python2.7/dist-packages/nova/compute/manager.py:702

  Mysql connection disappeared only for a moment - the next query
  reconnected / worked without any issues.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1241665/+subscriptions