yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => havana-3

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1212420

Title:
  some exceptions raised in terminate_instance() wedge the instance in
  the 'deleting' state

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  Some exceptions raised by _delete_instance() in nova-compute's manager
  cause terminate_instance() to leave the instance task as 'deleting'.
  The task entry keeps nova-compute from noticing further power state
  transitions via _sync_power_states(), wedging the virtual machine in
  the 'deleting' state.

  This bug affects the nova-compute manager module:
  https://github.com/openstack/nova/blob/master/nova/compute/manager.py

  Bug 1177584 may be an example of this in action:

   * terminate_instance() calls the baremetal driver to destroy an instance
   * a slow system call in the baremetal driver.destroy() operation raises an InstancePowerOffFailure exception
   * the exception bubbles back up to terminate_instance(), which ignores it
   * the instance is left in power_state.RUNNING, vm_state.ACTIVE, task_state.DELETING
   * _sync_power_states() ignores further power state changes because of the active task

  This could happen with any virtual machine driver that raises an
  exception during a call to its driver.destroy() method.

  It's worth noting that keeping the task as-is after exceptions in
  terminate_instance() was done on purpose to resolve bug 1046236 (see
  the terminate_instance() source for details).

  Reproducing this bug is difficult: terminate_instance() handles some
  exceptions and not others, and the same applies to the try/except
  blocks in _delete_instance() and _shutdown_instance().   However, it
  is possible to reproduce this bug in a unit test that mocks out
  _shudown_instance() and raises an InstancePowerOffFailure.  A poisoned
  driver that injects an InstancePowerOffFailure would also work.

  Reported against nova master: commit
  8fb450fb3aa033d42c5dddb907392efd70f54a6b

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1212420/+subscriptions