yahoo-eng-team team mailing list archive

[Balazs Gibizer <1978983@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
A compute host dies but before anything notices it a VM that was running on that host is requested to be stopped by the user. The VM task_state is set to powering-off and the shutdown RPC is sent to the dead compute. A bit later the monitoring system detect that the compute is dead and fences the compute, set the compute to forced_down in nova and triggers the evacuation of the VM. However the evacuation is rejected by nova:

Cannot 'evacuate' instance 81451eb2-4600-4036-a6f1-b99139f0d277 while it
is in task_state powering-off (HTTP 409) (Request-ID:
req-363ca0a3-0d68-42f6-95d2-122bd2a53463)


Steps to reproduce
==================
0) deploy a multi node devstack
1) create a VM
   $openstack --os-compute-api-version 2.80 server create --image cirros-0.5.2-x86_64-disk --flavor c1 --nic net-id=public  --use-config-drive vm1 --wait
2) stop the nova-compute service of the host the VM is scheduled to:
   $sudo systemctl stop devstack@n-cpu
3) stop the VM
   $openstack server stop vm1
4) fence the host and force the host down in nova
5) try to evacuate the VM
   $server evacuate vm1

See also [1]

Expected result
===============
The VM is evacuated successfully 

Actual result
=============
Cannot 'evacuate' instance 81451eb2-4600-4036-a6f1-b99139f0d277 while it is in task_state powering-off (HTTP 409) (Request-ID: req-363ca0a3-0d68-42f6-95d2-122bd2a53463)

Environment
===========
devstack on recent master

Workaround
==========
The admin can reset the state of the VM with 
$nova reset-state --active vm1 
then retry the evacuation.

[1] https://paste.opendev.org/show/bQphEfOf8eLBnM6XmleQ/
[2] https://paste.opendev.org/show/bVI7D8H5g9Oqjjo4rKfk/

** Affects: nova
     Importance: Medium
         Status: Triaged


** Tags: evacuate

** Tags added: evacuate

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1978983

Title:
  evacuate is not possible if the instance has task_state

Status in OpenStack Compute (nova):
  Triaged

Bug description:
  Description
  ===========
  A compute host dies but before anything notices it a VM that was running on that host is requested to be stopped by the user. The VM task_state is set to powering-off and the shutdown RPC is sent to the dead compute. A bit later the monitoring system detect that the compute is dead and fences the compute, set the compute to forced_down in nova and triggers the evacuation of the VM. However the evacuation is rejected by nova:

  Cannot 'evacuate' instance 81451eb2-4600-4036-a6f1-b99139f0d277 while
  it is in task_state powering-off (HTTP 409) (Request-ID:
  req-363ca0a3-0d68-42f6-95d2-122bd2a53463)


  Steps to reproduce
  ==================
  0) deploy a multi node devstack
  1) create a VM
     $openstack --os-compute-api-version 2.80 server create --image cirros-0.5.2-x86_64-disk --flavor c1 --nic net-id=public  --use-config-drive vm1 --wait
  2) stop the nova-compute service of the host the VM is scheduled to:
     $sudo systemctl stop devstack@n-cpu
  3) stop the VM
     $openstack server stop vm1
  4) fence the host and force the host down in nova
  5) try to evacuate the VM
     $server evacuate vm1

  See also [1]

  Expected result
  ===============
  The VM is evacuated successfully 

  Actual result
  =============
  Cannot 'evacuate' instance 81451eb2-4600-4036-a6f1-b99139f0d277 while it is in task_state powering-off (HTTP 409) (Request-ID: req-363ca0a3-0d68-42f6-95d2-122bd2a53463)

  Environment
  ===========
  devstack on recent master

  Workaround
  ==========
  The admin can reset the state of the VM with 
  $nova reset-state --active vm1 
  then retry the evacuation.

  [1] https://paste.opendev.org/show/bQphEfOf8eLBnM6XmleQ/
  [2] https://paste.opendev.org/show/bVI7D8H5g9Oqjjo4rKfk/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1978983/+subscriptions