yahoo-eng-team team mailing list archive

[Manas Mandlekar <manas.mandlekar@xxxxxxxxxx>]

Public bug reported:

Description
===========
In the current implementation we have the method '_destroy_evacuated_instances' in compute.manager which deletes any instances that may have failed a rebuild. This method is called as part of host initialization (init_host) and checks the migration records for VMs which were evacuated. 

There is a possibility that if a VM fails as part of rebuild operation
on destination host after creating a migration record, then when the
source host is brought back up it may end up deleting the VM from source
as well.

To fix this we should check the 'host' attribute in instances table
before deleting the VM and delete VM from source only if the host has
been updated in db after rebuild.

Steps to reproduce
==================
* deploy a VM
* Bring down the host where VM was deployed.
* Evacuate the instance to another host where the rebuild operation may fail (insufficient resources or storage issue)
* This will result in VM not being present on destination host.
* Check that a migration record of type 'evacuation' is present in db.
* Bring the source host up. 

Expected result
===============
The VM should be present on the source host.

Actual result
=============
VM gets deleted as part of evacuated instance cleanup on stat-up of compute service on source host.

Environment
===========
1. Exact version of OpenStack you are running. See the following
Openstack Newton

2. Which hypervisor did you use?
 PowerVM

3. Which networking type did you use?
   Neutron with OpenVSwitch

Logs & Configs
==============
In the logs following message is seen on startup -
2016-10-24 09:32:11.131 3169 INFO nova.compute.manager [req-6611fe85-0515-4cb4-b1c0-3f34f196a0c7 - - - - -] [instance: ed9ca4b9-8938-4d7b-9eec-1dd6ca7bc8c8] Deleting instance as it has been evacuated from this host

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1636825

Title:
  Instances for which rebuild failed get deleted from source host

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  In the current implementation we have the method '_destroy_evacuated_instances' in compute.manager which deletes any instances that may have failed a rebuild. This method is called as part of host initialization (init_host) and checks the migration records for VMs which were evacuated. 

  There is a possibility that if a VM fails as part of rebuild operation
  on destination host after creating a migration record, then when the
  source host is brought back up it may end up deleting the VM from
  source as well.

  To fix this we should check the 'host' attribute in instances table
  before deleting the VM and delete VM from source only if the host has
  been updated in db after rebuild.

  Steps to reproduce
  ==================
  * deploy a VM
  * Bring down the host where VM was deployed.
  * Evacuate the instance to another host where the rebuild operation may fail (insufficient resources or storage issue)
  * This will result in VM not being present on destination host.
  * Check that a migration record of type 'evacuation' is present in db.
  * Bring the source host up. 

  Expected result
  ===============
  The VM should be present on the source host.

  Actual result
  =============
  VM gets deleted as part of evacuated instance cleanup on stat-up of compute service on source host.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
  Openstack Newton

  2. Which hypervisor did you use?
   PowerVM

  3. Which networking type did you use?
     Neutron with OpenVSwitch

  Logs & Configs
  ==============
  In the logs following message is seen on startup -
  2016-10-24 09:32:11.131 3169 INFO nova.compute.manager [req-6611fe85-0515-4cb4-b1c0-3f34f196a0c7 - - - - -] [instance: ed9ca4b9-8938-4d7b-9eec-1dd6ca7bc8c8] Deleting instance as it has been evacuated from this host

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1636825/+subscriptions