yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #82411
[Bug 1636825] Re: Instances for which rebuild failed get deleted from source host
This is indeed fixed upstream, as you can see in the source code there
https://github.com/openstack/nova/blob/2cddf595a8cdedbdb844e800d853ea143817b545/nova/compute/manager.py#L721-L738
We only delete instances if the evacuation was either done, or just precreated.
If the migration wasn't good, then we don't delete the instance.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1636825
Title:
Instances for which rebuild failed get deleted from source host
Status in OpenStack Compute (nova):
Invalid
Bug description:
Description
===========
In the current implementation we have the method '_destroy_evacuated_instances' in compute.manager which deletes any instances from source after they have been evacuated. This method is called as part of host initialization (init_host) and checks the migration records for VMs which were evacuated.
There is a possibility that if a VM fails as part of rebuild operation
on destination host after creating a migration record, then when the
source host is brought back up it may end up deleting the VM from
source as well.
To fix this we should check the 'host' attribute in instances table
before deleting the VM and delete VM from source only if the host has
been updated in db after rebuild.
Steps to reproduce
==================
* deploy a VM
* Bring down the host where VM was deployed.
* Evacuate the instance to another host where the rebuild operation may fail (insufficient resources or storage issue)
* This will result in VM not being present on destination host.
* Check that a migration record of type 'evacuation' is present in db.
* Bring the source host up.
Expected result
===============
The VM should be present on the source host.
Actual result
=============
VM gets deleted as part of evacuated instance cleanup on stat-up of compute service on source host.
Environment
===========
1. Exact version of OpenStack you are running. See the following
Openstack Newton
2. Which hypervisor did you use?
PowerVM
3. Which networking type did you use?
Neutron with OpenVSwitch
Logs & Configs
==============
In the logs following message is seen on startup -
2016-10-24 09:32:11.131 3169 INFO nova.compute.manager [req-6611fe85-0515-4cb4-b1c0-3f34f196a0c7 - - - - -] [instance: ed9ca4b9-8938-4d7b-9eec-1dd6ca7bc8c8] Deleting instance as it has been evacuated from this host
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1636825/+subscriptions
References