yahoo-eng-team team mailing list archive

[zhangyujun <1841509@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

I found an instance disappeared after do restore instance, check the
nova code and log, I think its a logic bug here

1. restore instance with power on failed

nova-api `restore` set  `instance.task_state = task_states.RESTORING instance.deleted_at = None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344

nova-compute `restore_instance`  will call `self._power_on` if virt driver did not implement the `restore` method
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009

instance state will be set to None if any exceptions raise when call `self._power_on` in `reverts_task_state`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L178

finally the instnace state will be set to
{vm_state=vm_state.SOFT_DELETED, task_state=None, deleted_at=None}

2. reclaim instance

nova-compute periodic task `_reclaim_queued_deletes` running every 60s,
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8209

it will select instance with filte `{'vm_state': vm_states.SOFT_DELETED, 'task_state': None,'host': self.host}`,  the instance of step 1 will be slected
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8216

and it will be in the return list of `_deleted_old_enough` with its `deleted_at=None`
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8430

and then be deleted soon
https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8229

I don't think the instance should be reclaimed with the above situation

** Affects: nova
     Importance: Undecided
     Assignee: zhangyujun (zhangyujun)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => zhangyujun (zhangyujun)

** Description changed:

  I found an instance disappeared after do restore instance, check the
  nova code and log, I think its a logic bug here
  
  1. restore instance with power on failed
  
  nova-api `restore` set  `instance.task_state = task_states.RESTORING instance.deleted_at = None`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344
  
  nova-compute `restore_instance`  will call `self._power_on` if virt driver did not implement the `restore` method
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009
  
  instance state will be set to None if any exceptions raise when call `self._power_on` in `reverts_task_state`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L178
  
  finally the instnace state will be set to
- {vm_state=vm_state.SOFT_DELETED, task_state=None, deleted=None}
+ {vm_state=vm_state.SOFT_DELETED, task_state=None, deleted_at=None}
  
  2. reclaim instance
  
  nova-compute periodic task `_reclaim_queued_deletes` running every 60s,
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8209
  
  it will select instance with filte `{'vm_state': vm_states.SOFT_DELETED, 'task_state': None,'host': self.host}`,  the instance of step 1 will be slected
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8216
  
  and it will be in the return list of `_deleted_old_enough` with its `deleted_at=None`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8430
  
  and then be deleted soon
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8229
  
  I don't think the instance should be reclaimed with the above situation

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1841509

Title:
  soft delete instance will be reclaimed if power on failed when do
  restore

Status in OpenStack Compute (nova):
  New

Bug description:
  I found an instance disappeared after do restore instance, check the
  nova code and log, I think its a logic bug here

  1. restore instance with power on failed

  nova-api `restore` set  `instance.task_state = task_states.RESTORING instance.deleted_at = None`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/api.py#L2344

  nova-compute `restore_instance`  will call `self._power_on` if virt driver did not implement the `restore` method
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L3009

  instance state will be set to None if any exceptions raise when call `self._power_on` in `reverts_task_state`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L178

  finally the instnace state will be set to
  {vm_state=vm_state.SOFT_DELETED, task_state=None, deleted_at=None}

  2. reclaim instance

  nova-compute periodic task `_reclaim_queued_deletes` running every 60s,
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8209

  it will select instance with filte `{'vm_state': vm_states.SOFT_DELETED, 'task_state': None,'host': self.host}`,  the instance of step 1 will be slected
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8216

  and it will be in the return list of `_deleted_old_enough` with its `deleted_at=None`
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8430

  and then be deleted soon
  https://github.com/openstack/nova/blob/4b8b4217fed897755f742afcb42f7994aea4c9a1/nova/compute/manager.py#L8229

  I don't think the instance should be reclaimed with the above
  situation

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1841509/+subscriptions