yahoo-eng-team team mailing list archive

[Oleksandr Savatieiev <osavatieiev@xxxxxxxxxxxx>]

Hit this issue on Queens. The behavior is the same as above.

Tried already with no success:
- Reset state
- Forced
- Investigating logs shows that nova keeps "checking" those for something
- Once nova-compute service restarted it starts the instance update task and deletes those fake instances.

My personal best guess is that something wrong with the communication
channel while Nova compute has its hands full running instances
(degraded performance)

** Changed in: nova
       Status: Invalid => Confirmed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1461055

Title:
  Can't delete instance stuck in deleting task

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Description of problem:  On a Juno HA deployment, nova over shared nfs
  storage, when I deleted an instance it was "deleted":

  2015-06-02 11:57:36.273 3505 INFO nova.virt.libvirt.driver [req-
  4cc54412-a449-4c7a-bbe1-b21d202bcfe7 None] [instance:
  7b6c8ad5-7633-4d53-9f84-93b12a701cd3] Deletion of
  /var/lib/nova/instances/7b6c8ad5-7633-4d53-9f84-93b12a701cd3_del
  complete

  Also instance wasn't found with virsh list all. 
  Yet nova list and Horizon both still show this instance as stuck in task deleting, two hours+ pasted since I deleted it. 

  Version-Release number of selected component (if applicable):
  rhel 7.1
  python-nova-2014.2.2-19.el7ost.noarch
  openstack-nova-compute-2014.2.2-19.el7ost.noarch
  python-novaclient-2.20.0-1.el7ost.noarch
  openstack-nova-common-2014.2.2-19.el7ost.noarch

  How reproducible:
  Unsure, it doesn't happen with every instance deletion, but happened more than this one time. 

  Steps to Reproduce:
  1. Boot an instance
  2. Delete instance 
  3. Instance is stuck in deleting task on nova/Horozon. 

  Actual results:
  Stuck with a phantom "deleting" instance, which is basically already dead from Virsh's point of view. 

  Expected results:
  Instance should get deleted including from nova list/Horizon. 

  Additional info:

  
  Workaround doing openstack-service restart for nova on compute node fixed my problem. Instance is totally gone from Nova/Horizon. 

  instance virsh id instance-00000d4d.log
  instanceID  7b6c8ad5-7633-4d53-9f84-93b12a701cd3

  | OS-EXT-STS:power_state               | 1                                                                                                  |
  | OS-EXT-STS:task_state                | deleting                                                                                           |
  | OS-EXT-STS:vm_state                  | deleted                                                                                            |
  | OS-SRV-USG:launched_at               | 2015-05-28T11:06:33.000000                                                                         |
  | OS-SRV-USG:terminated_at             | 2015-06-02T08:57:37.000000                                                                         |
  ......                                 |
  | status                               | DELETED

  Attached nova log from compute and controller.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1461055/+subscriptions