yahoo-eng-team team mailing list archive

[Tzach Shefi <tshefi@xxxxxxxxxx>]

Public bug reported:

Description of problem:  On a Juno HA deployment, nova over shared nfs
storage, when I deleted an instance it was "deleted":

2015-06-02 11:57:36.273 3505 INFO nova.virt.libvirt.driver [req-
4cc54412-a449-4c7a-bbe1-b21d202bcfe7 None] [instance:
7b6c8ad5-7633-4d53-9f84-93b12a701cd3] Deletion of
/var/lib/nova/instances/7b6c8ad5-7633-4d53-9f84-93b12a701cd3_del
complete

Also instance wasn't found with virsh list all. 
Yet nova list and Horizon both still show this instance as stuck in task deleting, two hours+ pasted since I deleted it. 

Version-Release number of selected component (if applicable):
rhel 7.1
python-nova-2014.2.2-19.el7ost.noarch
openstack-nova-compute-2014.2.2-19.el7ost.noarch
python-novaclient-2.20.0-1.el7ost.noarch
openstack-nova-common-2014.2.2-19.el7ost.noarch

How reproducible:
Unsure, it doesn't happen with every instance deletion, but happened more than this one time. 

Steps to Reproduce:
1. Boot an instance
2. Delete instance 
3. Instance is stuck in deleting task on nova/Horozon. 

Actual results:
Stuck with a phantom "deleting" instance, which is basically already dead from Virsh's point of view. 

Expected results:
Instance should get deleted including from nova list/Horizon. 

Additional info:


Workaround doing openstack-service restart for nova on compute node fixed my problem. Instance is totally gone from Nova/Horizon. 

instance virsh id instance-00000d4d.log
instanceID  7b6c8ad5-7633-4d53-9f84-93b12a701cd3

| OS-EXT-STS:power_state               | 1                                                                                                  |
| OS-EXT-STS:task_state                | deleting                                                                                           |
| OS-EXT-STS:vm_state                  | deleted                                                                                            |
| OS-SRV-USG:launched_at               | 2015-05-28T11:06:33.000000                                                                         |
| OS-SRV-USG:terminated_at             | 2015-06-02T08:57:37.000000                                                                         |
......                                 |
| status                               | DELETED

Attached nova log from compute and controller.

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "logs"
   https://bugs.launchpad.net/bugs/1461055/+attachment/4408565/+files/logs.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1461055

Title:
  Can't delete instance stuck in deleting task

Status in OpenStack Compute (Nova):
  New

Bug description:
  Description of problem:  On a Juno HA deployment, nova over shared nfs
  storage, when I deleted an instance it was "deleted":

  2015-06-02 11:57:36.273 3505 INFO nova.virt.libvirt.driver [req-
  4cc54412-a449-4c7a-bbe1-b21d202bcfe7 None] [instance:
  7b6c8ad5-7633-4d53-9f84-93b12a701cd3] Deletion of
  /var/lib/nova/instances/7b6c8ad5-7633-4d53-9f84-93b12a701cd3_del
  complete

  Also instance wasn't found with virsh list all. 
  Yet nova list and Horizon both still show this instance as stuck in task deleting, two hours+ pasted since I deleted it. 

  Version-Release number of selected component (if applicable):
  rhel 7.1
  python-nova-2014.2.2-19.el7ost.noarch
  openstack-nova-compute-2014.2.2-19.el7ost.noarch
  python-novaclient-2.20.0-1.el7ost.noarch
  openstack-nova-common-2014.2.2-19.el7ost.noarch

  How reproducible:
  Unsure, it doesn't happen with every instance deletion, but happened more than this one time. 

  Steps to Reproduce:
  1. Boot an instance
  2. Delete instance 
  3. Instance is stuck in deleting task on nova/Horozon. 

  Actual results:
  Stuck with a phantom "deleting" instance, which is basically already dead from Virsh's point of view. 

  Expected results:
  Instance should get deleted including from nova list/Horizon. 

  Additional info:

  
  Workaround doing openstack-service restart for nova on compute node fixed my problem. Instance is totally gone from Nova/Horizon. 

  instance virsh id instance-00000d4d.log
  instanceID  7b6c8ad5-7633-4d53-9f84-93b12a701cd3

  | OS-EXT-STS:power_state               | 1                                                                                                  |
  | OS-EXT-STS:task_state                | deleting                                                                                           |
  | OS-EXT-STS:vm_state                  | deleted                                                                                            |
  | OS-SRV-USG:launched_at               | 2015-05-28T11:06:33.000000                                                                         |
  | OS-SRV-USG:terminated_at             | 2015-06-02T08:57:37.000000                                                                         |
  ......                                 |
  | status                               | DELETED

  Attached nova log from compute and controller.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1461055/+subscriptions