yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1641523] [NEW] Nova can't force-delete instance that is stuck in deleting process

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Aidin Alihodzic <ssbljk@xxxxxxxxx>
Date: Mon, 14 Nov 2016 07:48:23 -0000
Reply-to: Bug 1641523 <1641523@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I tried to delete an instance that got into error state during creation. And it's stuck in deleting "Task State". 
I checked with virsh and instance is not present. And after I tried to force-delete it nova client crashed with a message that it should be reported as bug.

Installation is Newton on Centos 7.

Here is part of nova-api.log

2016-11-14 08:37:05.099 4189 INFO nova.osapi_compute.wsgi.server [req-37d453be-6f18-479d-bae8-6b24493a3366 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] 10.6.0.11 "GET /v2.1/ HTTP/1.1" status: 200 len: 714 time: 0.0065050
2016-11-14 08:37:05.569 4189 INFO nova.osapi_compute.wsgi.server [req-9a5d94a1-a2cf-45b8-bfac-5f231f3e07b7 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] 10.6.0.11 "GET /v2.1/43dec04d0696450787fee3580b9780de/servers/198b4ce6-84cc-4a45-8eda-d0b3262f3df4 HTTP/1.1" status: 200 len: 2243 time: 0.2764120
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions [req-fa12df45-bf9b-471b-8914-24a57427e358 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] Unexpected exception in API method
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/deferred_delete.py", line 64, in _force_delete
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     self.compute_api.force_delete(context, instance)
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 165, in inner
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 138, in inner
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     method=f.__name__)
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions InstanceInvalidState: Instance 198b4ce6-84cc-4a45-8eda-d0b3262f3df4 in task_state deleting. Cannot force_delete while the instance is in this state.
2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions 
2016-11-14 08:37:05.651 4189 INFO nova.api.openstack.wsgi [req-fa12df45-bf9b-471b-8914-24a57427e358 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.InstanceInvalidState'>

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1641523

Title:
  Nova can't force-delete instance that is stuck in deleting process

Status in OpenStack Compute (nova):
  New

Bug description:
  I tried to delete an instance that got into error state during creation. And it's stuck in deleting "Task State". 
  I checked with virsh and instance is not present. And after I tried to force-delete it nova client crashed with a message that it should be reported as bug.

  Installation is Newton on Centos 7.

  Here is part of nova-api.log

  2016-11-14 08:37:05.099 4189 INFO nova.osapi_compute.wsgi.server [req-37d453be-6f18-479d-bae8-6b24493a3366 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] 10.6.0.11 "GET /v2.1/ HTTP/1.1" status: 200 len: 714 time: 0.0065050
  2016-11-14 08:37:05.569 4189 INFO nova.osapi_compute.wsgi.server [req-9a5d94a1-a2cf-45b8-bfac-5f231f3e07b7 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] 10.6.0.11 "GET /v2.1/43dec04d0696450787fee3580b9780de/servers/198b4ce6-84cc-4a45-8eda-d0b3262f3df4 HTTP/1.1" status: 200 len: 2243 time: 0.2764120
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions [req-fa12df45-bf9b-471b-8914-24a57427e358 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] Unexpected exception in API method
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/deferred_delete.py", line 64, in _force_delete
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     self.compute_api.force_delete(context, instance)
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 165, in inner
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 138, in inner
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions     method=f.__name__)
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions InstanceInvalidState: Instance 198b4ce6-84cc-4a45-8eda-d0b3262f3df4 in task_state deleting. Cannot force_delete while the instance is in this state.
  2016-11-14 08:37:05.651 4189 ERROR nova.api.openstack.extensions 
  2016-11-14 08:37:05.651 4189 INFO nova.api.openstack.wsgi [req-fa12df45-bf9b-471b-8914-24a57427e358 24bef1856bf643c2bda92b710a7a6fcd 43dec04d0696450787fee3580b9780de - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'nova.exception.InstanceInvalidState'>

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1641523/+subscriptions
Follow ups

[Bug 1641523] Re: Nova can't force-delete instance that is stuck in deleting process
From: Ghanshyam Mann, 2018-11-28
[Bug 1641523] Re: Nova can't force-delete instance that is stuck in deleting process
From: Matt Riedemann, 2016-12-01