yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1621574] [NEW] Instance fails to delete while building in cells v1 due to "ObjectActionError: Object action destroy failed because: host changed"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 08 Sep 2016 18:06:59 -0000
Reply-to: Bug 1621574 <1621574@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Seeing this in a cells v1 job here:

http://logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
cells/1d8af83/logs/screen-n-api.txt.gz#_2016-09-08_15_59_02_553

2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions [req-0f23fed6-0aaf-461f-96d8-f4aa5055b208 tempest-ListServersNegativeTestJSON-1414534476 tempest-ListServersNegativeTestJSON-1414534476] Unexpected exception in API method
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 915, in delete
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self._delete(req.environ['nova.context'], req, id)
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 767, in _delete
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self.compute_api.delete(context, instance)
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 213, in delete
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self._handle_cell_delete(context, instance, 'delete')
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 224, in _handle_cell_delete
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     if self._delete_while_booting(context, instance):
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 1648, in _delete_while_booting
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     instance.destroy()
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 555, in destroy
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     reason='host changed')
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions ObjectActionError: Object action destroy failed because: host changed
2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions 

http://logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
cells/1d8af83/console.html#_2016-09-08_16_13_48_892443

logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
cells/1d8af83/logs/tempest.txt.gz#_2016-09-08_15_59_02_558

In this test it creates a server and then immediately deletes it without
waiting for it to be ACTIVE first.

I wonder if we're racing with any of the BuildRequest stuff that's new
in Newton.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ObjectActionError%3A%20Object%20action%20destroy%20failed%20because%3A%20host%20changed%5C%22%20AND%20message%3A%5C%22_handle_cell_delete%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

10 hits in the last 7 days, check and gate, master (newton) only at this
point.

** Affects: nova
     Importance: Medium
     Assignee: Andrew Laski (alaski)
         Status: Confirmed


** Tags: cells

** Changed in: nova
       Status: New => Confirmed

** Summary changed:

- Instance fails to delete in cells v1 due to "ObjectActionError: Object action destroy failed because: host changed"
+ Instance fails to delete while building in cells v1 due to "ObjectActionError: Object action destroy failed because: host changed"

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1621574

Title:
  Instance fails to delete while building in cells v1 due to
  "ObjectActionError: Object action destroy failed because: host
  changed"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  Seeing this in a cells v1 job here:

  http://logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
  cells/1d8af83/logs/screen-n-api.txt.gz#_2016-09-08_15_59_02_553

  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions [req-0f23fed6-0aaf-461f-96d8-f4aa5055b208 tempest-ListServersNegativeTestJSON-1414534476 tempest-ListServersNegativeTestJSON-1414534476] Unexpected exception in API method
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 915, in delete
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self._delete(req.environ['nova.context'], req, id)
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 767, in _delete
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self.compute_api.delete(context, instance)
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 213, in delete
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     self._handle_cell_delete(context, instance, 'delete')
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 224, in _handle_cell_delete
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     if self._delete_while_booting(context, instance):
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/compute/api.py", line 1648, in _delete_while_booting
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     instance.destroy()
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     return fn(self, *args, **kwargs)
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions   File "/opt/stack/new/nova/nova/objects/instance.py", line 555, in destroy
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions     reason='host changed')
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions ObjectActionError: Object action destroy failed because: host changed
  2016-09-08 15:59:02.553 15924 ERROR nova.api.openstack.extensions 

  http://logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
  cells/1d8af83/console.html#_2016-09-08_16_13_48_892443

  logs.openstack.org/55/367455/1/check/gate-tempest-dsvm-
  cells/1d8af83/logs/tempest.txt.gz#_2016-09-08_15_59_02_558

  In this test it creates a server and then immediately deletes it
  without waiting for it to be ACTIVE first.

  I wonder if we're racing with any of the BuildRequest stuff that's new
  in Newton.

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ObjectActionError%3A%20Object%20action%20destroy%20failed%20because%3A%20host%20changed%5C%22%20AND%20message%3A%5C%22_handle_cell_delete%5C%22%20AND%20tags%3A%5C%22screen-n-api.txt%5C%22&from=7d

  10 hits in the last 7 days, check and gate, master (newton) only at
  this point.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1621574/+subscriptions
Follow ups

[Bug 1621574] Re: Instance fails to delete while building in cells v1 due to "ObjectActionError: Object action destroy failed because: host changed"
From: OpenStack Infra, 2016-09-09