yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1454839] [NEW] cells: error if deleting an instance while host is being set

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Laski <andrew@xxxxxxxxxx>
Date: Wed, 13 May 2015 20:45:28 -0000
Reply-to: Bug 1454839 <1454839@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If a delete request gets past some checks to perform a local delete, but
then has a host set on it before the local delete finishes it will fail
a db constraint check and spew this to the logs:

2015-05-13 11:14:58.666 ERROR nova.api.openstack [req-cb52db9a-d313-49c4-acce-b627b778ccd1 ListServersNegativeTestJSON-1942377486 ListServersNegativeTestJSON-625281866] Caught error: Object action destroy failed because: host changed
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack Traceback (most recent call last):
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return req.get_response(self.application)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self._call_app(env, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     content_type, body, accept)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return method(req=request, **action_args)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 833, in delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._delete(req.environ['nova.context'], req, id)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 668, in _delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self.compute_api.delete(context, instance)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 212, in delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._handle_cell_delete(context, instance, 'delete')
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 226, in _handle_cell_delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._do_delete)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 1766, in _local_delete
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     instance.destroy()
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/base.py", line 205, in wrapper
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return fn(self, *args, **kwargs)
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/instance.py", line 648, in destroy
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     reason='host changed')
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack ObjectActionError: Object action destroy failed because: host changed
2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack 


This should be fixed.

** Affects: nova
     Importance: Low
     Assignee: Andrew Laski (alaski)
         Status: In Progress

** Changed in: nova
   Importance: Undecided => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1454839

Title:
  cells: error if deleting an instance while host is being set

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  If a delete request gets past some checks to perform a local delete,
  but then has a host set on it before the local delete finishes it will
  fail a db constraint check and spew this to the logs:

  2015-05-13 11:14:58.666 ERROR nova.api.openstack [req-cb52db9a-d313-49c4-acce-b627b778ccd1 ListServersNegativeTestJSON-1942377486 ListServersNegativeTestJSON-625281866] Caught error: Object action destroy failed because: host changed
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack Traceback (most recent call last):
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self._call_app(env, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     content_type, body, accept)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return method(req=request, **action_args)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 833, in delete
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._delete(req.environ['nova.context'], req, id)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 668, in _delete
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self.compute_api.delete(context, instance)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 212, in delete
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._handle_cell_delete(context, instance, 'delete')
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 226, in _handle_cell_delete
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     self._do_delete)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 1766, in _local_delete
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     instance.destroy()
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/base.py", line 205, in wrapper
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     return fn(self, *args, **kwargs)
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/instance.py", line 648, in destroy
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack     reason='host changed')
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack ObjectActionError: Object action destroy failed because: host changed
  2015-05-13 11:14:58.666 15399 TRACE nova.api.openstack 

  
  This should be fixed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1454839/+subscriptions
Follow ups

[Bug 1454839] Re: cells: error if deleting an instance while host is being set
From: Thierry Carrez, 2015-06-24
[Bug 1454839] [NEW] cells: error if deleting an instance while host is being set
From: Andrew Laski, 2015-05-13
References

[Bug 1454839] [NEW] cells: error if deleting an instance while host is being set
From: Andrew Laski, 2015-05-13