← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #32376

[Bug 1448316] [NEW] cells: Object action destroy failed because: host changed

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
cells/ce91d7e/logs/screen-n-api.txt.gz?level=TRACE#_2015-04-24_15_21_05_563

2015-04-24 15:21:05.563 ERROR nova.api.openstack [req-99d82867-7bd3-4ca7-9102-27bacb4ab6a0 ListServersNegativeTestJSON-1811537006 ListServersNegativeTestJSON-968745101] Caught error: Object action destroy failed because: host changed
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack Traceback (most recent call last):
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return req.get_response(self.application)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self._call_app(env, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     content_type, body, accept)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return method(req=request, **action_args)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 834, in delete
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._delete(req.environ['nova.context'], req, id)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 669, in _delete
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self.compute_api.delete(context, instance)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 212, in delete
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._handle_cell_delete(context, instance, 'delete')
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 226, in _handle_cell_delete
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._do_delete)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 1766, in _local_delete
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     instance.destroy()
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/base.py", line 208, in wrapper
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return fn(self, *args, **kwargs)
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/instance.py", line 646, in destroy
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     reason='host changed')
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack ObjectActionError: Object action destroy failed because: host changed
2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack 


http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2hhbmRsZV9jZWxsX2RlbGV0ZVwiIEFORCBtZXNzYWdlOlwiT2JqZWN0QWN0aW9uRXJyb3I6IE9iamVjdCBhY3Rpb24gZGVzdHJveSBmYWlsZWQgYmVjYXVzZTogaG9zdCBjaGFuZ2VkXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1hcGkudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mjk5MDk4OTQ5NDl9

21 hits in 7 days.  It's 90% failure when it happens though.

I also noticed this in the compute logs:

http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
cells/ce91d7e/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-04-24_15_16_26_866

A separate query on that shows it happens quite a bit though in normal
gate runs:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gY29tcHV0ZSBub2RlIHJlY29yZCBmb3IgaG9zdFwiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5OTA4OTUwOTE2LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: cells unified-objects

** Tags added: cells unified-objects

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1448316

Title:
  cells: Object action destroy failed because: host changed

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
  cells/ce91d7e/logs/screen-n-api.txt.gz?level=TRACE#_2015-04-24_15_21_05_563

  2015-04-24 15:21:05.563 ERROR nova.api.openstack [req-99d82867-7bd3-4ca7-9102-27bacb4ab6a0 ListServersNegativeTestJSON-1811537006 ListServersNegativeTestJSON-968745101] Caught error: Object action destroy failed because: host changed
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack Traceback (most recent call last):
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 125, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return req.get_response(self.application)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     application, catch_exc_info=False)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 639, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self._call_app(env, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 559, in _call_app
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self._app(env, _fake_start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return resp(environ, start_response)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 756, in __call__
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     content_type, body, accept)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return method(req=request, **action_args)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 834, in delete
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._delete(req.environ['nova.context'], req, id)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 669, in _delete
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self.compute_api.delete(context, instance)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 212, in delete
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._handle_cell_delete(context, instance, 'delete')
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/cells_api.py", line 226, in _handle_cell_delete
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     self._do_delete)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 1766, in _local_delete
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     instance.destroy()
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/base.py", line 208, in wrapper
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     return fn(self, *args, **kwargs)
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/objects/instance.py", line 646, in destroy
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack     reason='host changed')
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack ObjectActionError: Object action destroy failed because: host changed
  2015-04-24 15:21:05.563 32393 TRACE nova.api.openstack 

  
  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2hhbmRsZV9jZWxsX2RlbGV0ZVwiIEFORCBtZXNzYWdlOlwiT2JqZWN0QWN0aW9uRXJyb3I6IE9iamVjdCBhY3Rpb24gZGVzdHJveSBmYWlsZWQgYmVjYXVzZTogaG9zdCBjaGFuZ2VkXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1hcGkudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0Mjk5MDk4OTQ5NDl9

  21 hits in 7 days.  It's 90% failure when it happens though.

  I also noticed this in the compute logs:

  http://logs.openstack.org/58/175458/3/check/check-tempest-dsvm-
  cells/ce91d7e/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-04-24_15_16_26_866

  A separate query on that shows it happens quite a bit though in normal
  gate runs:

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTm8gY29tcHV0ZSBub2RlIHJlY29yZCBmb3IgaG9zdFwiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5OTA4OTUwOTE2LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1448316/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next