yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1288926] [NEW] incorrect error code when rebooting a rebooting_hard guest

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sarath Menon <1288926@xxxxxxxxxxxxxxxxxx>
Date: Thu, 06 Mar 2014 18:49:24 -0000
Reply-to: Bug 1288926 <1288926@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

This is using the latest nova from trunk. In our deployment, we had a
hypervisor go down and the tenant issued a hard reboot prior. When
attempting a reboot on a guest with the state HARD_REBOOT, nova
controller throws this error in it's logs and returns 'ERROR: The server
has either erred or is incapable of performing the requested operation.
(HTTP 500)' to the user:


2014-03-06 18:21:00,535 (routes.middleware): DEBUG middleware __call__ Matched POST /tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action
2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>}
2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>, 'project_id': u'tenant1', 'id': u'778032b2-469d-445e-abde-7b9b0b673324'}
2014-03-06 18:21:00,537 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Action: 'action', body: {"reboot": {"type": "SOFT"}}
2014-03-06 18:21:00,538 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Calling method <bound method Controller._action_reboot of <nova.api.openstack.compute.contrib.keypairs.Controller object at 0x4c35a50>>
2014-03-06 18:21:00,747 (nova.api.openstack): ERROR __init__ _error Caught error: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
Traceback (most recent call last):
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
    return req.get_response(self.application)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 598, in __call__
    return self.app(env, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 925, in __call__
    content_type, body, accept)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 987, in _process_stack
    action_result = self.dispatch(meth, request, action_args)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1074, in dispatch
    return method(req=request, **action_args)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1145, in _action_reboot
    self.compute_api.reboot(context, instance, reboot_type)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 199, in wrapped
    return func(self, context, target, *args, **kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 189, in inner
    return function(self, context, instance, *args, **kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 170, in inner
    return f(self, context, instance, *args, **kw)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 2073, in reboot
    instance.save(expected_task_state=[None, task_states.REBOOTING])
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/base.py", line 151, in wrapper
    return fn(self, ctxt, *args, **kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/instance.py", line 472, in save
    columns_to_join=_expected_cols(expected_attrs))
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/api.py", line 739, in instance_update_and_get_original
    columns_to_join=columns_to_join)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 128, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2164, in instance_update_and_get_original
    columns_to_join=columns_to_join)
  File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2215, in _instance_update
    actual=actual_state, expected=expected)
UnexpectedTaskStateError: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
2014-03-06 18:21:00,750 (nova.api.openstack): INFO __init__ _error http://nova-controller.isg.apple.com:8774/v2/tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action returned with HTTP 500


The actual error message back to the user must be something along the lines of 'Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard' with a 4xx HTTP code.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1288926

Title:
  incorrect error code when rebooting a rebooting_hard guest

Status in OpenStack Compute (Nova):
  New

Bug description:
  This is using the latest nova from trunk. In our deployment, we had a
  hypervisor go down and the tenant issued a hard reboot prior. When
  attempting a reboot on a guest with the state HARD_REBOOT, nova
  controller throws this error in it's logs and returns 'ERROR: The
  server has either erred or is incapable of performing the requested
  operation. (HTTP 500)' to the user:

  
  2014-03-06 18:21:00,535 (routes.middleware): DEBUG middleware __call__ Matched POST /tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action
  2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Route path: '/{project_id}/servers/:(id)/action', defaults: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>}
  2014-03-06 18:21:00,536 (routes.middleware): DEBUG middleware __call__ Match dict: {'action': u'action', 'controller': <nova.api.openstack.wsgi.Resource object at 0x5242c90>, 'project_id': u'tenant1', 'id': u'778032b2-469d-445e-abde-7b9b0b673324'}
  2014-03-06 18:21:00,537 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Action: 'action', body: {"reboot": {"type": "SOFT"}}
  2014-03-06 18:21:00,538 (nova.api.openstack.wsgi): DEBUG wsgi _process_stack Calling method <bound method Controller._action_reboot of <nova.api.openstack.compute.contrib.keypairs.Controller object at 0x4c35a50>>
  2014-03-06 18:21:00,747 (nova.api.openstack): ERROR __init__ _error Caught error: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
  Traceback (most recent call last):
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
      return req.get_response(self.application)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 598, in __call__
      return self.app(env, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 925, in __call__
      content_type, body, accept)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 987, in _process_stack
      action_result = self.dispatch(meth, request, action_args)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1074, in dispatch
      return method(req=request, **action_args)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1145, in _action_reboot
      self.compute_api.reboot(context, instance, reboot_type)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 199, in wrapped
      return func(self, context, target, *args, **kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 189, in inner
      return function(self, context, instance, *args, **kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 170, in inner
      return f(self, context, instance, *args, **kw)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/compute/api.py", line 2073, in reboot
      instance.save(expected_task_state=[None, task_states.REBOOTING])
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/base.py", line 151, in wrapper
      return fn(self, ctxt, *args, **kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/objects/instance.py", line 472, in save
      columns_to_join=_expected_cols(expected_attrs))
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/api.py", line 739, in instance_update_and_get_original
      columns_to_join=columns_to_join)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 128, in wrapper
      return f(*args, **kwargs)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2164, in instance_update_and_get_original
      columns_to_join=columns_to_join)
    File "/usr/local/gshare/csi-nova.venv/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 2215, in _instance_update
      actual=actual_state, expected=expected)
  UnexpectedTaskStateError: Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard
  2014-03-06 18:21:00,750 (nova.api.openstack): INFO __init__ _error http://nova-controller.isg.apple.com:8774/v2/tenant1/servers/778032b2-469d-445e-abde-7b9b0b673324/action returned with HTTP 500

  
  The actual error message back to the user must be something along the lines of 'Unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard' with a 4xx HTTP code.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1288926/+subscriptions
Follow ups

[Bug 1288926] Re: incorrect error code when rebooting a rebooting_hard guest
From: Thierry Carrez, 2014-03-31
[Bug 1288926] [NEW] incorrect error code when rebooting a rebooting_hard guest
From: Sarath Menon, 2014-03-06
References

[Bug 1288926] [NEW] incorrect error code when rebooting a rebooting_hard guest
From: Sarath Menon, 2014-03-06