yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1251446] Re: HTTP 500 error when running "nova list" after running out of RAM on compute node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Chris Friesen <cbf123@xxxxxxxxxxxxx>
Date: Thu, 14 Nov 2013 23:36:17 -0000
Reply-to: Bug 1251446 <1251446@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Never mind, seems like the stuff I thought was unrelated was in fact
related.

** Changed in: nova
       Status: New => Invalid

** Description changed:

+ This is likely due to some patches I'm testing, so probably not due to
+ upstream code.  Original description included below for completeness.
+ 
+ Sorry for the noise...
+ 
+ Chris
+ 
+ 
+ 
  I'm running devstack, running havana with a couple of unrelated local
  tweaks.
  
  I was playing with the instance group affinity stuff, trying to boot 4
  instances at a time.  It was unable to boot up all the instances due to
  a lack of RAM on the compute node.
  
  Now when I run "nova list", I get:
  
-  ERROR: The server has either erred or is incapable of performing the
+  ERROR: The server has either erred or is incapable of performing the
  requested operation. (HTTP 500) (Request-ID: req-aaecebbc-6074-495d-
  8b21-5822c5a4b5db)
  
  The logs in nova-api are as follows:
  
  2013-11-13 22:25:08.179 DEBUG routes.middleware [-] Matched GET /6c1cae5b7bce4fb4bb6e5a280e8bcab3/servers/detail from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
  2013-11-13 22:25:08.180 DEBUG routes.middleware [-] Route path: '/{project_id}/servers/detail', defaults: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x49bd450>} from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
  2013-11-13 22:25:08.180 DEBUG routes.middleware [-] Match dict: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x49bd450>, 'project_id': u'6c1cae5b7bce4fb4bb6e5a280e8bcab3'} from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
  2013-11-13 22:25:08.180 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] No Content-Type provided in request from (pid=17745) get_body /opt/stack/nova/nova/api/openstack/wsgi.py:835
  2013-11-13 22:25:08.181 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Calling method <bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x47d26d0>> from (pid=17745) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:962
  2013-11-13 22:25:08.182 DEBUG nova.compute.api [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Searching by: {'deleted': False, 'project_id': u'6c1cae5b7bce4fb4bb6e5a280e8bcab3'} from (pid=17745) get_all /opt/stack/nova/nova/compute/api.py:1695
  2013-11-13 22:25:08.224 ERROR nova.api.openstack [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Caught error: 'instance_type_memory_mb'
  2013-11-13 22:25:08.224 TRACE nova.api.openstack Traceback (most recent call last):
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 119, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 571, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 938, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     content_type, body, accept)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 997, in _process_stack
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1078, in dispatch
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return method(req=request, **action_args)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 502, in detail
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     servers = self._get_servers(req, is_detail=True)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 579, in _get_servers
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     response = self._view_builder.detail(req, instance_list)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 126, in detail
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self._list_view(self.show, request, instances)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 130, in _list_view
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     server_list = [func(request, server)["server"] for server in servers]
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 98, in show
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     "flavor": self._get_flavor(request, instance),
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 186, in _get_flavor
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     instance_type = flavors.extract_flavor(instance)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/flavors.py", line 282, in extract_flavor
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     instance_type[key] = type_fn(sys_meta[type_key])
  2013-11-13 22:25:08.224 TRACE nova.api.openstack KeyError: 'instance_type_memory_mb'
- 2013-11-13 22:25:08.224 TRACE nova.api.openstack 
+ 2013-11-13 22:25:08.224 TRACE nova.api.openstack
  2013-11-13 22:25:08.225 INFO nova.api.openstack [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] http://192.168.2.109:8774/v2/6c1cae5b7bce4fb4bb6e5a280e8bcab3/servers/detail returned with HTTP 500
  2013-11-13 22:25:08.226 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=17745) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1224

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1251446

Title:
  HTTP 500 error when running "nova list" after running out of RAM on
  compute node

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  This is likely due to some patches I'm testing, so probably not due to
  upstream code.  Original description included below for completeness.

  Sorry for the noise...

  Chris



  I'm running devstack, running havana with a couple of unrelated local
  tweaks.

  I was playing with the instance group affinity stuff, trying to boot 4
  instances at a time.  It was unable to boot up all the instances due
  to a lack of RAM on the compute node.

  Now when I run "nova list", I get:

   ERROR: The server has either erred or is incapable of performing the
  requested operation. (HTTP 500) (Request-ID: req-aaecebbc-6074-495d-
  8b21-5822c5a4b5db)

  The logs in nova-api are as follows:

  2013-11-13 22:25:08.179 DEBUG routes.middleware [-] Matched GET /6c1cae5b7bce4fb4bb6e5a280e8bcab3/servers/detail from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
  2013-11-13 22:25:08.180 DEBUG routes.middleware [-] Route path: '/{project_id}/servers/detail', defaults: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x49bd450>} from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
  2013-11-13 22:25:08.180 DEBUG routes.middleware [-] Match dict: {'action': u'detail', 'controller': <nova.api.openstack.wsgi.Resource object at 0x49bd450>, 'project_id': u'6c1cae5b7bce4fb4bb6e5a280e8bcab3'} from (pid=17745) __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
  2013-11-13 22:25:08.180 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] No Content-Type provided in request from (pid=17745) get_body /opt/stack/nova/nova/api/openstack/wsgi.py:835
  2013-11-13 22:25:08.181 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Calling method <bound method Controller.detail of <nova.api.openstack.compute.servers.Controller object at 0x47d26d0>> from (pid=17745) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:962
  2013-11-13 22:25:08.182 DEBUG nova.compute.api [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Searching by: {'deleted': False, 'project_id': u'6c1cae5b7bce4fb4bb6e5a280e8bcab3'} from (pid=17745) get_all /opt/stack/nova/nova/compute/api.py:1695
  2013-11-13 22:25:08.224 ERROR nova.api.openstack [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Caught error: 'instance_type_memory_mb'
  2013-11-13 22:25:08.224 TRACE nova.api.openstack Traceback (most recent call last):
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/__init__.py", line 119, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 571, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 938, in __call__
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     content_type, body, accept)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 997, in _process_stack
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1078, in dispatch
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return method(req=request, **action_args)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 502, in detail
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     servers = self._get_servers(req, is_detail=True)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 579, in _get_servers
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     response = self._view_builder.detail(req, instance_list)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 126, in detail
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     return self._list_view(self.show, request, instances)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 130, in _list_view
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     server_list = [func(request, server)["server"] for server in servers]
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 98, in show
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     "flavor": self._get_flavor(request, instance),
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/api/openstack/compute/views/servers.py", line 186, in _get_flavor
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     instance_type = flavors.extract_flavor(instance)
  2013-11-13 22:25:08.224 TRACE nova.api.openstack   File "/opt/stack/nova/nova/compute/flavors.py", line 282, in extract_flavor
  2013-11-13 22:25:08.224 TRACE nova.api.openstack     instance_type[key] = type_fn(sys_meta[type_key])
  2013-11-13 22:25:08.224 TRACE nova.api.openstack KeyError: 'instance_type_memory_mb'
  2013-11-13 22:25:08.224 TRACE nova.api.openstack
  2013-11-13 22:25:08.225 INFO nova.api.openstack [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] http://192.168.2.109:8774/v2/6c1cae5b7bce4fb4bb6e5a280e8bcab3/servers/detail returned with HTTP 500
  2013-11-13 22:25:08.226 DEBUG nova.api.openstack.wsgi [req-aaecebbc-6074-495d-8b21-5822c5a4b5db admin demo] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. from (pid=17745) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1224

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1251446/+subscriptions