yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1231643] Re: ERROR in n-api log after successful tempest run

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Thu, 18 Sep 2014 23:34:38 -0000
Reply-to: Bug 1231643 <1231643@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1231643

Title:
  ERROR in n-api log after successful tempest run

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  This is from http://logs.openstack.org/12/48412/1/check/gate-tempest-
  devstack-vm-full/7fdaa6b/logs/screen-n-api.txt.gz

  There seem to be several types of errors, with many cases of each.

  
  2013-09-26 09:44:52.688 23009 DEBUG glanceclient.common.http [-] curl -i -X HEAD -H 'X-Service-Catalog: [{"endpoints_links": [], "endpoints": [{"adminURL": "http://127.0.0.1:8776/v1/a512e8f95f484672a92933eecf38b830";, "region": "RegionOne", "publicURL": "http://127.0.0.1:8776/v1/a512e8f95f484672a92933eecf38b830";, "id": "0260ed4490a242afa56b8f72bc93814f", "internalURL": "http://127.0.0.1:8776/v1/a512e8f95f484672a92933eecf38b830"}], "type": "volume", "name": "cinder"}]' -H 'X-Identity-Status: Confirmed' -H 'X-Roles: _member_' -H 'User-Agent: python-glanceclient' -H 'X-Tenant-Id: a512e8f95f484672a92933eecf38b830' -H 'X-User-Id: bede90277bfe4bd2b44d397d8418aaaf' -H 'X-Auth-Token:  TOKEN_SKIPPED' -H 'Content-Type: application/octet-stream' http://127.0.0.1:9292/v1/images/999 log_curl_request /opt/stack/new/python-glanceclient/glanceclient/common/http.py:140
  2013-09-26 09:44:52.707 23009 DEBUG glanceclient.common.http [-] 
  HTTP/1.1 404 Not Found
  date: Thu, 26 Sep 2013 09:44:52 GMT
  content-length: 0
  content-type: text/plain; charset=UTF-8
  x-openstack-request-id: req-2beb0d25-bacf-4be2-879b-3646c5dddc4f
   log_http_response /opt/stack/new/python-glanceclient/glanceclient/common/http.py:150
  2013-09-26 09:44:52.707 23009 ERROR glanceclient.common.http [-] Request returned failure status.
  2013-09-26 09:44:52.708 INFO nova.api.openstack.wsgi [req-e3ef6f9a-f49e-4854-9a8c-839924b8a6c8 ImagesMetadataTestJSON1051049647-user ImagesMetadataTestJSON404523297-tenant] HTTP exception thrown: Image not found.
  2013-09-26 09:44:52.709 DEBUG nova.api.openstack.wsgi [req-e3ef6f9a-f49e-4854-9a8c-839924b8a6c8 ImagesMetadataTestJSON1051049647-user ImagesMetadataTestJSON404523297-tenant] Returning 404 to user: Image not found. __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1198
  2013-09-26 09:44:52.716 INFO nova.osapi_compute.wsgi.server [req-e3ef6f9a-f49e-4854-9a8c-839924b8a6c8 ImagesMetadataTestJSON1051049647-user ImagesMetadataTestJSON404523297-tenant] 127.0.0.1 "DELETE /v2/a512e8f95f484672a92933eecf38b830/images/999/metadata/key1 HTTP/1.1" status: 404 len: 275 time: 0.0309439

  2013-09-26 09:44:52.719 23008 INFO nova.osapi_compute.wsgi.server [-]
  (23008) accepted ('127.0.0.1', 41420)


  
  2013-09-26 09:45:42.488 ERROR nova.api.openstack [req-53bdcf9f-71d4-4436-8704-e156800e5899 ServersAdminTestJSON92006323-user ServersAdminTestJSON760329599-tenant] Caught error: Quota exceeded for ram: Requested 51137, but already used 128 of 51200 ram
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack Traceback (most recent call last):
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/__init__.py", line 119, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return req.get_response(self.application)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     application, catch_exc_info=False)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/python-keystoneclient/keystoneclient/middleware/auth_token.py", line 545, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return self.app(env, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return resp(environ, start_response)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 912, in __call__
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     content_type, body, accept)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 971, in _process_stack
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 1052, in dispatch
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return method(req=request, **action_args)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 1248, in _action_resize
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return self._resize(req, id, flavor_ref, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 1113, in _resize
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     self.compute_api.resize(context, instance, flavor_id, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 198, in wrapped
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 188, in inner
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return function(self, context, instance, *args, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 215, in _wrapped
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return fn(self, context, instance, *args, **kwargs)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 169, in inner
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack   File "/opt/stack/new/nova/nova/compute/api.py", line 2298, in resize
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack     resource=resource)
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack TooManyInstances: Quota exceeded for ram: Requested 51137, but already used 128 of 51200 ram
  2013-09-26 09:45:42.488 23010 TRACE nova.api.openstack 
  2013-09-26 09:45:42.493 INFO nova.api.openstack [req-53bdcf9f-71d4-4436-8704-e156800e5899 ServersAdminTestJSON92006323-user ServersAdminTestJSON760329599-tenant] http://127.0.0.1:8774/v2/109b69ffd5cc47bfa5a0e7a2970ba22f/servers/8229c042-8cef-49ea-b5bf-37717514a08e/action returned with HTTP 413
  2013-09-26 09:45:42.494 DEBUG nova.api.openstack.wsgi [req-53bdcf9f-71d4-4436-8704-e156800e5899 ServersAdminTestJSON92006323-user ServersAdminTestJSON760329599-tenant] Returning 413 to user: TooManyInstances: Quota exceeded for %(overs)s: Requested %(req)s, but already used %(used)d of %(allowed)d %(resource)s __call__ /opt/stack/new/nova/nova/api/openstack/wsgi.py:1198

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1231643/+subscriptions