yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1294973] Re: Better error message is required when glance is dead while capturing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Aaron Rosen <aaronorosen@xxxxxxxxx>
Date: Thu, 20 Mar 2014 08:02:16 -0000
Reply-to: Bug 1294973 <1294973@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
I don't think reporting the internal errors of exactly were the failure
occurred is helpful to the user of the apis as they would not be able to
do anything to fix this. A 500 error like this would also be raised
today if you were using nova with neutron and neutron was down.

** Changed in: nova
       Status: New => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1294973

Title:
  Better error message is required when glance is dead while capturing

Status in OpenStack Compute (Nova):
  Opinion

Bug description:
  1. Bring down the glance
  Stopping openstack-glance-api:                             [  OK  ]
  Stopping openstack-glance-registry:                        [  OK  ]
  [root@nimbuswrkl665 images]#
  2. Invoke createImage operation for the server.

  manjunath@manjunath-ThinkPad-T420:˜/Desktop$ curl -k -i -X POST -H "X
  -Auth-Token: 6928aacccede46d7921c57fb827cd9be" -H "Content-Type:
  application/json"  -d '{"createImage": {"name": "vm1_failed",
  "metadata": {}}}'
  https://nimbuswrkl665.rtp.stglabs.ibm.com/powervc/openstack/compute/v2/dd73d8ab4ea7498e8d103206e37c65b5/servers/93476dc9
  -c70e-4b73-af3e-1309fecdaf95/action

  Response :

  HTTP/1.1 500 Internal Server Error
  Date: Mon, 10 Mar 2014 14:34:27 GMT
  Content-Type: application/json; charset=UTF-8
  X-Compute-Request-Id: req-75d8138c-1d32-4bb7-888f-7aa48bdd2011
  Cache-control: no-cache
  Pragma: no-cache
  Content-Length: 128
  Connection: close

  {"computeFault": {"message": "The server has either erred or is
  incapable of performing the requested operation.", "code":
  500}}manjunath@manjunath-Thimanjunath@manjunath-
  ThinkPad-T420:˜/Desktop$

  
  Message needs to be Improved, I could see clear message in api.log but it is not passing to the REST call.!

  2014-03-10 10:34:27.130 4713 ERROR nova.image.glance [req-75d8138c-1d32-4bb7-888f-7aa48bdd2011 0 dd73d8ab4ea7498e8d103206e37c65b5] Error contacting glance server '9.37.74.232:9292' for 'get', retrying.
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance Traceback (most recent call last):
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 211, in call
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance     return getattr(client.images, method)(*args, **kwargs)
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 114, in get
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance     % urllib.quote(str(image_id)))
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 289, in raw_request
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance     return self._http_request(url, method, **kwargs)
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 235, in _http_request
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance     raise exc.CommunicationError(message=message)
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance CommunicationError: Error communicating with http://9.37.74.232:9292 [Errno 111] ECONNREFUSED
  2014-03-10 10:34:27.130 4713 TRACE nova.image.glance
  2014-03-10 10:34:27.240 4713 WARNING nova.compute.utils [req-75d8138c-1d32-4bb7-888f-7aa48bdd2011 0 dd73d8ab4ea7498e8d103206e37c65b5] [instance: 93476dc9-c70e-4b73-af3e-1309fecdaf95] NV-6BF9597 Can't access image 36f6d470-3a3f-4113-8724-b018280c8f27: NV-BAD1189 Connection to glance host 9.37.74.232:9292 failed: Error communicating with http://9.37.74.232:9292 [Errno 111] ECONNREFUSED
  2014-03-10 10:34:27.244 4713 ERROR nova.image.glance [req-75d8138c-1d32-4bb7-888f-7aa48bdd2011 0 dd73d8ab4ea7498e8d103206e37c65b5] Error contacting glance server '9.37.74.232:9292' for 'create', retrying.
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance Traceback (most recent call last):
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 211, in call
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance     return getattr(client.images, method)(*args, **kwargs)
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 253, in create
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance     'POST', '/v1/images', headers=hdrs, body=image_data)
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 289, in raw_request
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance     return self._http_request(url, method, **kwargs)
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 235, in _http_request
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance     raise exc.CommunicationError(message=message)
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance CommunicationError: Error communicating with http://9.37.74.232:9292 [Errno 111] ECONNREFUSED
  2014-03-10 10:34:27.244 4713 TRACE nova.image.glance
  2014-03-10 10:34:27.255 4713 ERROR nova.api.openstack [req-75d8138c-1d32-4bb7-888f-7aa48bdd2011 0 dd73d8ab4ea7498e8d103206e37c65b5] NV-A68A08C Caught error: NV-BAD1189 Connection to glance host 9.37.74.232:9292 failed: Error communicating with http://9.37.74.232:9292 [Errno 111] ECONNREFUSED
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack Traceback (most recent call last):
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 125, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 600, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 929, in __call__
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     content_type, body, accept)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 991, in _process_stack
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1078, in dispatch
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/common.py", line 471, in inner
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return f(*args, **kwargs)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/servers.py", line 1487, in _action_create_image
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     extra_properties=props)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 206, in wrapped
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return func(self, context, target, *args, **kwargs)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 223, in _wrapped
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return fn(self, context, instance, *args, **kwargs)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 177, in inner
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return f(self, context, instance, *args, **kw)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 1936, in snapshot
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     extra_properties=extra_properties)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/compute/api.py", line 1981, in _create_image
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     return self.image_service.create(context, sent_meta)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 374, in create
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     context, 1, 'create', **sent_service_image_meta)
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 225, in call
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack     host=host, port=port, reason=str(e))
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack GlanceConnectionFailed: NV-BAD1189 Connection to glance host 9.37.74.232:9292 failed: Error communicating with http://9.37.74.232:9292 [Errno 111] ECONNREFUSED
  2014-03-10 10:34:27.255 4713 TRACE nova.api.openstack

  I'm not  convenienced about the error message. I'm interested in
  generic sort of error message where it says glance is down instead of
  server is down(which does not make any sense in this case).

  Version - Icehouse version

  # rpm --info -q openstack-nova-compute-2014.1-201402161955.ibm.fc19.75.noarch
  Name        : openstack-nova-compute
  Version     : 2014.1
  Release     : 201402161955.ibm.fc19.75
  Architecture: noarch
  Install Date: Wednesday 19 March 2014 07:46:45 AM EDT
  Group       : Applications/System
  Size        : 12247
  License     : ASL 2.0
  Signature   : (none)
  Source RPM  : openstack-nova-2014.1-201402161955.ibm.fc19.75.src.rpm
  Build Date  : Sunday 16 February 2014 09:07:02 PM EST
  Relocations : (not relocatable)
  URL         : http://openstack.org/projects/compute/
  Summary     : OpenStack Nova Virtual Machine control service
  Description :
  OpenStack Compute (codename Nova) is open source software designed to
  provision and manage large networks of virtual machines, creating a
  redundant and scalable cloud computing platform. It gives you the
  software, control panels, and APIs required to orchestrate a cloud,
  including running instances, managing networks, and controlling access
  through users and projects. OpenStack Compute strives to be both
  hardware and hypervisor agnostic, currently supporting a variety of
  standard hardware configurations and seven major hypervisors.

  This package contains the Nova service for controlling Virtual
  Machines.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1294973/+subscriptions
References

[Bug 1294973] [NEW] Better error message is required when glance is dead while capturing
From: Manjunath, 2014-03-20