yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1271417] [NEW] glance_num_retries can't work

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Liping Mao <limao@xxxxxxxxx>
Date: Wed, 22 Jan 2014 05:36:41 -0000
Reply-to: Bug 1271417 <1271417@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Version : Havana

I have two controllers in my environment, and deploy glance-api on each controller.
In nova.conf :
glance_api_servers=controller2:9292,controller2:9292
glance_num_retries = 2

When I kill the glance on controller2, then run "nova image-list", I will get error sometimes:
# nova image-list
+--------------------------------------+--------+--------+--------+
| ID                                   | Name   | Status | Server |
+--------------------------------------+--------+--------+--------+
| 668a4e66-97d9-40d7-888e-dd9db53438c4 | centos | ACTIVE |        |
| 7e7917a6-96c5-4000-b167-efc13be4124c | cirros | ACTIVE |        |
+--------------------------------------+--------+--------+--------+
# nova image-list
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09d764df-4e8b-4330-b39e-14e4736c2e68)

The error in /var/log/nova/api.log is:
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack Traceback (most recent call last):
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 119, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return req.get_response(self.application)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     application, catch_exc_info=False)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 574, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.app(env, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     response = self.app(environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     content_type, body, accept)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 976, in _process_stack
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1057, in dispatch
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return method(req=request, **action_args)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/images.py", line 203, in detail
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     **page_params)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 264, in detail
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     for image in images:
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 174, in paginate
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     images = self._list(url, "images")
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/base.py", line 53, in _list
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body = self.api.json_request('GET', url)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 266, in json_request
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body_iter = self._http_request(url, method, **kwargs)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 235, in _http_request
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     raise exc.CommunicationError(message=message)
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack CommunicationError: Error communicating with http://10.224.159.75:9292 [Errno 111] ECONNREFUSED
2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack
2014-01-22 05:33:08.572 8866 INFO nova.api.openstack [req-5d5b5dd1-36a5-48d9-8d6e-21be5551b135 feea78294ac84fb29583327109e899d6 3a5541d10e6b4a0fbcd45213e0e17ada] http://10.224.159.111/v2/3a5541d10e6b4a0fbcd45213e0e17ada/images/detail returned with HTTP 500


"10.224.159.75:9292" is my controller2 IP.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1271417

Title:
  glance_num_retries can't work

Status in OpenStack Compute (Nova):
  New

Bug description:
  Version : Havana

  I have two controllers in my environment, and deploy glance-api on each controller.
  In nova.conf :
  glance_api_servers=controller2:9292,controller2:9292
  glance_num_retries = 2

  When I kill the glance on controller2, then run "nova image-list", I will get error sometimes:
  # nova image-list
  +--------------------------------------+--------+--------+--------+
  | ID                                   | Name   | Status | Server |
  +--------------------------------------+--------+--------+--------+
  | 668a4e66-97d9-40d7-888e-dd9db53438c4 | centos | ACTIVE |        |
  | 7e7917a6-96c5-4000-b167-efc13be4124c | cirros | ACTIVE |        |
  +--------------------------------------+--------+--------+--------+
  # nova image-list
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09d764df-4e8b-4330-b39e-14e4736c2e68)

  The error in /var/log/nova/api.log is:
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack Traceback (most recent call last):
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 119, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 574, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     content_type, body, accept)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 976, in _process_stack
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1057, in dispatch
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/images.py", line 203, in detail
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     **page_params)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 264, in detail
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     for image in images:
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 174, in paginate
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     images = self._list(url, "images")
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/base.py", line 53, in _list
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body = self.api.json_request('GET', url)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 266, in json_request
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body_iter = self._http_request(url, method, **kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 235, in _http_request
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     raise exc.CommunicationError(message=message)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack CommunicationError: Error communicating with http://10.224.159.75:9292 [Errno 111] ECONNREFUSED
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack
  2014-01-22 05:33:08.572 8866 INFO nova.api.openstack [req-5d5b5dd1-36a5-48d9-8d6e-21be5551b135 feea78294ac84fb29583327109e899d6 3a5541d10e6b4a0fbcd45213e0e17ada] http://10.224.159.111/v2/3a5541d10e6b4a0fbcd45213e0e17ada/images/detail returned with HTTP 500


  "10.224.159.75:9292" is my controller2 IP.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1271417/+subscriptions
Follow ups

[Bug 1271417] Re: glance_num_retries can't work
From: Joe Gordon, 2014-02-05
[Bug 1271417] [NEW] glance_num_retries can't work
From: Liping Mao, 2014-01-22
References

[Bug 1271417] [NEW] glance_num_retries can't work
From: Liping Mao, 2014-01-22