yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1271417] Re: glance_num_retries can't work

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Joe Gordon <1271417@xxxxxxxxxxxxxxxxxx>
Date: Wed, 05 Feb 2014 22:37:06 -0000
Reply-to: Bug 1271417 <1271417@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
If you want to do load balancing you should do it by putting both glance
servers behind the same IP somehow.

The current glance logic randomly picks one of the glance_api_servers,
and has no concept of knowing which glance_api_server is down.

We are trying to move away from using the glance_api_servers config
option and use the keystone registry as well.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1271417

Title:
  glance_num_retries can't work

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  Version : Havana

  I have two controllers in my environment, and deploy glance-api on each controller.
  In nova.conf :
  glance_api_servers=controller2:9292,controller2:9292
  glance_num_retries = 2

  When I kill the glance on controller2, then run "nova image-list", I will get error sometimes:
  # nova image-list
  +--------------------------------------+--------+--------+--------+
  | ID                                   | Name   | Status | Server |
  +--------------------------------------+--------+--------+--------+
  | 668a4e66-97d9-40d7-888e-dd9db53438c4 | centos | ACTIVE |        |
  | 7e7917a6-96c5-4000-b167-efc13be4124c | cirros | ACTIVE |        |
  +--------------------------------------+--------+--------+--------+
  # nova image-list
  ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-09d764df-4e8b-4330-b39e-14e4736c2e68)

  The error in /var/log/nova/api.log is:
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack Traceback (most recent call last):
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 119, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return req.get_response(self.application)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     application, catch_exc_info=False)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 574, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.app(env, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     response = self.app(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return resp(environ, start_response)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 917, in __call__
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     content_type, body, accept)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 976, in _process_stack
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1057, in dispatch
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     return method(req=request, **action_args)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/images.py", line 203, in detail
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     **page_params)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 264, in detail
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     for image in images:
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 174, in paginate
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     images = self._list(url, "images")
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/base.py", line 53, in _list
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body = self.api.json_request('GET', url)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 266, in json_request
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     resp, body_iter = self._http_request(url, method, **kwargs)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 235, in _http_request
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack     raise exc.CommunicationError(message=message)
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack CommunicationError: Error communicating with http://10.224.159.75:9292 [Errno 111] ECONNREFUSED
  2014-01-22 05:33:08.568 8866 TRACE nova.api.openstack
  2014-01-22 05:33:08.572 8866 INFO nova.api.openstack [req-5d5b5dd1-36a5-48d9-8d6e-21be5551b135 feea78294ac84fb29583327109e899d6 3a5541d10e6b4a0fbcd45213e0e17ada] http://10.224.159.111/v2/3a5541d10e6b4a0fbcd45213e0e17ada/images/detail returned with HTTP 500


  "10.224.159.75:9292" is my controller2 IP.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1271417/+subscriptions
References

[Bug 1271417] [NEW] glance_num_retries can't work
From: Liping Mao, 2014-01-22