yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #75992
[Bug 1791478] Re: nova-api get 504 when keepalived restart
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791478
Title:
nova-api get 504 when keepalived restart
Status in OpenStack Compute (nova):
Invalid
Bug description:
With haproxy + keepalived in openstack controller service HA, it got
504 error:
first keystone:
DEBUG (base:176) Making authentication request to http://192.168.21.1:35357/v3/auth/tokens
DEBUG (connectionpool:393) http://192.168.21.1:35357 "POST /v3/auth/tokens HTTP/1.1" 504 None
DEBUG (session:868) Request returned failure status: 504
DEBUG (shell:791) Gateway Timeout (HTTP 504)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 657, in main
api_version = api_versions.discover_version(self.cs, api_version)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
client)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
version = client.versions.get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
return self._get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
url = "%s" % self.api.client.get_endpoint()
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
return self.session.get_endpoint(auth or self.auth, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
allow_version_hack=allow_version_hack, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
service_catalog = self.get_access(session).service_catalog
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
return self._plugin.get_auth_ref(session, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 869, in request
raise exceptions.from_response(resp, method, url)
GatewayTimeout: Gateway Timeout (HTTP 504)
ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)
then after about 1 min, keystone has worked but nova-api:
DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.22.21.1:8774
DEBUG (connectionpool:393) http://10.22.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d HTTP/1.1" 404 112
RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
DEBUG (session:479) RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
DEBUG (session:511) RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
It reported service not found, and then got the 504:
DEBUG (session:448) REQ: curl -g -i -X GET http://192.168.21.1:8774/v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.42" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}d0dd50b3f49d2ab39fae428e4eff0050bb2088ab" -H "X-OpenStack-Nova-API-Version: 2.42"
DEBUG (connectionpool:393) http://192.168.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail HTTP/1.1" 504 None
DEBUG (session:479) RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
DEBUG (session:511) RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
DEBUG (shell:791) Unknown Error (HTTP 504)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 711, in main
args.func(self.cs, args)
File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 1565, in do_list
limit=args.limit)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 854, in list
"servers")
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
resp, body = self.api.client.get(url)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: Unknown Error (HTTP 504)
ERROR (ClientException): Unknown Error (HTTP 504)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791478/+subscriptions
References