yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1791478] [NEW] nova-api get 504 when keepalived restart

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Henry <1791478@xxxxxxxxxxxxxxxxxx>
Date: Sun, 09 Sep 2018 01:01:20 -0000
Reply-to: Bug 1791478 <1791478@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With haproxy + keepalived in openstack controller service HA, it got 504
error:

first keystone:
DEBUG (base:176) Making authentication request to http://192.168.21.1:35357/v3/auth/tokens
DEBUG (connectionpool:393) http://192.168.21.1:35357 "POST /v3/auth/tokens HTTP/1.1" 504 None
DEBUG (session:868) Request returned failure status: 504
DEBUG (shell:791) Gateway Timeout (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 657, in main
    api_version = api_versions.discover_version(self.cs, api_version)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
    client)
  File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
    version = client.versions.get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
    return self._get_current()
  File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
    url = "%s" % self.api.client.get_endpoint()
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
    return self._plugin.get_auth_ref(session, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 869, in request
    raise exceptions.from_response(resp, method, url)
GatewayTimeout: Gateway Timeout (HTTP 504)
ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)

then after about 1 min, keystone has worked but nova-api:

DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.22.21.1:8774
DEBUG (connectionpool:393) http://10.22.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d HTTP/1.1" 404 112
RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
DEBUG (session:479) RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
DEBUG (session:511) RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}

It reported service not found, and then got the 504:

DEBUG (session:448) REQ: curl -g -i -X GET http://192.168.21.1:8774/v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.42" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}d0dd50b3f49d2ab39fae428e4eff0050bb2088ab" -H "X-OpenStack-Nova-API-Version: 2.42"
DEBUG (connectionpool:393) http://192.168.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail HTTP/1.1" 504 None
DEBUG (session:479) RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
DEBUG (session:511) RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
DEBUG (shell:791) Unknown Error (HTTP 504)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 711, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 1565, in do_list
    limit=args.limit)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 854, in list
    "servers")
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
    raise exceptions.from_response(resp, body, url, method)
ClientException: Unknown Error (HTTP 504)
ERROR (ClientException): Unknown Error (HTTP 504)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791478

Title:
  nova-api get 504 when keepalived restart

Status in OpenStack Compute (nova):
  New

Bug description:
  With haproxy + keepalived in openstack controller service HA, it got
  504 error:

  first keystone:
  DEBUG (base:176) Making authentication request to http://192.168.21.1:35357/v3/auth/tokens
  DEBUG (connectionpool:393) http://192.168.21.1:35357 "POST /v3/auth/tokens HTTP/1.1" 504 None
  DEBUG (session:868) Request returned failure status: 504
  DEBUG (shell:791) Gateway Timeout (HTTP 504)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 657, in main
      api_version = api_versions.discover_version(self.cs, api_version)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
      client)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
      version = client.versions.get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
      return self._get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
      url = "%s" % self.api.client.get_endpoint()
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
      return self.session.get_endpoint(auth or self.auth, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
      return auth.get_endpoint(self, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
      allow_version_hack=allow_version_hack, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
      service_catalog = self.get_access(session).service_catalog
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
      self.auth_ref = self.get_auth_ref(session)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
      return self._plugin.get_auth_ref(session, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
      authenticated=False, log=False, **rkwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
      return self.request(url, 'POST', **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 869, in request
      raise exceptions.from_response(resp, method, url)
  GatewayTimeout: Gateway Timeout (HTTP 504)
  ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)

  then after about 1 min, keystone has worked but nova-api:

  DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.22.21.1:8774
  DEBUG (connectionpool:393) http://10.22.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d HTTP/1.1" 404 112
  RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
  DEBUG (session:479) RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
  RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
  DEBUG (session:511) RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}

  It reported service not found, and then got the 504:

  DEBUG (session:448) REQ: curl -g -i -X GET http://192.168.21.1:8774/v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.42" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}d0dd50b3f49d2ab39fae428e4eff0050bb2088ab" -H "X-OpenStack-Nova-API-Version: 2.42"
  DEBUG (connectionpool:393) http://192.168.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail HTTP/1.1" 504 None
  DEBUG (session:479) RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
  DEBUG (session:511) RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
  DEBUG (shell:791) Unknown Error (HTTP 504)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 711, in main
      args.func(self.cs, args)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 1565, in do_list
      limit=args.limit)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 854, in list
      "servers")
    File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unknown Error (HTTP 504)
  ERROR (ClientException): Unknown Error (HTTP 504)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791478/+subscriptions
Follow ups

[Bug 1791478] Re: nova-api get 504 when keepalived restart
From: Ghanshyam Mann, 2018-11-28