← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1791478] Re: nova-api get 504 when keepalived restart

 

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1791478

Title:
  nova-api get 504 when keepalived restart

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  With haproxy + keepalived in openstack controller service HA, it got
  504 error:

  first keystone:
  DEBUG (base:176) Making authentication request to http://192.168.21.1:35357/v3/auth/tokens
  DEBUG (connectionpool:393) http://192.168.21.1:35357 "POST /v3/auth/tokens HTTP/1.1" 504 None
  DEBUG (session:868) Request returned failure status: 504
  DEBUG (shell:791) Gateway Timeout (HTTP 504)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 657, in main
      api_version = api_versions.discover_version(self.cs, api_version)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 261, in discover_version
      client)
    File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 242, in _get_server_version_range
      version = client.versions.get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 70, in get_current
      return self._get_current()
    File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 52, in _get_current
      url = "%s" % self.api.client.get_endpoint()
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
      return self.session.get_endpoint(auth or self.auth, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
      return auth.get_endpoint(self, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
      allow_version_hack=allow_version_hack, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
      service_catalog = self.get_access(session).service_catalog
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
      self.auth_ref = self.get_auth_ref(session)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
      return self._plugin.get_auth_ref(session, **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
      authenticated=False, log=False, **rkwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1019, in post
      return self.request(url, 'POST', **kwargs)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 869, in request
      raise exceptions.from_response(resp, method, url)
  GatewayTimeout: Gateway Timeout (HTTP 504)
  ERROR (GatewayTimeout): Gateway Timeout (HTTP 504)

  then after about 1 min, keystone has worked but nova-api:

  DEBUG (connectionpool:205) Starting new HTTP connection (1): 10.22.21.1:8774
  DEBUG (connectionpool:393) http://10.22.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d HTTP/1.1" 404 112
  RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
  DEBUG (session:479) RESP: [404] Content-Length: 112 Content-Type: application/json; charset=UTF-8 Date: Sun, 09 Sep 2018 00:56:18 GMT X-Compute-Request-Id: req-9e16a30b-3407-4f4f-89d7-1b64f4637b25
  RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}
  DEBUG (session:511) RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"}

  It reported service not found, and then got the 504:

  DEBUG (session:448) REQ: curl -g -i -X GET http://192.168.21.1:8774/v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.42" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}d0dd50b3f49d2ab39fae428e4eff0050bb2088ab" -H "X-OpenStack-Nova-API-Version: 2.42"
  DEBUG (connectionpool:393) http://192.168.21.1:8774 "GET /v2.1/75bd7e950f4b4792a486b6bf240c7a3d/servers/detail HTTP/1.1" 504 None
  DEBUG (session:479) RESP: [504] Cache-Control: no-cache Connection: close Content-Type: text/html
  DEBUG (session:511) RESP BODY: Omitted, Content-Type is set to text/html. Only application/json responses have their bodies logged.
  DEBUG (shell:791) Unknown Error (HTTP 504)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 789, in main
      OpenStackComputeShell().main(argv)
    File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 711, in main
      args.func(self.cs, args)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 1565, in do_list
      limit=args.limit)
    File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 854, in list
      "servers")
    File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 257, in _list
      resp, body = self.api.client.get(url)
    File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 328, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 83, in request
      raise exceptions.from_response(resp, body, url, method)
  ClientException: Unknown Error (HTTP 504)
  ERROR (ClientException): Unknown Error (HTTP 504)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1791478/+subscriptions


References