yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1681866] Re: Bad response code while validating token: 502

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ihar Hrachyshka <1681866@xxxxxxxxxxxxxxxxxx>
Date: Mon, 03 Jul 2017 17:43:57 -0000
Reply-to: Bug 1681866 <1681866@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Happening in Ocata too: http://logs.openstack.org/86/474286/1/gate/gate-
tempest-dsvm-neutron-dvr-ubuntu-
xenial/2b247e0/logs/screen-q-svc.txt.gz#_2017-06-26_21_16_20_582

** Changed in: tempest
   Importance: Medium => High

** Also affects: devstack
   Importance: Undecided
       Status: New

** Also affects: keystone
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1681866

Title:
  Bad response code while validating token: 502

Status in devstack:
  New
Status in OpenStack Identity (keystone):
  New
Status in tempest:
  Confirmed

Bug description:
  Found this while investigating a gate failure [1].

  Tempest logs say "2017-04-11 10:07:02,765 23082 INFO
  [tempest.lib.common.rest_client] Request
  (TestSecurityGroupsBasicOps:_run_cleanups): 503 DELETE
  https://198.72.124.138:8774/v2.1/servers/f736a878-2ac4-4c37-b6a8-e5cd8df5a7fd
  0.018s"

  That 503 looks suspicious. So I go to the nova-api logs. Which gives

  2017-04-11 10:07:02.762 32191 ERROR keystonemiddleware.auth_token [...] Bad response code while validating token: 502
  2017-04-11 10:07:02.763 32191 WARNING keystonemiddleware.auth_token [...] Identity response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>502 Proxy Error</title>
  </head><body>
  <h1>Proxy Error</h1>
  <p>The proxy server received an invalid
  response from an upstream server.<br />
  The proxy server could not handle the request <em><a href="/identity_admin/v3/auth/tokens">GET&nbsp;/identity_admin/v3/auth/tokens</a></em>.<p>
  Reason: <strong>Error reading from remote server</strong></p></p>
  <hr>
  <address>Apache/2.4.18 (Ubuntu) Server at 198.72.124.138 Port 443</address>
  </body></html>

  2017-04-11 10:07:02.763 32191 CRITICAL keystonemiddleware.auth_token
  [...] Unable to validate token: Failed to fetch token data from
  identity server

  So Apache is complaining, some network connection issue, related to
  proxy-ing. So I open "logs/apache/tls-proxy_error.txt.gz" and find

  [Tue Apr 11 10:07:02.761420 2017] [proxy_http:error] [pid 7136:tid 140090189690624] (20014)Internal error (specific information not available): [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH01102: error reading status line from remote server 198.72.124.138:80
  [Tue Apr 11 10:07:02.761454 2017] [proxy:error] [pid 7136:tid 140090189690624] [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH00898: Error reading from remote server returned by /identity_admin/v3/auth/tokens

  Interesting. Google says that adding "proxy-initial-not-pooled" to the
  apache2 vhost config could help.

  Anyway, a good elasticsearch query for this is

  message:"Bad response code while validating token: 502"

  8 hits, no worries.


  [1] : http://logs.openstack.org/03/455303/2/check/gate-tempest-dsvm-
  neutron-full-ubuntu-xenial/aa8c7fd/console.html

To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1681866/+subscriptions