yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2059032] [NEW] Neutron metadata service returns http code 500 if nova metadata service is down

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Anton Kurbatov <2059032@xxxxxxxxxxxxxxxxxx>
Date: Mon, 25 Mar 2024 18:48:56 -0000
Reply-to: Bug 2059032 <2059032@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

We discovered that if the nova metadata service is down, then the
neutron metadata service starts printing stack traces with a 500 HTTP
code to the user.

Demo on a newly installed devstack

$ systemctl stop devstack@n-api-meta.service

Then inside a VM:

$ curl http://169.254.169.254/latest/meta-data/hostname
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  An unknown error has occurred. Please try your request again.<br /><br />
 </body>
</html>$

Stack trace:

ERROR neutron.agent.metadata.agent Traceback (most recent call last):
ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 85, in __call__
ERROR neutron.agent.metadata.agent     res = self._proxy_request(instance_id, tenant_id, req)
ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 249, in _proxy_request
ERROR neutron.agent.metadata.agent     resp = requests.request(method=req.method, url=url,
ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request
ERROR neutron.agent.metadata.agent     return session.request(method=method, url=url, **kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
ERROR neutron.agent.metadata.agent     resp = self.send(prep, **send_kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
ERROR neutron.agent.metadata.agent     r = adapter.send(request, **kwargs)
ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
ERROR neutron.agent.metadata.agent     raise ConnectionError(e, request=request)
ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.136.16.184', port=8775): Max retries exceeded with url: /latest/meta-data/hostname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ce93f38b0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
ERROR neutron.agent.metadata.agent
INFO eventlet.wsgi.server [-] ::ffff:192.168.100.14,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500  len: 362 time: 0.1392403


Also, in our installation the nova service is behind nginx. And if we stop nova metadata service we also get 500 http code but with another traceback:

2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent [-] Unexpected error.: Exception: Unexpected response code: 502
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 93, in __call__
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent     res = self._proxy_request(instance_id, tenant_id, req)
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 288, in _proxy_request
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent     resp.status_code)
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 502
2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent
2024-03-25 20:27:01.988 24 INFO eventlet.wsgi.server [-] 10.197.115.207,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500  len: 362 time: 0.1369441

It seems to me that it is also better to handle nginx-like gateway
errors a bit more correctly.

These 500 HTTP codes worries us because we are trying to create an alert
system one of the criteria for which is 500 codes.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2059032

Title:
  Neutron metadata service returns http code 500 if nova metadata
  service is down

Status in neutron:
  New

Bug description:
  We discovered that if the nova metadata service is down, then the
  neutron metadata service starts printing stack traces with a 500 HTTP
  code to the user.

  Demo on a newly installed devstack

  $ systemctl stop devstack@n-api-meta.service

  Then inside a VM:

  $ curl http://169.254.169.254/latest/meta-data/hostname
  <html>
   <head>
    <title>500 Internal Server Error</title>
   </head>
   <body>
    <h1>500 Internal Server Error</h1>
    An unknown error has occurred. Please try your request again.<br /><br />
   </body>
  </html>$

  Stack trace:

  ERROR neutron.agent.metadata.agent Traceback (most recent call last):
  ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 85, in __call__
  ERROR neutron.agent.metadata.agent     res = self._proxy_request(instance_id, tenant_id, req)
  ERROR neutron.agent.metadata.agent   File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 249, in _proxy_request
  ERROR neutron.agent.metadata.agent     resp = requests.request(method=req.method, url=url,
  ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request
  ERROR neutron.agent.metadata.agent     return session.request(method=method, url=url, **kwargs)
  ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
  ERROR neutron.agent.metadata.agent     resp = self.send(prep, **send_kwargs)
  ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
  ERROR neutron.agent.metadata.agent     r = adapter.send(request, **kwargs)
  ERROR neutron.agent.metadata.agent   File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
  ERROR neutron.agent.metadata.agent     raise ConnectionError(e, request=request)
  ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.136.16.184', port=8775): Max retries exceeded with url: /latest/meta-data/hostname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ce93f38b0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED'))
  ERROR neutron.agent.metadata.agent
  INFO eventlet.wsgi.server [-] ::ffff:192.168.100.14,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500  len: 362 time: 0.1392403

  
  Also, in our installation the nova service is behind nginx. And if we stop nova metadata service we also get 500 http code but with another traceback:

  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent [-] Unexpected error.: Exception: Unexpected response code: 502
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Traceback (most recent call last):
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 93, in __call__
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent     res = self._proxy_request(instance_id, tenant_id, req)
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 288, in _proxy_request
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent     resp.status_code)
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 502
  2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent
  2024-03-25 20:27:01.988 24 INFO eventlet.wsgi.server [-] 10.197.115.207,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500  len: 362 time: 0.1369441

  It seems to me that it is also better to handle nginx-like gateway
  errors a bit more correctly.

  These 500 HTTP codes worries us because we are trying to create an
  alert system one of the criteria for which is 500 codes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2059032/+subscriptions
Follow ups

[Bug 2059032] Re: Neutron metadata service returns http code 500 if nova metadata service is down
From: OpenStack Infra, 2024-03-26