yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2120723] [NEW] [OVN] Metadata agent errors when returning gzip-encoded data

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Zachary Seguin <2120723@xxxxxxxxxxxxxxxxxx>
Date: Fri, 15 Aug 2025 14:28:12 -0000
Reply-to: Bug 2120723 <2120723@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

In 2025.1 (deployed with OpenStack-Ansible 31.0.0), when requesting the
user_data for an instance that is gzip-encoded, the client gets a 502
Bad Gateway response. This worked previously in our 2024.2 deployment
(and rolling back just the neutron-ovn-metadata-agent to 2024.2 fixes
the issue).

This is the error found in the neutron-ovn-metadata-agent logs:

2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket [-] Error while receiving data.: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket Traceback (most recent call last):
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 144, in handle
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     res = self._proxy_request(instance_id, project_id, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 103, in _proxy_request
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     return self._http_response(resp, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 49, in _http_response
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     out = request.http_version + ' ' + str(_res)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket                                        ^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 403, in __str__
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     parts += ['', self.body if PY2 else self.text]
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket                                         ^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 622, in _text__get
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     return body.decode(decoding, self.unicode_errors)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
----------------------------------------
Exception occurred during processing of request from
Traceback (most recent call last):
  File "/usr/lib/python3.11/socketserver.py", line 691, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 133, in __init__
    super().__init__(self._conf, has_cache=False, request=request,
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/metadata/proxy_base.py", line 48, in __init__
    super().__init__(**kwargs)
  File "/usr/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 165, in handle
    raise exc
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 144, in handle
    res = self._proxy_request(instance_id, project_id, req)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 103, in _proxy_request
    return self._http_response(resp, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 49, in _http_response
    out = request.http_version + ' ' + str(_res)
                                       ^^^^^^^^^
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 403, in __str__
    parts += ['', self.body if PY2 else self.text]
                                        ^^^^^^^^^
  File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 622, in _text__get
    return body.decode(decoding, self.unicode_errors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
----------------------------------------

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2120723

Title:
  [OVN] Metadata agent errors when returning gzip-encoded data

Status in neutron:
  New

Bug description:
  In 2025.1 (deployed with OpenStack-Ansible 31.0.0), when requesting
  the user_data for an instance that is gzip-encoded, the client gets a
  502 Bad Gateway response. This worked previously in our 2024.2
  deployment (and rolling back just the neutron-ovn-metadata-agent to
  2024.2 fixes the issue).

  This is the error found in the neutron-ovn-metadata-agent logs:

  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket [-] Error while receiving data.: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket Traceback (most recent call last):
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 144, in handle
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     res = self._proxy_request(instance_id, project_id, req)
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 103, in _proxy_request
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     return self._http_response(resp, req)
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 49, in _http_response
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     out = request.http_version + ' ' + str(_res)
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket                                        ^^^^^^^^^
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 403, in __str__
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     parts += ['', self.body if PY2 else self.text]
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket                                         ^^^^^^^^^
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket   File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 622, in _text__get
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket     return body.decode(decoding, self.unicode_errors)
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
  2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
  ----------------------------------------
  Exception occurred during processing of request from
  Traceback (most recent call last):
    File "/usr/lib/python3.11/socketserver.py", line 691, in process_request_thread
      self.finish_request(request, client_address)
    File "/usr/lib/python3.11/socketserver.py", line 361, in finish_request
      self.RequestHandlerClass(request, client_address, self)
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 133, in __init__
      super().__init__(self._conf, has_cache=False, request=request,
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/metadata/proxy_base.py", line 48, in __init__
      super().__init__(**kwargs)
    File "/usr/lib/python3.11/socketserver.py", line 755, in __init__
      self.handle()
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 165, in handle
      raise exc
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 144, in handle
      res = self._proxy_request(instance_id, project_id, req)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 103, in _proxy_request
      return self._http_response(resp, req)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py", line 49, in _http_response
      out = request.http_version + ' ' + str(_res)
                                         ^^^^^^^^^
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 403, in __str__
      parts += ['', self.body if PY2 else self.text]
                                          ^^^^^^^^^
    File "/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py", line 622, in _text__get
      return body.decode(decoding, self.unicode_errors)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
  ----------------------------------------

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2120723/+subscriptions