yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1804173] [NEW] Metadata proxy server SSL handshake problem if Python >= 3

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thomas Goirand <thomas@xxxxxxxxxx>
Date: Tue, 20 Nov 2018 09:01:54 -0000
Reply-to: Bug 1804173 <1804173@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Neutron API can now work over SSL using uwsgi or mod_wsgi since Rocky
and beyond, though it looks like nothing has been thought for the proxy
metadata server. Indeed, if using SSL, I'm getting the infamous SSL
handshake problem:

X-Forwarded-For: 192.168.110.4
X-Neutron-Router-Id: 9b4c243d-51e4-45df-84b3-010b38a9e5bd __call__ /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:86
[-] Gotten ports for remote_address 192.168.110.4, network_id None, router_id 9b4c243d-51e4-45df-84b3-010b38a9e5bd are: [{'binding:host_id': 'swift01-compute-1.infomaniak.ch', 'description': '', 'device_id': '0acfcbbc-8365-4304-b9c1-c2b0161c9ff6', 'status': 'ACTIVE', 'network_id': '92783eba-7635-4639-aef9-5c8854c3d2f8', 'created_at': '2018-11-19T15:37:38Z', 'port_security_enabled': True, 'tenant_id': '1994b8ab99c1478ab0a3072ea0118474', 'device_owner': 'compute:nova', 'extra_dhcp_opts': [], 'name': '', 'binding:vif_type': 'ovs', 'binding:profile': {}, 'fixed_ips': [{'ip_address': '192.168.110.4', 'subnet_id': 'a573293e-d5b1-46dd-b301-80a29fc6b3c2'}], 'mac_address': 'fa:16:3e:02:b0:46', 'admin_state_up': True, 'revision_number': 5, 'binding:vnic_type': 'normal', 'binding:vif_details': {'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system'}, 'id': '2629ac29-a6d9-4526-9324-1798a7a7c785', 'tags': [], 'project_id': '1994b8ab99c1478ab0a3072ea0118474', 'security_groups': ['28e6a0d1-f8dc-48c4-a914-927bc07ecf88'], 'updated_at': '2018-11-19T15:37:44Z', 'allowed_address_pairs': [], 'qos_policy_id': None}] _get_instance_and_tenant_id /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:166

[-] Unexpected error.: ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 90, in __call__
   return self._proxy_request(instance_id, tenant_id, req)
 File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 198, in _proxy_request
   body=req.body)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1313, in request
   (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1063, in _request
  (response, content) = self._conn_request(conn, request_uri, method, body, headers)
 File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 986, in _conn_request
   conn.connect()
 File "/usr/lib/python3.5/http/client.py", line 1261, in connect
   server_hostname=server_hostname)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 402, in wrap_socket
   return GreenSSLSocket(sock, *a, _context=self, **kw)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 91, in __init__
   self.do_handshake()
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 270, in do_handshake
   super(GreenSSLSocket, self).do_handshake)
 File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
   return func(*a, **kw)
 File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
   self._sslobj.do_handshake()
 File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
   self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)

2018-11-20 08:43:52.966 2876 INFO eventlet.wsgi.server [-]
192.168.110.4,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1"
status: 500  len: 362 time: 0.2473052

Moreover, currently, we can only configure the metadata server to
contact the API through a hostname and a port. This isn't sufficient at
all. Indeed, in my case, the Nova API is running on:

https://api.example.com/compute

with haproxy rewriting the URL and removing "/compute" from the URL.
Therefore, it's for me impossible to configure neutron-metadata
correctly, unless setting nova-api to bind a non-SSL IP directly on the
root, not protected by haproxy, which makes the whole thing completely
insecure and not resilient to issue (ie: it's not HA anymore).

Please get all of this fixed ASAP, especially the Python 3 + Eventlet +
SSL problem.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1804173

Title:
  Metadata proxy server SSL handshake problem if Python >= 3

Status in neutron:
  New

Bug description:
  Neutron API can now work over SSL using uwsgi or mod_wsgi since Rocky
  and beyond, though it looks like nothing has been thought for the
  proxy metadata server. Indeed, if using SSL, I'm getting the infamous
  SSL handshake problem:

  X-Forwarded-For: 192.168.110.4
  X-Neutron-Router-Id: 9b4c243d-51e4-45df-84b3-010b38a9e5bd __call__ /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:86
  [-] Gotten ports for remote_address 192.168.110.4, network_id None, router_id 9b4c243d-51e4-45df-84b3-010b38a9e5bd are: [{'binding:host_id': 'swift01-compute-1.infomaniak.ch', 'description': '', 'device_id': '0acfcbbc-8365-4304-b9c1-c2b0161c9ff6', 'status': 'ACTIVE', 'network_id': '92783eba-7635-4639-aef9-5c8854c3d2f8', 'created_at': '2018-11-19T15:37:38Z', 'port_security_enabled': True, 'tenant_id': '1994b8ab99c1478ab0a3072ea0118474', 'device_owner': 'compute:nova', 'extra_dhcp_opts': [], 'name': '', 'binding:vif_type': 'ovs', 'binding:profile': {}, 'fixed_ips': [{'ip_address': '192.168.110.4', 'subnet_id': 'a573293e-d5b1-46dd-b301-80a29fc6b3c2'}], 'mac_address': 'fa:16:3e:02:b0:46', 'admin_state_up': True, 'revision_number': 5, 'binding:vnic_type': 'normal', 'binding:vif_details': {'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system'}, 'id': '2629ac29-a6d9-4526-9324-1798a7a7c785', 'tags': [], 'project_id': '1994b8ab99c1478ab0a3072ea0118474', 'security_groups': ['28e6a0d1-f8dc-48c4-a914-927bc07ecf88'], 'updated_at': '2018-11-19T15:37:44Z', 'allowed_address_pairs': [], 'qos_policy_id': None}] _get_instance_and_tenant_id /usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py:166

  [-] Unexpected error.: ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)
  Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 90, in __call__
     return self._proxy_request(instance_id, tenant_id, req)
   File "/usr/lib/python3/dist-packages/neutron/agent/metadata/agent.py", line 198, in _proxy_request
     body=req.body)
   File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1313, in request
     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
   File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 1063, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
   File "/usr/lib/python3/dist-packages/httplib2/__init__.py", line 986, in _conn_request
     conn.connect()
   File "/usr/lib/python3.5/http/client.py", line 1261, in connect
     server_hostname=server_hostname)
   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 402, in wrap_socket
     return GreenSSLSocket(sock, *a, _context=self, **kw)
   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 91, in __init__
     self.do_handshake()
   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 270, in do_handshake
     super(GreenSSLSocket, self).do_handshake)
   File "/usr/lib/python3/dist-packages/eventlet/green/ssl.py", line 113, in _call_trampolining
     return func(*a, **kw)
   File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
     self._sslobj.do_handshake()
   File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
     self._sslobj.do_handshake()
  ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:720)

  2018-11-20 08:43:52.966 2876 INFO eventlet.wsgi.server [-]
  192.168.110.4,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1"
  status: 500  len: 362 time: 0.2473052

  Moreover, currently, we can only configure the metadata server to
  contact the API through a hostname and a port. This isn't sufficient
  at all. Indeed, in my case, the Nova API is running on:

  https://api.example.com/compute

  with haproxy rewriting the URL and removing "/compute" from the URL.
  Therefore, it's for me impossible to configure neutron-metadata
  correctly, unless setting nova-api to bind a non-SSL IP directly on
  the root, not protected by haproxy, which makes the whole thing
  completely insecure and not resilient to issue (ie: it's not HA
  anymore).

  Please get all of this fixed ASAP, especially the Python 3 + Eventlet
  + SSL problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1804173/+subscriptions