yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1888237] Re: nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <1888237@xxxxxxxxxxxxxxxxxx>
Date: Thu, 17 Sep 2020 14:31:28 -0000
Reply-to: Bug 1888237 <1888237@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888237

Title:
  nova-next job fails as novnc service fails with TypeError:
  _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket

Status in OpenStack Compute (nova):
  Fix Released
Status in Ubuntu:
  New

Bug description:
  Since 18th of July nova-next job fails [1] with the following stack
  trace in the n-novnc-cell1 service[2]:

  Jul 20 10:44:42.419916 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: DEBUG nova.console.rfb.authvencrypt [None req-
  4c318998-3a74-4929-beab-b774a72844cf None None] Server accepted the
  requested sub-auth type {{(pid=23498) security_handshake
  /opt/stack/nova/nova/console/rfb/authvencrypt.py:126}}

  Jul 20 10:44:42.425350 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: INFO nova.console.websocketproxy [None req-
  4c318998-3a74-4929-beab-b774a72844cf None None] handler exception:
  _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: DEBUG nova.console.websocketproxy [None req-
  4c318998-3a74-4929-beab-b774a72844cf None None] exception
  {{(pid=23498) vmsg /usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py:634}}

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy Traceback (most
  recent call last):

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 691, in top_new_client

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     client =
  self.do_handshake(startsock, address)

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 619, in do_handshake

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  self.RequestHandlerClass(retsock, address, self)

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/opt/stack/nova/nova/console/websocketproxy.py", line 100, in
  __init__

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 99, in __init__

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  SimpleHTTPRequestHandler.__init__(self, req, addr, server)

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/lib/python3.6/socketserver.py", line 724, in __init__

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     self.handle()

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 315, in handle

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  SimpleHTTPRequestHandler.handle(self)

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/lib/python3.6/http/server.py", line 418, in handle

  Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  self.handle_one_request()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websocketserver.py", line 47, in
  handle_one_request

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  super(WebSocketRequestHandlerMixIn, self).handle_one_request()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     method()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websocketserver.py", line 60, in _websocket_do_GET

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  self.handle_upgrade()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 221, in handle_upgrade

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  WebSocketRequestHandlerMixIn.handle_upgrade(self)

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websocketserver.py", line 87, in handle_upgrade

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  self.handle_websocket()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-
  packages/websockify/websockifyserver.py", line 259, in
  handle_websocket

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  self.new_websocket_client()

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/opt/stack/nova/nova/console/websocketproxy.py", line 265, in
  new_websocket_client

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     tsock =
  self.server.security_proxy.connect(tenant_sock, tsock)

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/opt/stack/nova/nova/console/securityproxy/rfb.py", line 181, in
  connect

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     compute_sock
  = scheme.security_handshake(compute_sock)

  Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/opt/stack/nova/nova/console/rfb/authvencrypt.py", line 143, in
  security_handshake

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  ca_certs=CONF.vnc.vencrypt_ca_certs)

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-packages/eventlet/green/ssl.py", line
  414, in wrap_socket

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     return
  GreenSSLSocket(sock, *a, **kw)

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/local/lib/python3.6/dist-packages/eventlet/green/ssl.py", line
  106, in __init__

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy     ca_certs,
  do_handshake_on_connect and six.PY2, *args, **kw)

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy   File
  "/usr/lib/python3.6/ssl.py", line 809, in __init__

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy
  server_hostname)

  Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-
  novncproxy[22231]: ERROR nova.console.websocketproxy TypeError:
  _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket

  Sean Mooney opened an upstream bug report in eventlet [3].

  The last passing job run [4] and the failing run[2] have the same
  eventlet, websockify, openssl and libssl-dev versions.

  [1] https://zuul.opendev.org/t/openstack/builds?job_name=nova-
  next&project=openstack%2Fnova&branch=master

  [2]
  https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/screen-n
  -novnc-cell1.txt#29

  [3] https://github.com/eventlet/eventlet/issues/632

  [4]
  https://zuul.opendev.org/t/openstack/build/1d23dbdf6fa84702ab543602cae1ba67/log
  /job-output.txt

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888237/+subscriptions
References

[Bug 1888237] [NEW] nova-next job fails as novnc service fails with TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket
From: Balazs Gibizer, 2020-07-20