yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1809418] [NEW] ironic serial console port validate always failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: int32bit <krystism@xxxxxxxxx>
Date: Fri, 21 Dec 2018 08:27:36 -0000
Reply-to: Bug 1809418 <1809418@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I deployed latest ironic and used IPMI_socat console interface alongs
with nova-serialproxy service, but always failed to open console in
dashboard, and the nova-serialproxy error as follows:

2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy Traceback (most recent call last):
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 930, in top_new_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     client = self.do_handshake(startsock, address)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 860, in do_handshake
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.RequestHandlerClass(retsock, address, self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 324, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 114, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.__init__(self, req, addr, server)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.handle()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 581, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.handle(self)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.handle_one_request()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     method()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 543, in do_GET
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     if not self.handle_websocket():
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 531, in handle_websocket
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.new_websocket_client()
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 236, in new_websocket_client
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     connect_info = self._get_connect_info(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 192, in _get_connect_info
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     connect_info = self._get_connect_info_database(ctxt, token)
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 166, in _get_connect_info_database
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     raise exception.InvalidToken(token='***')
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy InvalidToken: The token '***' is invalid or has expired
2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy
2018-12-20 19:11:40.867 14996 DEBUG nova.console.websocketproxy [-] 197.3.133.254: new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:875

So I created a breakpoint in this code and found the port type must be
str in nova-serialproxy side as we converted it to str type forcefully
before rpc validate_console to compute service[1]. But on the compute
side, the drive may be return int type like ironic serial console[2]. So
the validate_console_port may always return False as type
inconsistence[3].


[1] https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L123-L125
[2] https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1819
[3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5369

** Affects: nova
     Importance: Undecided
     Assignee: int32bit (int32bit)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => int32bit (int32bit)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1809418

Title:
  ironic serial console port validate always failed

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  I deployed latest ironic and used IPMI_socat console interface alongs
  with nova-serialproxy service, but always failed to open console in
  dashboard, and the nova-serialproxy error as follows:

  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy Traceback (most recent call last):
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 930, in top_new_client
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     client = self.do_handshake(startsock, address)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 860, in do_handshake
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.RequestHandlerClass(retsock, address, self)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 324, in __init__
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 114, in __init__
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.__init__(self, req, addr, server)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.handle()
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 581, in handle
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     SimpleHTTPRequestHandler.handle(self)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.handle_one_request()
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib64/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     method()
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 543, in do_GET
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     if not self.handle_websocket():
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/websockify/websocket.py", line 531, in handle_websocket
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     self.new_websocket_client()
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 236, in new_websocket_client
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     connect_info = self._get_connect_info(ctxt, token)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 192, in _get_connect_info
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     connect_info = self._get_connect_info_database(ctxt, token)
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy   File "/usr/lib/python2.7/site-packages/nova/console/websocketproxy.py", line 166, in _get_connect_info_database
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy     raise exception.InvalidToken(token='***')
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy InvalidToken: The token '***' is invalid or has expired
  2018-12-20 19:11:16.642 15391 ERROR nova.console.websocketproxy
  2018-12-20 19:11:40.867 14996 DEBUG nova.console.websocketproxy [-] 197.3.133.254: new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:875

  So I created a breakpoint in this code and found the port type must be
  str in nova-serialproxy side as we converted it to str type forcefully
  before rpc validate_console to compute service[1]. But on the compute
  side, the drive may be return int type like ironic serial console[2].
  So the validate_console_port may always return False as type
  inconsistence[3].

  
  [1] https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L123-L125
  [2] https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1819
  [3] https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5369

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1809418/+subscriptions
Follow ups

[Bug 1809418] Re: ironic serial console port validate always failed
From: OpenStack Infra, 2019-01-23
[Bug 1809418] Re: ironic serial console port validate always failed
From: Matt Riedemann, 2019-01-22