yahoo-eng-team team mailing list archive

[Slawek Kaplonski <1926109@xxxxxxxxxxxxxxxxxx>]

I believe this isn't an issue in the gate anymore. Maybe it was fixed by
https://review.opendev.org/c/openstack/tempest/+/761964

** Changed in: tempest
       Status: New => Fix Released

** Changed in: neutron
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926109

Title:
  SSH timeout (wait timeout) due to potential paramiko issue

Status in neutron:
  Fix Released
Status in tempest:
  Fix Released

Bug description:
  Recently, nothing changed on the test case, but we got this failure:
  https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html

  LOG:

  Traceback (most recent call last):
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
      utils.wait_until_true(
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 79, in wait_until_true
      while not predicate():
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 153, in no_fip_pf_connectivity
      return not fip_pf_connectivity(6)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 146, in fip_pf_connectivity
      self.check_servers_hostnames(
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py", line 537, in check_servers_hostnames
      ssh_client.get_hostname())
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 292, in get_hostname
      return self.exec_command('hostname')
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
      return self(f, *args, **kw)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
      do = self.iter(retry_state=retry_state)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
      return fut.result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
      result = fn(*args, **kwargs)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 171, in exec_command
      return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
    File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in exec_command
      channel.exec_command(cmd)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 72, in _check
      return func(self, *args, **kwds)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 257, in exec_command
      self._wait_for_event()
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 1219, in _wait_for_event
      self.event.wait()
    File "/usr/lib/python3.8/threading.py", line 558, in wait
      signaled = self._cond.wait(timeout)
    File "/usr/lib/python3.8/threading.py", line 302, in wait
      waiter.acquire()
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  fixtures._fixtures.timeout.TimeoutException

  
  Some search in the neutron LP bug list, there are some old bugs related to this project "paramiko":
  https://bugs.launchpad.net/neutron/+bug/1892861
  https://bugs.launchpad.net/neutron/+bug/1571486
  https://bugs.launchpad.net/neutron/+bug/1253896

  The issue seems to be _wait_for_event is waiting for something, and
  finally get timeout.

  After some searching in the repo of paramiko in github, I saw this issue:
  https://github.com/paramiko/paramiko/pull/1248
  With a commit for this:
  https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
  But it is not included in any release version.

  Some thoughts:
  1. if it is possible to not use paramiko?
  2. run (ssh) commands directly by some oslo libs?
  3. ping paramiko maintainers to release once to see if the test success rate can get better

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926109/+subscriptions