← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #85896

[Bug 1926109] [NEW] SSH timeout (wait timeout) due to potential paramiko issue

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Recently, nothing changed on the test case, but we got this failure:
https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html

LOG:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
    utils.wait_until_true(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 79, in wait_until_true
    while not predicate():
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 153, in no_fip_pf_connectivity
    return not fip_pf_connectivity(6)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 146, in fip_pf_connectivity
    self.check_servers_hostnames(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py", line 537, in check_servers_hostnames
    ssh_client.get_hostname())
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 292, in get_hostname
    return self.exec_command('hostname')
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
    return self(f, *args, **kw)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
    do = self.iter(retry_state=retry_state)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
    return fut.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 171, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
  File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in exec_command
    channel.exec_command(cmd)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 72, in _check
    return func(self, *args, **kwds)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 257, in exec_command
    self._wait_for_event()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 1219, in _wait_for_event
    self.event.wait()
  File "/usr/lib/python3.8/threading.py", line 558, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException


Some search in the neutron LP bug list, there are some old bugs related to this project "paramiko":
https://bugs.launchpad.net/neutron/+bug/1892861
https://bugs.launchpad.net/neutron/+bug/1571486
https://bugs.launchpad.net/neutron/+bug/1253896

The issue seems to be _wait_for_event is waiting for something, and
finally get timeout.

After some searching in the repo of paramiko in github, I saw this issue:
https://github.com/paramiko/paramiko/pull/1248
With a commit for this:
https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
But it is not included in any release version.

Some thoughts:
1. if it is possible to not use paramiko?
2. run (ssh) commands directly by some oslo libs?
3. ping paramiko maintainers to release once to see if the test success rate can get better

** Affects: neutron
     Importance: Undecided
         Status: New

** Affects: tempest
     Importance: Undecided
         Status: New

** Also affects: tempest
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1926109

Title:
  SSH timeout (wait timeout) due to potential paramiko issue

Status in neutron:
  New
Status in tempest:
  New

Bug description:
  Recently, nothing changed on the test case, but we got this failure:
  https://04a9f9fdd9afdf12de4e-f889a65b4dfb1f628c8309e9eb44b225.ssl.cf2.rackcdn.com/787304/5/check/neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid/bb9e9a9/testr_results.html

  LOG:

  Traceback (most recent call last):
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 160, in test_port_forwarding_editing_and_deleting_tcp_rule
      utils.wait_until_true(
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 79, in wait_until_true
      while not predicate():
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 153, in no_fip_pf_connectivity
      return not fip_pf_connectivity(6)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_port_forwardings.py", line 146, in fip_pf_connectivity
      self.check_servers_hostnames(
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py", line 537, in check_servers_hostnames
      ssh_client.get_hostname())
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 292, in get_hostname
      return self.exec_command('hostname')
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
      return self(f, *args, **kw)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __call__
      do = self.iter(retry_state=retry_state)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
      return fut.result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
      result = fn(*args, **kwargs)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 171, in exec_command
      return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
    File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 162, in exec_command
      channel.exec_command(cmd)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 72, in _check
      return func(self, *args, **kwds)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 257, in exec_command
      self._wait_for_event()
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/paramiko/channel.py", line 1219, in _wait_for_event
      self.event.wait()
    File "/usr/lib/python3.8/threading.py", line 558, in wait
      signaled = self._cond.wait(timeout)
    File "/usr/lib/python3.8/threading.py", line 302, in wait
      waiter.acquire()
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
      raise TimeoutException()
  fixtures._fixtures.timeout.TimeoutException

  
  Some search in the neutron LP bug list, there are some old bugs related to this project "paramiko":
  https://bugs.launchpad.net/neutron/+bug/1892861
  https://bugs.launchpad.net/neutron/+bug/1571486
  https://bugs.launchpad.net/neutron/+bug/1253896

  The issue seems to be _wait_for_event is waiting for something, and
  finally get timeout.

  After some searching in the repo of paramiko in github, I saw this issue:
  https://github.com/paramiko/paramiko/pull/1248
  With a commit for this:
  https://github.com/paramiko/paramiko/pull/1248/commits/b059139ea712a46f632fe20e3c278414f311c21d
  But it is not included in any release version.

  Some thoughts:
  1. if it is possible to not use paramiko?
  2. run (ssh) commands directly by some oslo libs?
  3. ping paramiko maintainers to release once to see if the test success rate can get better

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1926109/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next