yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1897898] [NEW] Scenario test test_multiple_ports_portrange_remote is unstable

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1897898@xxxxxxxxxxxxxxxxxx>
Date: Wed, 30 Sep 2020 12:55:23 -0000
Reply-to: Bug 1897898 <1897898@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Scenario test
neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest,test_multiple_ports_portrange_remote
is pretty unstable and failing quite often.

Example of failure:
https://4546a2c16614badbfe53-42e1d9ee11051727c90d203e3ecae96b.ssl.cf5.rackcdn.com/753717/2/check
/neutron-tempest-plugin-scenario-linuxbridge/0c45aa5/testr_results.html

Logstash query: 
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20399%2C%20in%20test_multiple_ports_portrange_remote%5C%22

Stacktrace:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 399, in test_multiple_ports_portrange_remote
    self._verify_http_connection(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 60, in _verify_http_connection
    raise e
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 52, in _verify_http_connection
    ret = utils.call_url_remote(ssh_client, url)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 138, in call_url_remote
    return ssh_client.exec_command(cmd)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
    return self.call(f, *args, **kw)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
    do = self.iter(retry_state=retry_state)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
    return fut.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
    result = fn(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 178, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
  File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 209, in exec_command
    raise exceptions.SSHExecCommandFailed(
neutron_tempest_plugin.common.utils.SSHExecCommandFailed: Command 'curl http://10.1.0.12:80 --retry 3 --connect-timeout 2' failed, exit status: 28, stderr:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (28) Connection timed out after 2002 milliseconds

stdout:

** Affects: neutron
     Importance: Critical
         Status: Confirmed


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1897898

Title:
  Scenario test test_multiple_ports_portrange_remote is unstable

Status in neutron:
  Confirmed

Bug description:
  Scenario test
  neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest,test_multiple_ports_portrange_remote
  is pretty unstable and failing quite often.

  Example of failure:
  https://4546a2c16614badbfe53-42e1d9ee11051727c90d203e3ecae96b.ssl.cf5.rackcdn.com/753717/2/check
  /neutron-tempest-plugin-scenario-
  linuxbridge/0c45aa5/testr_results.html

  Logstash query: 
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20399%2C%20in%20test_multiple_ports_portrange_remote%5C%22

  Stacktrace:

  Traceback (most recent call last):
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 399, in test_multiple_ports_portrange_remote
      self._verify_http_connection(
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 60, in _verify_http_connection
      raise e
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 52, in _verify_http_connection
      ret = utils.call_url_remote(ssh_client, url)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/utils.py", line 138, in call_url_remote
      return ssh_client.exec_command(cmd)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
      return self.call(f, *args, **kw)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
      do = self.iter(retry_state=retry_state)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
      return fut.result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
      return self.__get_result()
    File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
      raise self._exception
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
      result = fn(*args, **kwargs)
    File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/common/ssh.py", line 178, in exec_command
      return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
    File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 209, in exec_command
      raise exceptions.SSHExecCommandFailed(
  neutron_tempest_plugin.common.utils.SSHExecCommandFailed: Command 'curl http://10.1.0.12:80 --retry 3 --connect-timeout 2' failed, exit status: 28, stderr:
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
  curl: (28) Connection timed out after 2002 milliseconds

  stdout:

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1897898/+subscriptions
Follow ups

[Bug 1897898] Re: Scenario test test_multiple_ports_portrange_remote is unstable
From: Launchpad Bug Tracker, 2022-02-09