yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2069744] [NEW] [FT] Run test_periodic_sync_routers_task tests serially

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2069744@xxxxxxxxxxxxxxxxxx>
Date: Tue, 18 Jun 2024 13:48:24 -0000
Reply-to: Bug 2069744 <2069744@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

It's seen that these tests interfered(removed router namespace) with test_metadata_proxy_rate_limiting_ipv6 but can interfere with others too, let's run these serially to avoid random failures.
    
Seen in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08a/885449/22/check/neutron-functional-with-uwsgi/08af95d/testr_results.html

Fails like:-
ft1.5: neutron.tests.functional.agent.l3.test_metadata_proxy.MetadataL3AgentTestCase.test_metadata_proxy_rate_limiting_ipv6testtools.testresult.real._StringException: stderr: {{{
Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
    timer()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
    cb(*args, **kw)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
    result = function(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
    return f(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 274, in _check_child_processes
    self._execute_action(service_id)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 285, in _execute_action
    action_function(service_id)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 291, in _respawn_action
    self._monitored_processes[service_id].enable()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 105, in enable
    ip_wrapper.netns.execute(cmd, addl_env=self.cmd_addl_env,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
    return utils.execute(cmd, check_exit_code=check_exit_code,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
    raise exceptions.ProcessExecutionError(msg,
neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory
}}}

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 109, in _query_metadata_proxy
    raw_headers = machine.execute(cmd)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 56, in execute
    return ns_ip_wrapper.netns.execute(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
    return utils.execute(cmd, check_exit_code=check_exit_code,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
    raise exceptions.ProcessExecutionError(msg,
neutron_lib.exceptions.ProcessExecutionError: Exit code: 28; Cmd: ['ip', 'netns', 'exec', 'test-458fcfac-f975-44ef-8a9d-86076a7af8c9', 'curl', '--max-time', 60, '-D-', 'http://[fe80::a9fe:a9fe%port0e40b9]:80']; Stdin: ; Stdout: ; Stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

curl: (28) Connection timed out after 60000 milliseconds


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 194, in test_metadata_proxy_rate_limiting_ipv6
    self._test_rate_limiting(2, machine, ipv6=True, interface=interface)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 169, in _test_rate_limiting
    firstline = self._query_metadata_proxy(machine, ipv6=ipv6,
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 116, in _query_metadata_proxy
    self.fail('metadata proxy unreachable '
  File "/usr/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)
AssertionError: metadata proxy unreachable on http://[fe80::a9fe:a9fe%port0e40b9]:80 before timeout

>From test_periodic_sync_routers_task_routers_deleted_while_agent_sync.txt
2024-06-11 01:28:12.647 38519 DEBUG neutron.privileged.agent.linux.ip_lib [-] Namespace qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624 deleted. remove_netns /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:607

>From failing test:-
2024-06-11 01:29:20.707 38222 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory

** Affects: neutron
     Importance: Medium
         Status: In Progress


** Tags: functional-tests

** Changed in: neutron
   Importance: Undecided => Medium

** Changed in: neutron
       Status: New => Triaged

** Tags added: functional-tests

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2069744

Title:
  [FT] Run test_periodic_sync_routers_task tests serially

Status in neutron:
  In Progress

Bug description:
  It's seen that these tests interfered(removed router namespace) with test_metadata_proxy_rate_limiting_ipv6 but can interfere with others too, let's run these serially to avoid random failures.
      
  Seen in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08a/885449/22/check/neutron-functional-with-uwsgi/08af95d/testr_results.html

  Fails like:-
  ft1.5: neutron.tests.functional.agent.l3.test_metadata_proxy.MetadataL3AgentTestCase.test_metadata_proxy_rate_limiting_ipv6testtools.testresult.real._StringException: stderr: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
      timer()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
      result = function(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 274, in _check_child_processes
      self._execute_action(service_id)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 285, in _execute_action
      action_function(service_id)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 291, in _respawn_action
      self._monitored_processes[service_id].enable()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 105, in enable
      ip_wrapper.netns.execute(cmd, addl_env=self.cmd_addl_env,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
      return utils.execute(cmd, check_exit_code=check_exit_code,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
      raise exceptions.ProcessExecutionError(msg,
  neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 109, in _query_metadata_proxy
      raw_headers = machine.execute(cmd)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 56, in execute
      return ns_ip_wrapper.netns.execute(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
      return utils.execute(cmd, check_exit_code=check_exit_code,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
      raise exceptions.ProcessExecutionError(msg,
  neutron_lib.exceptions.ProcessExecutionError: Exit code: 28; Cmd: ['ip', 'netns', 'exec', 'test-458fcfac-f975-44ef-8a9d-86076a7af8c9', 'curl', '--max-time', 60, '-D-', 'http://[fe80::a9fe:a9fe%port0e40b9]:80']; Stdin: ; Stdout: ; Stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

  curl: (28) Connection timed out after 60000 milliseconds

  
  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 194, in test_metadata_proxy_rate_limiting_ipv6
      self._test_rate_limiting(2, machine, ipv6=True, interface=interface)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 169, in _test_rate_limiting
      firstline = self._query_metadata_proxy(machine, ipv6=ipv6,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 116, in _query_metadata_proxy
      self.fail('metadata proxy unreachable '
    File "/usr/lib/python3.10/unittest/case.py", line 675, in fail
      raise self.failureException(msg)
  AssertionError: metadata proxy unreachable on http://[fe80::a9fe:a9fe%port0e40b9]:80 before timeout

  From test_periodic_sync_routers_task_routers_deleted_while_agent_sync.txt
  2024-06-11 01:28:12.647 38519 DEBUG neutron.privileged.agent.linux.ip_lib [-] Namespace qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624 deleted. remove_netns /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:607

  From failing test:-
  2024-06-11 01:29:20.707 38222 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2069744/+subscriptions
Follow ups

[Bug 2069744] Re: [FT] Run test_periodic_sync_routers_task tests serially
From: OpenStack Infra, 2024-06-18