yahoo-eng-team team mailing list archive

[OpenStack Infra <2069744@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/neutron/+/922218
Committed: https://opendev.org/openstack/neutron/commit/bf82263df027a5c5213422feb12eefd6de9fa867
Submitter: "Zuul (22348)"
Branch:    master

commit bf82263df027a5c5213422feb12eefd6de9fa867
Author: yatinkarel <ykarel@xxxxxxxxxx>
Date:   Tue Jun 18 19:08:14 2024 +0530

    [FT] Run test_periodic_sync_routers_task tests serially
    
    It's seen that these tests interfered(removed router namespace) with
    test_metadata_proxy_rate_limiting_ipv6 but can interfere with others
    too, let's run these serially to avoid random failures.
    
    Following tests will run serially now:-
    - test_periodic_sync_routers_task
    - test_periodic_sync_routers_task_routers_deleted_while_agent_down
    - test_periodic_sync_routers_task_routers_deleted_while_agent_sync
    
    Closes-Bug: #2069744
    Change-Id: I34598cb9ad39c96f5e46d98af1185992c5eb3446


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2069744

Title:
  [FT] Run test_periodic_sync_routers_task tests serially

Status in neutron:
  Fix Released

Bug description:
  It's seen that these tests interfered(removed router namespace) with test_metadata_proxy_rate_limiting_ipv6 but can interfere with others too, let's run these serially to avoid random failures.
      
  Seen in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_08a/885449/22/check/neutron-functional-with-uwsgi/08af95d/testr_results.html

  Fails like:-
  ft1.5: neutron.tests.functional.agent.l3.test_metadata_proxy.MetadataL3AgentTestCase.test_metadata_proxy_rate_limiting_ipv6testtools.testresult.real._StringException: stderr: {{{
  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
      timer()
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/eventlet/greenthread.py", line 265, in main
      result = function(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
      return f(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 274, in _check_child_processes
      self._execute_action(service_id)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 285, in _execute_action
      action_function(service_id)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 291, in _respawn_action
      self._monitored_processes[service_id].enable()
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/external_process.py", line 105, in enable
      ip_wrapper.netns.execute(cmd, addl_env=self.cmd_addl_env,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
      return utils.execute(cmd, check_exit_code=check_exit_code,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
      raise exceptions.ProcessExecutionError(msg,
  neutron_lib.exceptions.ProcessExecutionError: Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory
  }}}

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 109, in _query_metadata_proxy
      raw_headers = machine.execute(cmd)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 56, in execute
      return ns_ip_wrapper.netns.execute(*args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
      return utils.execute(cmd, check_exit_code=check_exit_code,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
      raise exceptions.ProcessExecutionError(msg,
  neutron_lib.exceptions.ProcessExecutionError: Exit code: 28; Cmd: ['ip', 'netns', 'exec', 'test-458fcfac-f975-44ef-8a9d-86076a7af8c9', 'curl', '--max-time', 60, '-D-', 'http://[fe80::a9fe:a9fe%port0e40b9]:80']; Stdin: ; Stdout: ; Stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

  curl: (28) Connection timed out after 60000 milliseconds

  
  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 194, in test_metadata_proxy_rate_limiting_ipv6
      self._test_rate_limiting(2, machine, ipv6=True, interface=interface)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 169, in _test_rate_limiting
      firstline = self._query_metadata_proxy(machine, ipv6=ipv6,
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 116, in _query_metadata_proxy
      self.fail('metadata proxy unreachable '
    File "/usr/lib/python3.10/unittest/case.py", line 675, in fail
      raise self.failureException(msg)
  AssertionError: metadata proxy unreachable on http://[fe80::a9fe:a9fe%port0e40b9]:80 before timeout

  From test_periodic_sync_routers_task_routers_deleted_while_agent_sync.txt
  2024-06-11 01:28:12.647 38519 DEBUG neutron.privileged.agent.linux.ip_lib [-] Namespace qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624 deleted. remove_netns /home/zuul/src/opendev.org/openstack/neutron/neutron/privileged/agent/linux/ip_lib.py:607

  From failing test:-
  2024-06-11 01:29:20.707 38222 ERROR neutron.agent.linux.utils [-] Exit code: 255; Cmd: ['ip', 'netns', 'exec', 'qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624', 'env', 'PROCESS_TAG=haproxy-0156eaec-06cc-43a7-83cb-1a72504f7624', 'haproxy', '-f', '/tmp/tmpgh6ctwf3/tmpuategvfx/ns-metadata-proxy/0156eaec-06cc-43a7-83cb-1a72504f7624.conf']; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-0156eaec-06cc-43a7-83cb-1a72504f7624": No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2069744/+subscriptions