yahoo-eng-team team mailing list archive

[yatin <2115026@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Fails as:-
ft1.24: neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_direct_route_for_address_scopetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 2234, in test_direct_route_for_address_scope
    net_helpers.assert_ping(src_machine.namespace, machine_same_scope.ip)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 119, in assert_ping
    raise exc
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 115, in assert_ping
    ns_ip_wrapper.netns.execute(cmd, privsep_exec=True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 776, in execute
    return utils.execute(cmd, check_exit_code=check_exit_code,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
    raise exceptions.ProcessExecutionError(msg,
neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'test-bf742106-6c0e-426a-b5ac-e8469c6c3f68', 'ping', '-W', 1, '-c', 3, '35.4.0.14']; Stdin: ; Stdout: PING 35.4.0.14 (35.4.0.14) 56(84) bytes of data.

--- 35.4.0.14 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2054ms

; Stderr:


Example failures:-
   - https://cf4c4bf28df114e17db2-adf31c6c395d3cc6dc699662249bf599.ssl.cf1.rackcdn.com/openstack/2483dda37db8414890961db806384973/testr_results.html
   - https://375f698e05cdbc99e0f2-291b1590cac729e819e1c5a3e1181673.ssl.cf2.rackcdn.com/openstack/c94afce8617e45d381c3dbba6e318353/testr_results.html
   - https://16fe99781a414f8f22c6-d9c28f5658476db1e4ca5968d196888d.ssl.cf5.rackcdn.com/openstack/54aab3908c654b62bedd8baa4ae7671a/testr_results.html

Reproducible locally too randomly, seeing different behavior in some failures
- The failed ping passed after manually ran ping from namespace
- In one case noticed snat- and fip- namespace were missing deleted by concurrent test
- In another case ping started work after pinging the q-router port ip

Running this tests single threaded should help in lowering the failures

** Affects: neutron
     Importance: High
     Assignee: yatin (yatinkarel)
         Status: New


** Tags: functional-tests gate-failure

** Changed in: neutron
     Assignee: (unassigned) => yatin (yatinkarel)

** Changed in: neutron
   Importance: Undecided => High

** Tags added: functional-tests gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2115026

Title:
  test_direct_route_for_address_scope Functional test randomly fails

Status in neutron:
  New

Bug description:
  Fails as:-
  ft1.24: neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_direct_route_for_address_scopetesttools.testresult.real._StringException: Traceback (most recent call last):
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
      return f(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_dvr_router.py", line 2234, in test_direct_route_for_address_scope
      net_helpers.assert_ping(src_machine.namespace, machine_same_scope.ip)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 119, in assert_ping
      raise exc
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 115, in assert_ping
      ns_ip_wrapper.netns.execute(cmd, privsep_exec=True)
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 776, in execute
      return utils.execute(cmd, check_exit_code=check_exit_code,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 156, in execute
      raise exceptions.ProcessExecutionError(msg,
  neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'test-bf742106-6c0e-426a-b5ac-e8469c6c3f68', 'ping', '-W', 1, '-c', 3, '35.4.0.14']; Stdin: ; Stdout: PING 35.4.0.14 (35.4.0.14) 56(84) bytes of data.

  --- 35.4.0.14 ping statistics ---
  3 packets transmitted, 0 received, 100% packet loss, time 2054ms

  ; Stderr:

  
  Example failures:-
     - https://cf4c4bf28df114e17db2-adf31c6c395d3cc6dc699662249bf599.ssl.cf1.rackcdn.com/openstack/2483dda37db8414890961db806384973/testr_results.html
     - https://375f698e05cdbc99e0f2-291b1590cac729e819e1c5a3e1181673.ssl.cf2.rackcdn.com/openstack/c94afce8617e45d381c3dbba6e318353/testr_results.html
     - https://16fe99781a414f8f22c6-d9c28f5658476db1e4ca5968d196888d.ssl.cf5.rackcdn.com/openstack/54aab3908c654b62bedd8baa4ae7671a/testr_results.html

  Reproducible locally too randomly, seeing different behavior in some failures
  - The failed ping passed after manually ran ping from namespace
  - In one case noticed snat- and fip- namespace were missing deleted by concurrent test
  - In another case ping started work after pinging the q-router port ip

  Running this tests single threaded should help in lowering the
  failures

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2115026/+subscriptions