yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #46514
[Bug 1547486] [NEW] ARPSpoofOFCtlTestCase functional gate tests failing intermittently
Public bug reported:
Tests inside
neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase
seems to fail intermittently. The root cause seems like to be the same
for all the tests...
Here are two different stack traces from two different patches...
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.288 | 2016-02-19 09:49:54.264 | ==============================
2016-02-19 09:49:54.312 | 2016-02-19 09:49:54.279 | Failed 1 tests - output below:
2016-02-19 09:49:54.321 | 2016-02-19 09:49:54.292 | ==============================
2016-02-19 09:49:54.332 | 2016-02-19 09:49:54.308 |
2016-02-19 09:49:54.335 | 2016-02-19 09:49:54.310 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_disable_network_port(native)
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.312 | ---------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.314 |
2016-02-19 09:49:54.343 | 2016-02-19 09:49:54.316 | Captured traceback:
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.317 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.322 | Traceback (most recent call last):
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.328 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 242, in test_arp_spoof_disable_network_port
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.351 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:49:54.378 | 2016-02-19 09:49:54.354 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:49:54.380 | 2016-02-19 09:49:54.357 | dst_ip])
2016-02-19 09:49:54.385 | 2016-02-19 09:49:54.361 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:49:54.386 | 2016-02-19 09:49:54.363 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:49:54.389 | 2016-02-19 09:49:54.365 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:49:54.392 | 2016-02-19 09:49:54.368 | raise RuntimeError(msg)
2016-02-19 09:49:54.395 | 2016-02-19 09:49:54.371 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:49:54.397 | 2016-02-19 09:49:54.374 |
2016-02-19 09:49:54.420 | 2016-02-19 09:49:54.376 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.378 | 2 packets transmitted, 0 received, 100% packet loss, time 1005ms
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.379 |
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.382 | ; Stderr:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |
** Affects: neutron
Importance: Undecided
Status: New
** Tags: gate-failure
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1547486
Title:
ARPSpoofOFCtlTestCase functional gate tests failing intermittently
Status in neutron:
New
Bug description:
Tests inside
neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase
seems to fail intermittently. The root cause seems like to be the same
for all the tests...
Here are two different stack traces from two different patches...
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.288 | 2016-02-19 09:49:54.264 | ==============================
2016-02-19 09:49:54.312 | 2016-02-19 09:49:54.279 | Failed 1 tests - output below:
2016-02-19 09:49:54.321 | 2016-02-19 09:49:54.292 | ==============================
2016-02-19 09:49:54.332 | 2016-02-19 09:49:54.308 |
2016-02-19 09:49:54.335 | 2016-02-19 09:49:54.310 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_disable_network_port(native)
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.312 | ---------------------------------------------------------------------------------------------------------------
2016-02-19 09:49:54.339 | 2016-02-19 09:49:54.314 |
2016-02-19 09:49:54.343 | 2016-02-19 09:49:54.316 | Captured traceback:
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.317 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.322 | Traceback (most recent call last):
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.328 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 242, in test_arp_spoof_disable_network_port
2016-02-19 09:49:54.375 | 2016-02-19 09:49:54.351 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:49:54.378 | 2016-02-19 09:49:54.354 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:49:54.380 | 2016-02-19 09:49:54.357 | dst_ip])
2016-02-19 09:49:54.385 | 2016-02-19 09:49:54.361 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:49:54.386 | 2016-02-19 09:49:54.363 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:49:54.389 | 2016-02-19 09:49:54.365 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:49:54.392 | 2016-02-19 09:49:54.368 | raise RuntimeError(msg)
2016-02-19 09:49:54.395 | 2016-02-19 09:49:54.371 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:49:54.397 | 2016-02-19 09:49:54.374 |
2016-02-19 09:49:54.420 | 2016-02-19 09:49:54.376 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.378 | 2 packets transmitted, 0 received, 100% packet loss, time 1005ms
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.379 |
2016-02-19 09:49:54.421 | 2016-02-19 09:49:54.382 | ; Stderr:
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:55.965 | 2016-02-19 09:12:55.947 | ==============================
2016-02-19 09:12:55.972 | 2016-02-19 09:12:55.951 | Failed 1 tests - output below:
2016-02-19 09:12:55.977 | 2016-02-19 09:12:55.959 | ==============================
2016-02-19 09:12:55.986 | 2016-02-19 09:12:55.967 |
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.976 | neutron.tests.functional.agent.test_ovs_flows.ARPSpoofOFCtlTestCase.test_arp_spoof_doesnt_block_normal_traffic(native)
2016-02-19 09:12:56.007 | 2016-02-19 09:12:55.978 | ----------------------------------------------------------------------------------------------------------------------
2016-02-19 09:12:56.009 | 2016-02-19 09:12:55.990 |
2016-02-19 09:12:56.025 | 2016-02-19 09:12:56.006 | Captured pythonlogging:
2016-02-19 09:12:56.034 | 2016-02-19 09:12:56.016 | ~~~~~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.036 | 2016-02-19 09:12:56.018 | DEBUG [oslo_policy._cache_handler] Reloading cached file /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.038 | 2016-02-19 09:12:56.020 | DEBUG [oslo_policy.policy] Reloaded policy file: /opt/stack/new/neutron/neutron/tests/etc/policy.json
2016-02-19 09:12:56.046 | 2016-02-19 09:12:56.028 |
2016-02-19 09:12:56.051 | 2016-02-19 09:12:56.030 |
2016-02-19 09:12:56.060 | 2016-02-19 09:12:56.036 | Captured traceback:
2016-02-19 09:12:56.062 | 2016-02-19 09:12:56.044 | ~~~~~~~~~~~~~~~~~~~
2016-02-19 09:12:56.066 | 2016-02-19 09:12:56.048 | Traceback (most recent call last):
2016-02-19 09:12:56.071 | 2016-02-19 09:12:56.053 | File "neutron/tests/functional/agent/test_ovs_flows.py", line 145, in test_arp_spoof_doesnt_block_normal_traffic
2016-02-19 09:12:56.073 | 2016-02-19 09:12:56.055 | net_helpers.assert_ping(self.src_namespace, self.dst_addr, count=2)
2016-02-19 09:12:56.078 | 2016-02-19 09:12:56.058 | File "neutron/tests/common/net_helpers.py", line 93, in assert_ping
2016-02-19 09:12:56.080 | 2016-02-19 09:12:56.062 | dst_ip])
2016-02-19 09:12:56.082 | 2016-02-19 09:12:56.064 | File "neutron/agent/linux/ip_lib.py", line 885, in execute
2016-02-19 09:12:56.084 | 2016-02-19 09:12:56.066 | log_fail_as_error=log_fail_as_error, **kwargs)
2016-02-19 09:12:56.086 | 2016-02-19 09:12:56.068 | File "neutron/agent/linux/utils.py", line 140, in execute
2016-02-19 09:12:56.094 | 2016-02-19 09:12:56.076 | raise RuntimeError(msg)
2016-02-19 09:12:56.102 | 2016-02-19 09:12:56.080 | RuntimeError: Exit code: 1; Stdin: ; Stdout: PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
2016-02-19 09:12:56.109 | 2016-02-19 09:12:56.091 |
2016-02-19 09:12:56.121 | 2016-02-19 09:12:56.101 | --- 192.168.0.2 ping statistics ---
2016-02-19 09:12:56.132 | 2016-02-19 09:12:56.113 | 2 packets transmitted, 0 received, 100% packet loss, time 1003ms
2016-02-19 09:12:56.137 | 2016-02-19 09:12:56.118 |
2016-02-19 09:12:56.138 | 2016-02-19 09:12:56.121 | ; Stderr:
2016-02-19 09:12:56.141 | 2016-02-19 09:12:56.123 |
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1547486/+subscriptions
