canonical-ubuntu-qa team mailing list archive

Thread
Date
[Bug 2072501] Re: net:pmtu.sh of ubuntu_kselftests_net and subsequent subtests time out

To: canonical-ubuntu-qa@xxxxxxxxxxxxxxxxxxx
From: Mitchell Augustin <2072501@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Sep 2024 02:04:21 -0000
Reply-to: Bug 2072501 <2072501@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
I might have found a fix. I've been testing the latest pmtu.sh from June
on v6.11-rc6
(https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/tools/testing/selftests/net/pmtu.sh?h=v6.11-rc6&id=b7ce46fc614d4d31d438878ade1e4d0c3ef890c7)
on a few different configurations, and so far it seems like it fixes
this issue. Running the pmtu.sh loop with that version does not trigger
this bug after around 30 minutes w/ 5+ tests on Grace/Hopper. (In my
tests with the old pmtu.sh, I would see the `unregister_netdevice` dmesg
message in under a minute, reliably.)

This is the case when running the v6.11-rc6 pmtu.sh on both the
v6.11-rc6 kernel and the 6.8.0-1012-nvidia-64k kernel.

I'm also going to let this test loop run overnight on this version to
ensure the bug is truly not triggering anymore.

It isn't immediately obvious to me which of the changes in this patch
would impact this buggy behavior, but even if the problem does
eventually occur, it is *certainly* not as prevalent. I'm going to do
more digging into that patch tomorrow to get a better idea of what part
of it might be helping us here.

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2072501

Title:
  net:pmtu.sh of ubuntu_kselftests_net and subsequent subtests time out

Status in ubuntu-kernel-tests:
  New

Bug description:
  Found on N-nvidia-64k with node hinyari. After net:pmtu.sh times out,
  these tests also timeout until "TEST SYSTEM FAILURE DETECTED" is
  eventually triggered: net:udpgso.sh, net:ip_defrag.sh,
  net:udpgso_bench.sh, net:fib_rule_tests.sh.

  14:44:14 INFO | 	START	ubuntu_kselftests_net.net:pmtu.sh	ubuntu_kselftests_net.net:pmtu.sh	timeout=2700	timestamp=1719413054	localtime=Jun 26 14:44:14	
  14:44:14 DEBUG| Persistent state client._record_indent now set to 2
  14:44:14 DEBUG| Persistent state client.unexpected_reboot now set to ('ubuntu_kselftests_net.net:pmtu.sh', 'ubuntu_kselftests_net.net:pmtu.sh')
  14:44:14 DEBUG| Waiting for pid 43936 for 2700 seconds
  14:44:14 WARNI| System python is too old, crash handling disabled
  14:44:14 DEBUG| Running 'make run_tests -C net TEST_PROGS=pmtu.sh TEST_GEN_PROGS='' TEST_CUSTOM_PROGS='''
  14:44:14 DEBUG| [stdout] make: Entering directory '/home/ubuntu/autotest/client/tmp/ubuntu_kselftests_net/src/linux/tools/testing/selftests/net'
  14:44:14 DEBUG| [stdout] TAP version 13
  14:44:14 DEBUG| [stdout] 1..1
  14:44:14 DEBUG| [stdout] # timeout set to 0
  14:44:14 DEBUG| [stdout] # selftests: net: pmtu.sh
  14:44:14 DEBUG| [stdout] # TEST: ipv4: PMTU exceptions                                         [ OK ]
  14:44:14 DEBUG| [stdout] # TEST: ipv4: PMTU exceptions - nexthop objects                       [ OK ]
  14:44:14 DEBUG| [stdout] # TEST: ipv6: PMTU exceptions                                         [ OK ]
  14:44:14 DEBUG| [stdout] # TEST: ipv6: PMTU exceptions - nexthop objects                       [ OK ]
  14:44:14 DEBUG| [stdout] # TEST: ICMPv4 with DSCP and ECN: PMTU exceptions                     [ OK ]
  14:44:15 DEBUG| [stdout] # TEST: ICMPv4 with DSCP and ECN: PMTU exceptions - nexthop objects   [ OK ]
  14:44:15 DEBUG| [stdout] # 2024/06/26 14:44:15 socat[44926] E xioopen_ipdgram_listen(): unknown address family 0
  14:44:15 DEBUG| [stdout] # TEST: UDPv4 with DSCP and ECN: PMTU exceptions                      [ OK ]
  14:44:15 DEBUG| [stdout] # ./pmtu.sh: line 937: kill: (44926) - No such process
  14:44:15 DEBUG| [stdout] # 2024/06/26 14:44:15 socat[45066] E xioopen_ipdgram_listen(): unknown address family 0
  14:44:15 DEBUG| [stdout] # TEST: UDPv4 with DSCP and ECN: PMTU exceptions - nexthop objects    [ OK ]
  14:44:15 DEBUG| [stdout] # ./pmtu.sh: line 937: kill: (45066) - No such process
  14:44:16 DEBUG| [stdout] # TEST: IPv4 over vxlan4: PMTU exceptions                             [ OK ]
  14:44:17 DEBUG| [stdout] # TEST: IPv4 over vxlan4: PMTU exceptions - nexthop objects           [ OK ]
  14:44:18 DEBUG| [stdout] # TEST: IPv6 over vxlan4: PMTU exceptions                             [ OK ]
  14:44:19 DEBUG| [stdout] # TEST: IPv6 over vxlan4: PMTU exceptions - nexthop objects           [ OK ]
  14:44:20 DEBUG| [stdout] # TEST: IPv4 over vxlan6: PMTU exceptions                             [ OK ]
  14:44:22 DEBUG| [stdout] # TEST: IPv4 over vxlan6: PMTU exceptions - nexthop objects           [ OK ]
  14:44:23 DEBUG| [stdout] # TEST: IPv6 over vxlan6: PMTU exceptions                             [ OK ]
  14:44:24 DEBUG| [stdout] # TEST: IPv6 over vxlan6: PMTU exceptions - nexthop objects           [ OK ]
  14:44:25 DEBUG| [stdout] # TEST: IPv4 over geneve4: PMTU exceptions                            [ OK ]
  14:44:26 DEBUG| [stdout] # TEST: IPv4 over geneve4: PMTU exceptions - nexthop objects          [ OK ]
  14:44:27 DEBUG| [stdout] # TEST: IPv6 over geneve4: PMTU exceptions                            [ OK ]
  14:44:28 DEBUG| [stdout] # TEST: IPv6 over geneve4: PMTU exceptions - nexthop objects          [ OK ]
  14:44:30 DEBUG| [stdout] # TEST: IPv4 over geneve6: PMTU exceptions                            [ OK ]
  14:44:31 DEBUG| [stdout] # TEST: IPv4 over geneve6: PMTU exceptions - nexthop objects          [ OK ]
  14:44:32 DEBUG| [stdout] # TEST: IPv6 over geneve6: PMTU exceptions                            [ OK ]
  14:44:33 DEBUG| [stdout] # TEST: IPv6 over geneve6: PMTU exceptions - nexthop objects          [ OK ]
  14:44:35 DEBUG| [stdout] # TEST: IPv4, bridged vxlan4: PMTU exceptions                         [ OK ]
  14:44:37 DEBUG| [stdout] # TEST: IPv4, bridged vxlan4: PMTU exceptions - nexthop objects       [ OK ]
  14:44:40 DEBUG| [stdout] # TEST: IPv6, bridged vxlan4: PMTU exceptions                         [ OK ]
  14:44:42 DEBUG| [stdout] # TEST: IPv6, bridged vxlan4: PMTU exceptions - nexthop objects       [ OK ]
  14:44:44 DEBUG| [stdout] # TEST: IPv4, bridged vxlan6: PMTU exceptions                         [ OK ]
  14:44:46 DEBUG| [stdout] # TEST: IPv4, bridged vxlan6: PMTU exceptions - nexthop objects       [ OK ]
  14:44:49 DEBUG| [stdout] # TEST: IPv6, bridged vxlan6: PMTU exceptions                         [ OK ]
  14:44:51 DEBUG| [stdout] # TEST: IPv6, bridged vxlan6: PMTU exceptions - nexthop objects       [ OK ]
  14:44:53 DEBUG| [stdout] # TEST: IPv4, bridged geneve4: PMTU exceptions                        [ OK ]
  14:44:55 DEBUG| [stdout] # TEST: IPv4, bridged geneve4: PMTU exceptions - nexthop objects      [ OK ]
  14:44:58 DEBUG| [stdout] # TEST: IPv6, bridged geneve4: PMTU exceptions                        [ OK ]
  14:45:00 DEBUG| [stdout] # TEST: IPv6, bridged geneve4: PMTU exceptions - nexthop objects      [ OK ]
  14:45:02 DEBUG| [stdout] # TEST: IPv4, bridged geneve6: PMTU exceptions                        [ OK ]
  14:45:04 DEBUG| [stdout] # TEST: IPv4, bridged geneve6: PMTU exceptions - nexthop objects      [ OK ]
  14:45:07 DEBUG| [stdout] # TEST: IPv6, bridged geneve6: PMTU exceptions                        [ OK ]
  14:45:09 DEBUG| [stdout] # TEST: IPv6, bridged geneve6: PMTU exceptions - nexthop objects      [ OK ]
  14:45:09 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:09 DEBUG| [stdout] # TEST: IPv4, OVS vxlan4: PMTU exceptions                             [SKIP]
  14:45:09 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:09 DEBUG| [stdout] # TEST: IPv6, OVS vxlan4: PMTU exceptions                             [SKIP]
  14:45:09 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:09 DEBUG| [stdout] # TEST: IPv4, OVS vxlan6: PMTU exceptions                             [SKIP]
  14:45:09 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:09 DEBUG| [stdout] # TEST: IPv6, OVS vxlan6: PMTU exceptions                             [SKIP]
  14:45:09 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:09 DEBUG| [stdout] # TEST: IPv4, OVS geneve4: PMTU exceptions                            [SKIP]
  14:45:10 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:10 DEBUG| [stdout] # TEST: IPv6, OVS geneve4: PMTU exceptions                            [SKIP]
  14:45:10 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:10 DEBUG| [stdout] # TEST: IPv4, OVS geneve6: PMTU exceptions                            [SKIP]
  14:45:10 DEBUG| [stdout] #   ovs_bridge not supported
  14:45:10 DEBUG| [stdout] # TEST: IPv6, OVS geneve6: PMTU exceptions                            [SKIP]
  14:45:11 DEBUG| [stdout] # TEST: IPv4 over fou4: PMTU exceptions                               [ OK ]
  14:45:11 DEBUG| [stdout] # TEST: IPv4 over fou4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:11 DEBUG| [stdout] # TEST: IPv6 over fou4: PMTU exceptions                               [ OK ]
  14:45:12 DEBUG| [stdout] # TEST: IPv6 over fou4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:12 DEBUG| [stdout] # TEST: IPv4 over fou6: PMTU exceptions                               [ OK ]
  14:45:12 DEBUG| [stdout] # TEST: IPv4 over fou6: PMTU exceptions - nexthop objects             [ OK ]
  14:45:12 DEBUG| [stdout] # TEST: IPv6 over fou6: PMTU exceptions                               [ OK ]
  14:45:13 DEBUG| [stdout] # TEST: IPv6 over fou6: PMTU exceptions - nexthop objects             [ OK ]
  14:45:13 DEBUG| [stdout] # TEST: IPv4 over gue4: PMTU exceptions                               [ OK ]
  14:45:13 DEBUG| [stdout] # TEST: IPv4 over gue4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:13 DEBUG| [stdout] # TEST: IPv6 over gue4: PMTU exceptions                               [ OK ]
  14:45:14 DEBUG| [stdout] # TEST: IPv6 over gue4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:14 DEBUG| [stdout] # TEST: IPv4 over gue6: PMTU exceptions                               [ OK ]
  14:45:14 DEBUG| [stdout] # TEST: IPv4 over gue6: PMTU exceptions - nexthop objects             [ OK ]
  14:45:14 DEBUG| [stdout] # TEST: IPv6 over gue6: PMTU exceptions                               [ OK ]
  14:45:14 DEBUG| [stdout] # TEST: IPv6 over gue6: PMTU exceptions - nexthop objects             [ OK ]
  14:45:15 DEBUG| [stdout] # TEST: IPv4 over IPv4: PMTU exceptions                               [ OK ]
  14:45:15 DEBUG| [stdout] # TEST: IPv4 over IPv4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:15 DEBUG| [stdout] # TEST: IPv6 over IPv4: PMTU exceptions                               [ OK ]
  14:45:15 DEBUG| [stdout] # TEST: IPv6 over IPv4: PMTU exceptions - nexthop objects             [ OK ]
  14:45:15 DEBUG| [stdout] # TEST: IPv4 over IPv6: PMTU exceptions                               [ OK ]
  14:45:16 DEBUG| [stdout] # TEST: IPv4 over IPv6: PMTU exceptions - nexthop objects             [ OK ]
  14:45:16 DEBUG| [stdout] # TEST: IPv6 over IPv6: PMTU exceptions                               [ OK ]
  14:45:16 DEBUG| [stdout] # TEST: IPv6 over IPv6: PMTU exceptions - nexthop objects             [ OK ]
  15:29:14 INFO | Timer expired (2700 sec.), nuking pid 43936
  15:29:14 INFO | 		ERROR	ubuntu_kselftests_net.net:pmtu.sh	ubuntu_kselftests_net.net:pmtu.sh	timestamp=1719415754	localtime=Jun 26 15:29:14	Test timeout expired, rc=15
  15:29:14 INFO | 	END ERROR	ubuntu_kselftests_net.net:pmtu.sh	ubuntu_kselftests_net.net:pmtu.sh	timestamp=1719415754	localtime=Jun 26 15:29:14	
  15:29:14 DEBUG| Persistent state client._record_indent now set to 1
  15:29:14 DEBUG| Persistent state client.unexpected_reboot deleted
  15:29:14 DEBUG| Test has timeout: 2700 sec.
  15:29:14 INFO | 	START	ubuntu_kselftests_net.net:udpgso.sh	ubuntu_kselftests_net.net:udpgso.sh	timeout=2700	timestamp=1719415754	localtime=Jun 26 15:29:14	
  15:29:14 DEBUG| Persistent state client._record_indent now set to 2
  15:29:14 DEBUG| Persistent state client.unexpected_reboot now set to ('ubuntu_kselftests_net.net:udpgso.sh', 'ubuntu_kselftests_net.net:udpgso.sh')
  15:29:14 DEBUG| Waiting for pid 54956 for 2700 seconds
  15:29:14 WARNI| System python is too old, crash handling disabled
  15:29:14 DEBUG| Running 'make run_tests -C net TEST_PROGS=udpgso.sh TEST_GEN_PROGS='' TEST_CUSTOM_PROGS='''
  15:29:14 DEBUG| [stdout] make: Entering directory '/home/ubuntu/autotest/client/tmp/ubuntu_kselftests_net/src/linux/tools/testing/selftests/net'
  15:29:14 DEBUG| [stdout] TAP version 13
  15:29:14 DEBUG| [stdout] 1..1
  15:29:14 DEBUG| [stdout] # timeout set to 0
  15:29:14 DEBUG| [stdout] # selftests: net: udpgso.sh
  15:29:14 DEBUG| [stdout] # ipv4 cmsg
  16:14:14 INFO | Timer expired (2700 sec.), nuking pid 54956
  16:14:14 INFO | 		ERROR	ubuntu_kselftests_net.net:udpgso.sh	ubuntu_kselftests_net.net:udpgso.sh	timestamp=1719418454	localtime=Jun 26 16:14:14	Test timeout expired, rc=15
  16:14:14 INFO | 	END ERROR	ubuntu_kselftests_net.net:udpgso.sh	ubuntu_kselftests_net.net:udpgso.sh	timestamp=1719418454	localtime=Jun 26 
  ...

  
  Accompanied by the following dmesg entries, "waiting for veth_A-R1 to become free" appears to repeat until the system is shutdown:
  [  574.404587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  584.652587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  594.900587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  605.148587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  615.396587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  625.644587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  635.892587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  646.140587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  656.388587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  666.636587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  676.884586] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  687.132586] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  697.380587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  707.628586] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  717.876586] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  728.124587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  738.284631] INFO: task modprobe:54505 blocked for more than 122 seconds.
  [  738.291488]       Not tainted 6.8.0-1009-nvidia-64k #9-Ubuntu
  [  738.297364] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  [  738.372587] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  [  748.620586] unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 5
  ...

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2072501/+subscriptions
References

[Bug 2072501] [NEW] net:pmtu.sh of ubuntu_kselftests_net and subsequent subtests time out
From: Jacob Martin, 2024-07-08