canonical-ubuntu-qa team mailing list archive
-
canonical-ubuntu-qa team
-
Mailing list archive
-
Message #01216
[Bug 2034057] [NEW] function_graph tracer in ftrace related tests triggers kernel bug (arch/x86/xen/spinlock.c:62) and kernel panic on AWS cloud c3.xlarge
Public bug reported:
Test:
* ftrace:test.d--00basic--basic2.tc from ubuntu_kselftests_ftrace
* test_enable_all_tracers() and test_enable_all_tracers() from ubuntu_ftrace_smoke_test
Will crash AWS instance c3.xlarge when testing the "function_graph"
tracer.
We have a similar issue filed against Azure (bug 1882669). Filing a new
bug report because on AWS this is affecting 5.4 ~ 6.2 AWS kernel.
However on Azure this is not affecting newer kernels.
Take B-aws-5.4-1108 for example, with the ubuntu_kselftests_ftrace test the system will crash with the following error message and reboot (similar to bug 2029917):
[ 211.675624] kernel BUG at /build/linux-aws-5.4-VHOhmt/linux-aws-5.4-5.4.0/arch/x86/xen/spinlock.c:62!
[ 211.678258] invalid opcode: 0000 [#1] SMP PTI
[ 211.679596] CPU: 1 PID: 14 Comm: cpuhp/1 Not tainted 5.4.0-1108-aws #116~18.04.1-Ubuntu
[ 211.681825] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
[ 211.683728] RIP: 0010:dummy_handler+0x4/0x10
[ 211.685042] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
[ 211.690314] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
[ 211.691934] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
[ 211.694036] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
[ 211.696159] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.698232] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
[ 211.700365] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
[ 211.702482] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
[ 211.704894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 211.706598] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
[ 211.708731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 211.710835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 211.713448] Call Trace:
[ 211.714796] <IRQ>
[ 211.715996] __handle_irq_event_percpu+0x44/0x1a0
[ 211.717907] handle_irq_event_percpu+0x32/0x80
[ 211.719759] handle_percpu_irq+0x3d/0x60
[ 211.721474] generic_handle_irq+0x28/0x40
[ 211.723184] handle_irq_for_port+0x8f/0xe0
[ 211.724988] evtchn_2l_handle_events+0x157/0x270
[ 211.726913] __xen_evtchn_do_upcall+0x76/0xe0
[ 211.728749] xen_evtchn_do_upcall+0x2b/0x40
[ 211.730520] xen_hvm_callback_vector+0xf/0x20
[ 211.732271] </IRQ>
[ 211.733425] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
[ 211.735470] Code: e8 70 3d 64 ff 4c 29 e0 4c 39 f0 76 cf 80 0b 08 eb 8a 90 90 90 0f 1f 44 00 00 55 48 89 e5 e8 a6 ad 66 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6 07
[ 211.741920] RSP: 0000:ffffaecd400fbcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[ 211.744980] RAX: 0000000000000001 RBX: ffff8aefa34b8700 RCX: 000000000002cc00
[ 211.747420] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
[ 211.749882] RBP: ffffaecd400fbcf8 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.752340] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000003b
[ 211.754785] R13: 0000000000000000 R14: ffff8aef95bdfa00 R15: ffff8aef95bdfaa4
[ 211.757249] __setup_irq+0x456/0x760
[ 211.758779] ? kmem_cache_alloc_trace+0x170/0x230
[ 211.760639] request_threaded_irq+0xfb/0x160
[ 211.762358] bind_ipi_to_irqhandler+0xba/0x1c0
[ 211.764124] ? xen_qlock_wait+0x90/0x90
[ 211.765734] ? snr_uncore_mmio_init+0x20/0x20
[ 211.767496] xen_init_lock_cpu+0x78/0xd0
[ 211.769135] ? snr_uncore_mmio_init+0x20/0x20
[ 211.770864] xen_cpu_up_online+0xe/0x20
[ 211.772500] cpuhp_invoke_callback+0x8a/0x580
[ 211.774233] cpuhp_thread_fun+0xb8/0x120
[ 211.775866] smpboot_thread_fn+0xfc/0x170
[ 211.777524] kthread+0x121/0x140
[ 211.778968] ? sort_range+0x30/0x30
[ 211.780508] ? kthread_park+0x90/0x90
[ 211.782073] ret_from_fork+0x35/0x40
[ 211.783622] Modules linked in: nls_iso8859_1 binfmt_misc serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ixgbevf
[ 211.796497] ---[ end trace bb7d4e9bb7f852cb ]---
[ 211.798303] RIP: 0010:dummy_handler+0x4/0x10
[ 211.800022] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
[ 211.806379] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
[ 211.808356] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
[ 211.810792] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
[ 211.813263] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.815713] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
[ 211.818162] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
[ 211.820637] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
[ 211.823821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 211.825904] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
[ 211.828357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 211.830829] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 211.833284] Kernel panic - not syncing: Fatal exception in interrupt
[ 211.835575] Kernel Offset: 0x33600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Test output:
+ echo 1
+ . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
+ test -f available_tracers
+ cat available_tracers
+ echo hwlat
+ echo blk
+ echo mmiotrace
+ echo function_graph
(Test interrupted here)
To verify this, you can hack linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc and add "read -p" to check what tracer we're testing:
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
# description: Basic test for tracers
# flags: instance
test -f available_tracers
for t in `cat available_tracers`; do
read -p "testing $t" foo
echo $t > current_tracer
done
echo nop > current_tracer
And run this manually with sudo ./ftracetest -vvv
test.d/00basic/basic2.tc, you will see the system crashes right away
with "function_graph".
For ubuntu_ftrace_smoke_test, the "function_graph" tracer will be tested
with test_function_graph_tracer(). And since "function_graph" tracer is
in /sys/kernel/debug/tracing/available_tracers, it will be tested with
test_enable_all_tracers() as well.
** Affects: ubuntu-kernel-tests
Importance: Undecided
Status: New
** Affects: linux-aws (Ubuntu)
Importance: Undecided
Status: New
** Affects: linux-aws (Ubuntu Focal)
Importance: Undecided
Status: New
** Affects: linux-aws (Ubuntu Jammy)
Importance: Undecided
Status: New
** Affects: linux-aws (Ubuntu Lunar)
Importance: Undecided
Status: New
** Tags: 5.15 5.4 6.2 aws focal jammy lunar ubuntu-ftrace-smoke-test ubuntu-kselftests-ftrace
** Also affects: linux-aws (Ubuntu)
Importance: Undecided
Status: New
** Also affects: linux-aws (Ubuntu Focal)
Importance: Undecided
Status: New
** Also affects: linux-aws (Ubuntu Jammy)
Importance: Undecided
Status: New
** Also affects: linux-aws (Ubuntu Lunar)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2034057
Title:
function_graph tracer in ftrace related tests triggers kernel bug
(arch/x86/xen/spinlock.c:62) and kernel panic on AWS cloud c3.xlarge
Status in ubuntu-kernel-tests:
New
Status in linux-aws package in Ubuntu:
New
Status in linux-aws source package in Focal:
New
Status in linux-aws source package in Jammy:
New
Status in linux-aws source package in Lunar:
New
Bug description:
Test:
* ftrace:test.d--00basic--basic2.tc from ubuntu_kselftests_ftrace
* test_enable_all_tracers() and test_enable_all_tracers() from ubuntu_ftrace_smoke_test
Will crash AWS instance c3.xlarge when testing the "function_graph"
tracer.
We have a similar issue filed against Azure (bug 1882669). Filing a
new bug report because on AWS this is affecting 5.4 ~ 6.2 AWS kernel.
However on Azure this is not affecting newer kernels.
Take B-aws-5.4-1108 for example, with the ubuntu_kselftests_ftrace test the system will crash with the following error message and reboot (similar to bug 2029917):
[ 211.675624] kernel BUG at /build/linux-aws-5.4-VHOhmt/linux-aws-5.4-5.4.0/arch/x86/xen/spinlock.c:62!
[ 211.678258] invalid opcode: 0000 [#1] SMP PTI
[ 211.679596] CPU: 1 PID: 14 Comm: cpuhp/1 Not tainted 5.4.0-1108-aws #116~18.04.1-Ubuntu
[ 211.681825] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
[ 211.683728] RIP: 0010:dummy_handler+0x4/0x10
[ 211.685042] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
[ 211.690314] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
[ 211.691934] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
[ 211.694036] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
[ 211.696159] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.698232] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
[ 211.700365] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
[ 211.702482] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
[ 211.704894] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 211.706598] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
[ 211.708731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 211.710835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 211.713448] Call Trace:
[ 211.714796] <IRQ>
[ 211.715996] __handle_irq_event_percpu+0x44/0x1a0
[ 211.717907] handle_irq_event_percpu+0x32/0x80
[ 211.719759] handle_percpu_irq+0x3d/0x60
[ 211.721474] generic_handle_irq+0x28/0x40
[ 211.723184] handle_irq_for_port+0x8f/0xe0
[ 211.724988] evtchn_2l_handle_events+0x157/0x270
[ 211.726913] __xen_evtchn_do_upcall+0x76/0xe0
[ 211.728749] xen_evtchn_do_upcall+0x2b/0x40
[ 211.730520] xen_hvm_callback_vector+0xf/0x20
[ 211.732271] </IRQ>
[ 211.733425] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
[ 211.735470] Code: e8 70 3d 64 ff 4c 29 e0 4c 39 f0 76 cf 80 0b 08 eb 8a 90 90 90 0f 1f 44 00 00 55 48 89 e5 e8 a6 ad 66 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6 07
[ 211.741920] RSP: 0000:ffffaecd400fbcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[ 211.744980] RAX: 0000000000000001 RBX: ffff8aefa34b8700 RCX: 000000000002cc00
[ 211.747420] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
[ 211.749882] RBP: ffffaecd400fbcf8 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.752340] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000003b
[ 211.754785] R13: 0000000000000000 R14: ffff8aef95bdfa00 R15: ffff8aef95bdfaa4
[ 211.757249] __setup_irq+0x456/0x760
[ 211.758779] ? kmem_cache_alloc_trace+0x170/0x230
[ 211.760639] request_threaded_irq+0xfb/0x160
[ 211.762358] bind_ipi_to_irqhandler+0xba/0x1c0
[ 211.764124] ? xen_qlock_wait+0x90/0x90
[ 211.765734] ? snr_uncore_mmio_init+0x20/0x20
[ 211.767496] xen_init_lock_cpu+0x78/0xd0
[ 211.769135] ? snr_uncore_mmio_init+0x20/0x20
[ 211.770864] xen_cpu_up_online+0xe/0x20
[ 211.772500] cpuhp_invoke_callback+0x8a/0x580
[ 211.774233] cpuhp_thread_fun+0xb8/0x120
[ 211.775866] smpboot_thread_fn+0xfc/0x170
[ 211.777524] kthread+0x121/0x140
[ 211.778968] ? sort_range+0x30/0x30
[ 211.780508] ? kthread_park+0x90/0x90
[ 211.782073] ret_from_fork+0x35/0x40
[ 211.783622] Modules linked in: nls_iso8859_1 binfmt_misc serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ixgbevf
[ 211.796497] ---[ end trace bb7d4e9bb7f852cb ]---
[ 211.798303] RIP: 0010:dummy_handler+0x4/0x10
[ 211.800022] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
[ 211.806379] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
[ 211.808356] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
[ 211.810792] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
[ 211.813263] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
[ 211.815713] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
[ 211.818162] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
[ 211.820637] FS: 0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
[ 211.823821] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 211.825904] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
[ 211.828357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 211.830829] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 211.833284] Kernel panic - not syncing: Fatal exception in interrupt
[ 211.835575] Kernel Offset: 0x33600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
Test output:
+ echo 1
+ . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
+ test -f available_tracers
+ cat available_tracers
+ echo hwlat
+ echo blk
+ echo mmiotrace
+ echo function_graph
(Test interrupted here)
To verify this, you can hack linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc and add "read -p" to check what tracer we're testing:
#!/bin/sh
# SPDX-License-Identifier: GPL-2.0
# description: Basic test for tracers
# flags: instance
test -f available_tracers
for t in `cat available_tracers`; do
read -p "testing $t" foo
echo $t > current_tracer
done
echo nop > current_tracer
And run this manually with sudo ./ftracetest -vvv
test.d/00basic/basic2.tc, you will see the system crashes right away
with "function_graph".
For ubuntu_ftrace_smoke_test, the "function_graph" tracer will be
tested with test_function_graph_tracer(). And since "function_graph"
tracer is in /sys/kernel/debug/tracing/available_tracers, it will be
tested with test_enable_all_tracers() as well.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2034057/+subscriptions
Follow ups
