canonical-ubuntu-qa team mailing list archive

[Po-Hsu Lin <2034057@xxxxxxxxxxxxxxxxxx>]

And this is affecting 4.15 AWS as well. Here is the output from
b-aws-4.15-2099

Test output:
+ echo blk
+ echo mmiotrace
+ echo function_graph
 (Test interrupted here)

dmesg:
[  216.814155] kernel BUG at /build/linux-aws-fips-hcslsq/linux-aws-fips-4.15.0/arch/x86/xen/spinlock.c:69!
[  216.816072] invalid opcode: 0000 [#1] SMP PTI
[  216.816991] Modules linked in: nls_iso8859_1 binfmt_misc sb_edac intel_rapl_perf serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops aesni_intel drm aes_x86_64 crypto_simd i2c_piix4 glue_helper cryptd i2c_core ixgbevf pata_acpi
[  216.827515] CPU: 1 PID: 13 Comm: cpuhp/1 Not tainted 4.15.0-2099-aws-fips #105-Ubuntu
[  216.829132] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
[  216.830471] RIP: 0010:dummy_handler+0x4/0x10
[  216.831362] RSP: 0000:ffffa0c827a43e38 EFLAGS: 00010046
[  216.832477] RAX: ffffffff8302a9c0 RBX: 000000000000003b RCX: 000000000000003b
[  216.833951] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
[  216.835425] RBP: ffffa0c827a43e38 R08: ffffa0c827000db0 R09: ffffa0c81b14ea00
[  216.836902] R10: 0000000000000040 R11: 0000000000000000 R12: 0000000000000000
[  216.838378] R13: 0000000000000000 R14: 0000000000000000 R15: ffffa0c82444f400
[  216.839854] FS:  0000000000000000(0000) GS:ffffa0c827a40000(0000) knlGS:0000000000000000
[  216.841533] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  216.842708] CR2: 0000000000000000 CR3: 000000003b80a001 CR4: 00000000001606e0
[  216.844141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  216.845631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  216.847095] Call Trace:
[  216.847640]  <IRQ>
[  216.848079]  __handle_irq_event_percpu+0x44/0x1a0
[  216.849070]  handle_irq_event_percpu+0x32/0x80
[  216.850009]  handle_percpu_irq+0x3d/0x60
[  216.850829]  generic_handle_irq+0x28/0x40
[  216.851687]  handle_irq_for_port+0x8f/0xe0
[  216.852556]  evtchn_2l_handle_events+0x157/0x270
[  216.853537]  __xen_evtchn_do_upcall+0x76/0xe0
[  216.854453]  xen_evtchn_do_upcall+0x2b/0x50
[  216.855327]  xen_hvm_callback_vector+0x90/0xa0
[  216.856266]  </IRQ>
[  216.856724] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
[  216.857890] RSP: 0000:ffffb95780d37d00 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
[  216.859472] RAX: 0000000000000001 RBX: ffffa0c82444f400 RCX: 000000000002cc00
[  216.860907] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
[  216.862353] RBP: ffffb95780d37d00 R08: ffffa0c827000db0 R09: ffffa0c81b14ea00
[  216.863782] R10: 0000000000000040 R11: 0000000000000246 R12: 000000000000003b
[  216.865233] R13: 0000000000000000 R14: ffffa0c81b14ea00 R15: ffffa0c81b14eaa4
[  216.866662]  __setup_irq+0x424/0x6e0
[  216.867392]  request_threaded_irq+0xf9/0x170
[  216.868281]  bind_ipi_to_irqhandler+0xc6/0x1f0
[  216.869184]  ? xen_qlock_wait+0x90/0x90
[  216.871428]  ? skx_uncore_pci_init+0x40/0x40
[  216.873772]  xen_init_lock_cpu+0x97/0xf0
[  216.876010]  ? skx_uncore_pci_init+0x40/0x40
[  216.878279]  xen_cpu_up_online+0xe/0x20
[  216.880408]  cpuhp_invoke_callback+0x9d/0x570
[  216.882651]  cpuhp_thread_fun+0xc4/0x150
[  216.884751]  smpboot_thread_fn+0xfc/0x170
[  216.886799]  kthread+0x121/0x140
[  216.888664]  ? sort_range+0x30/0x30
[  216.890525]  ? kthread_create_worker_on_cpu+0x70/0x70
[  216.892663]  ret_from_fork+0x35/0x40
[  216.894462] Code: c0 8b 75 e4 74 d6 44 89 e7 e8 29 3b 57 00 eb d6 44 89 e7 e8 1f 5f 57 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 79 e9 3f 01 00 55 48 
[  216.900389] RIP: dummy_handler+0x4/0x10 RSP: ffffa0c827a43e38
[  216.902668] ---[ end trace af76218ae4efc4a6 ]---
[  216.904746] Kernel panic - not syncing: Fatal exception in interrupt
[  216.907189] Kernel Offset: 0x2000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

** Also affects: linux-aws (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Tags added: 4.15 bionic

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2034057

Title:
  function_graph tracer in ftrace related tests triggers kernel bug
  (arch/x86/xen/spinlock.c:62) and kernel panic on AWS cloud c3.xlarge

Status in ubuntu-kernel-tests:
  New
Status in linux-aws package in Ubuntu:
  New
Status in linux-aws source package in Bionic:
  New
Status in linux-aws source package in Focal:
  New
Status in linux-aws source package in Jammy:
  New
Status in linux-aws source package in Lunar:
  New

Bug description:
  Test:
    * ftrace:test.d--00basic--basic2.tc from ubuntu_kselftests_ftrace
    * test_enable_all_tracers() and test_enable_all_tracers() from ubuntu_ftrace_smoke_test

  Will crash AWS instance c3.xlarge when testing the "function_graph"
  tracer.

  We have a similar issue filed against Azure (bug 1882669). Filing a
  new bug report because on AWS this is affecting 5.4 ~ 6.2 AWS kernel.
  However on Azure this is not affecting newer kernels.

  Take B-aws-5.4-1108 for example, with the ubuntu_kselftests_ftrace test the system will crash with the following error message and reboot (similar to bug 2029917):
  [  211.675624] kernel BUG at /build/linux-aws-5.4-VHOhmt/linux-aws-5.4-5.4.0/arch/x86/xen/spinlock.c:62!
  [  211.678258] invalid opcode: 0000 [#1] SMP PTI
  [  211.679596] CPU: 1 PID: 14 Comm: cpuhp/1 Not tainted 5.4.0-1108-aws #116~18.04.1-Ubuntu
  [  211.681825] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
  [  211.683728] RIP: 0010:dummy_handler+0x4/0x10
  [  211.685042] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
  [  211.690314] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
  [  211.691934] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
  [  211.694036] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
  [  211.696159] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
  [  211.698232] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
  [  211.700365] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
  [  211.702482] FS:  0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
  [  211.704894] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  211.706598] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
  [  211.708731] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  [  211.710835] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  [  211.713448] Call Trace:
  [  211.714796]  <IRQ>
  [  211.715996]  __handle_irq_event_percpu+0x44/0x1a0
  [  211.717907]  handle_irq_event_percpu+0x32/0x80
  [  211.719759]  handle_percpu_irq+0x3d/0x60
  [  211.721474]  generic_handle_irq+0x28/0x40
  [  211.723184]  handle_irq_for_port+0x8f/0xe0
  [  211.724988]  evtchn_2l_handle_events+0x157/0x270
  [  211.726913]  __xen_evtchn_do_upcall+0x76/0xe0
  [  211.728749]  xen_evtchn_do_upcall+0x2b/0x40
  [  211.730520]  xen_hvm_callback_vector+0xf/0x20
  [  211.732271]  </IRQ>
  [  211.733425] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
  [  211.735470] Code: e8 70 3d 64 ff 4c 29 e0 4c 39 f0 76 cf 80 0b 08 eb 8a 90 90 90 0f 1f 44 00 00 55 48 89 e5 e8 a6 ad 66 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6 07
  [  211.741920] RSP: 0000:ffffaecd400fbcf8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff0c
  [  211.744980] RAX: 0000000000000001 RBX: ffff8aefa34b8700 RCX: 000000000002cc00
  [  211.747420] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
  [  211.749882] RBP: ffffaecd400fbcf8 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
  [  211.752340] R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000003b
  [  211.754785] R13: 0000000000000000 R14: ffff8aef95bdfa00 R15: ffff8aef95bdfaa4
  [  211.757249]  __setup_irq+0x456/0x760
  [  211.758779]  ? kmem_cache_alloc_trace+0x170/0x230
  [  211.760639]  request_threaded_irq+0xfb/0x160
  [  211.762358]  bind_ipi_to_irqhandler+0xba/0x1c0
  [  211.764124]  ? xen_qlock_wait+0x90/0x90
  [  211.765734]  ? snr_uncore_mmio_init+0x20/0x20
  [  211.767496]  xen_init_lock_cpu+0x78/0xd0
  [  211.769135]  ? snr_uncore_mmio_init+0x20/0x20
  [  211.770864]  xen_cpu_up_online+0xe/0x20
  [  211.772500]  cpuhp_invoke_callback+0x8a/0x580
  [  211.774233]  cpuhp_thread_fun+0xb8/0x120
  [  211.775866]  smpboot_thread_fn+0xfc/0x170
  [  211.777524]  kthread+0x121/0x140
  [  211.778968]  ? sort_range+0x30/0x30
  [  211.780508]  ? kthread_park+0x90/0x90
  [  211.782073]  ret_from_fork+0x35/0x40
  [  211.783622] Modules linked in: nls_iso8859_1 binfmt_misc serio_raw sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper ixgbevf
  [  211.796497] ---[ end trace bb7d4e9bb7f852cb ]---
  [  211.798303] RIP: 0010:dummy_handler+0x4/0x10
  [  211.800022] Code: 8b 75 e4 74 d6 44 89 e7 e8 f9 88 61 00 eb d6 44 89 e7 e8 6f ab 61 00 eb cc 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 d0 9f 01 00 75 02 f3
  [  211.806379] RSP: 0000:ffffaecd4000ee38 EFLAGS: 00010046
  [  211.808356] RAX: ffffffffb462e3e0 RBX: 000000000000003b RCX: 0000000000000000
  [  211.810792] RDX: 0000000000400e00 RSI: 0000000000000000 RDI: 000000000000003b
  [  211.813263] RBP: ffffaecd4000ee38 R08: ffff8aefa6c036c0 R09: ffff8aefa6c038c0
  [  211.815713] R10: 0000000000000000 R11: ffffffffb6064da8 R12: 0000000000000000
  [  211.818162] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8aefa34b8700
  [  211.820637] FS:  0000000000000000(0000) GS:ffff8aefa8040000(0000) knlGS:0000000000000000
  [  211.823821] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [  211.825904] CR2: 0000000000000000 CR3: 00000001b400a001 CR4: 00000000001606e0
  [  211.828357] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  [  211.830829] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  [  211.833284] Kernel panic - not syncing: Fatal exception in interrupt
  [  211.835575] Kernel Offset: 0x33600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

  Test output:
   + echo 1
   + . /home/ubuntu/autotest/client/tmp/ubuntu_kselftests_ftrace/src/linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc
   + test -f available_tracers
   + cat available_tracers
   + echo hwlat
   + echo blk
   + echo mmiotrace
   + echo function_graph
  (Test interrupted here)

  To verify this, you can hack linux/tools/testing/selftests/ftrace/test.d/00basic/basic2.tc and add "read -p" to check what tracer we're testing:
    #!/bin/sh
    # SPDX-License-Identifier: GPL-2.0
    # description: Basic test for tracers
    # flags: instance
    test -f available_tracers
    for t in `cat available_tracers`; do
      read -p "testing $t" foo
      echo $t > current_tracer
    done
    echo nop > current_tracer

  And run this manually with sudo ./ftracetest -vvv
  test.d/00basic/basic2.tc, you will see the system crashes right away
  with "function_graph".

  For ubuntu_ftrace_smoke_test, the "function_graph" tracer will be
  tested with test_function_graph_tracer(). And since "function_graph"
  tracer is in /sys/kernel/debug/tracing/available_tracers, it will be
  tested with  test_enable_all_tracers() as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2034057/+subscriptions