canonical-ubuntu-qa team mailing list archive

Thread
Date
[Bug 2109317] Re: ubuntu_zfs_stress locks up and kdumps on small ppc64el jammy vms

To: canonical-ubuntu-qa@xxxxxxxxxxxxxxxxxxx
From: Mehmet Basaran <2109317@xxxxxxxxxxxxxxxxxx>
Date: Fri, 25 Apr 2025 06:45:27 -0000
Reply-to: Bug 2109317 <2109317@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
** Attachment added: "full-console-log.txt"
   https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2109317/+attachment/5874126/+files/full-console-log.txt

** Description changed:

  When run on small powerpc64 openstack vm (builder-ppc64el-
  cpu2-ram4-disk20) ubuntu_zfs_stress test locks op workqueues which
  prints kernel dump. This happens with jammy:linux 5.15.0-140.150
  (2025.04.14) and 5.15.0-138.148 (2025.03.17). Doesn't seem to occur when
  I increase the RAM to 8GBs (builder-ppc64el-cpu2-ram8-disk20).
  
- Failure is also present on 5.15.0-135 howeever it is not as frequent as
- 5.15.0-138 and 5.15.0-140.
+ Failure is also present on 5.15.0-135 (kernel version in current jammy
+ image) however it is not as frequent as 5.15.0-138 and 5.15.0-140.
  
  zfs version: zfs-zed (2.1.5-1ubuntu6~22.04.5) zfs-utils
  2.1.5-1ubuntu6~22.04.5
  
  19:28:40 DEBUG| [stdout] --------------------------------------------------------------------------------
  19:28:40 DEBUG| [stdout] ZFS options:   recordsize=4096
  19:28:40 DEBUG| [stdout] Stress test:   /home/ubuntu/autotest/client/tmp/ubuntu_zfs_stress/src/stress-ng/stress-ng --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:28:40 DEBUG| [stdout] VDEV path:
  19:28:40 DEBUG| [stdout] Mount point:   /testpool/test
  19:28:40 DEBUG| [stdout] Date:          Thu Apr 24 19:28:40 UTC 2025
  19:28:40 DEBUG| [stdout] Host:          basaran-ppc64el
  19:28:40 DEBUG| [stdout] Kernel:        5.15.0-138-generic #148-Ubuntu SMP Fri Mar 14 19:05:06 UTC 2025
  19:28:40 DEBUG| [stdout] Machine:       basaran-ppc64el ppc64le ppc64le
  19:28:40 DEBUG| [stdout] CPUs online:   2
  19:28:40 DEBUG| [stdout] CPUs total:    2
  19:28:40 DEBUG| [stdout] Page size:     65536
  19:28:40 DEBUG| [stdout] Pages avail:   49278
  19:28:40 DEBUG| [stdout] Pages total:   64466
  19:28:40 DEBUG| [stdout] --------------------------------------------------------------------------------
  19:28:40 DEBUG| [stdout]
  19:28:40 DEBUG| [stdout] zfs set recordsize=4096 testpool/test
  19:28:40 DEBUG| [stdout]
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] aio stressor will be skipped, it is not implemented on this system: ppc64le Linux 5.15.0-138-generic gcc 11.4.0 (built without aio.h)
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] setting to a 5 secs run per stressor
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] dispatching hogs: 4 hdd, 4 link, 4 symlink, 4 lockf, 4 seek, 4 dentry, 4 dir, 4 fallocate, 4 fstat, 1 io, 4 lease, 2 mmap, 4 open, 4 rename, 4 chdir, 4 chmod, 4 filename, 4 rename
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] note: /proc/sys/kernel/sched_autogroup_enabled is 1 and this can impact scheduling throughput for processes not attached to a tty. Setting this to 0 may improve performance metrics
  19:28:40 DEBUG| [stdout] stress-ng: info:  [179044] open: using a maximum of 1048576 file descriptors
  19:28:40 DEBUG| [stdout] stress-ng: info:  [179037] io: this is a legacy I/O sync stressor, consider using iomix instead
  19:28:46 DEBUG| [stdout] stress-ng: info:  [179052] chdir: removing 8192 directories
  19:30:45 DEBUG| [stdout] stress-ng: warn:  [179042] cannot terminate process 179070, gave up after 120 seconds
  19:30:45 DEBUG| [stdout] stress-ng: warn:  [179043] cannot terminate process 179069, gave up after 120 seconds
  19:31:59 DEBUG| [stdout] Found kernel warning and/or call trace:
  19:31:59 DEBUG| [stdout]
  19:31:59 DEBUG| [stdout] [  768.603970] TESTING: --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:31:59 DEBUG| [stdout] [  769.222093] ubuntu_zfs_stre (14186): drop_caches: 1
  19:31:59 DEBUG| [stdout] [  769.222974] ubuntu_zfs_stre (14186): drop_caches: 2
  19:31:59 DEBUG| [stdout] [  769.224857] ubuntu_zfs_stre (14186): drop_caches: 3
  19:31:59 DEBUG| [stdout] [  769.251803] ZFS options:   recordsize=4096
  19:31:59 DEBUG| [stdout] [  769.251854] Stress test:   /home/ubuntu/autotest/client/tmp/ubuntu_zfs_stress/src/stress-ng/stress-ng --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:31:59 DEBUG| [stdout] [  769.251872] Mount point:   /testpool/test
  19:31:59 DEBUG| [stdout] [  967.673892] INFO: task kworker/u4:3:20388 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.674222]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.674253] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.674285] task:kworker/u4:3    state:D stack:    0 pid:20388 ppid:     2 flags:0x00000800
  19:31:59 DEBUG| [stdout] [  967.674291] Workqueue: writeback wb_workfn (flush-zfs-62)
  19:31:59 DEBUG| [stdout] [  967.674319] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.674320] [c000000056066fe0] [c00000006348cd80] 0xc00000006348cd80 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.674333] [c0000000560671d0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.674337] [c000000056067230] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.674341] [c0000000560672f0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674344] [c000000056067360] [c008000002720724] cv_wait_common+0x17c/0x200 [spl]
  19:31:59 DEBUG| [stdout] [  967.674354] [c000000056067400] [c00800000446bc1c] dmu_tx_wait+0x184/0x2d0 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674426] [c000000056067460] [c0080000045d30e8] zfs_putpage+0x5c0/0x5f0 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674517] [c0000000560675e0] [c0080000045e2a7c] zpl_putpage+0x54/0x90 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674607] [c000000056067620] [c000000000472490] write_cache_pages+0x220/0x530
  19:31:59 DEBUG| [stdout] [  967.674612] [c000000056067750] [c0080000045e289c] zpl_writepages+0xc4/0x250 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674700] [c0000000560677b0] [c0000000004752e0] do_writepages+0xc0/0x2e0
  19:31:59 DEBUG| [stdout] [  967.674704] [c000000056067870] [c000000000619f68] __writeback_single_inode+0x68/0x4c0
  19:31:59 DEBUG| [stdout] [  967.674708] [c0000000560678c0] [c00000000061aac4] writeback_sb_inodes+0x2f4/0x730
  19:31:59 DEBUG| [stdout] [  967.674711] [c0000000560679f0] [c00000000061af84] __writeback_inodes_wb+0x84/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674715] [c000000056067a60] [c00000000061b40c] wb_writeback+0x36c/0x490
  19:31:59 DEBUG| [stdout] [  967.674718] [c000000056067b20] [c00000000061ce0c] wb_workfn+0x51c/0x700
  19:31:59 DEBUG| [stdout] [  967.674722] [c000000056067c70] [c0000000001aa634] process_one_work+0x2f4/0x5e0
  19:31:59 DEBUG| [stdout] [  967.674725] [c000000056067d10] [c0000000001aaf08] worker_thread+0xa8/0x610
  19:31:59 DEBUG| [stdout] [  967.674728] [c000000056067da0] [c0000000001b7a04] kthread+0x184/0x190
  19:31:59 DEBUG| [stdout] [  967.674732] [c000000056067e10] [c00000000000cf64] ret_from_kernel_thread+0x5c/0x64
  19:31:59 DEBUG| [stdout] [  967.674738] INFO: task txg_quiesce:178678 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.674773]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.674804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.674835] task:txg_quiesce     state:D stack:    0 pid:178678 ppid:     2 flags:0x00000800
  19:31:59 DEBUG| [stdout] [  967.674838] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.674839] [c000000043c1b850] [c0000000ffea2100] 0xc0000000ffea2100 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.674843] [c000000043c1ba40] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.674847] [c000000043c1baa0] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.674850] [c000000043c1bb60] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674853] [c000000043c1bbd0] [c008000002720724] cv_wait_common+0x17c/0x200 [spl]
  19:31:59 DEBUG| [stdout] [  967.674861] [c000000043c1bc70] [c00800000450d1a0] txg_quiesce+0x1e8/0x300 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674947] [c000000043c1bce0] [c00800000450d398] txg_quiesce_thread+0xe0/0x180 [zfs]
  19:31:59 DEBUG| [stdout] [  967.675033] [c000000043c1bd60] [c00800000272e28c] thread_generic_wrapper+0x94/0xb8 [spl]
  19:31:59 DEBUG| [stdout] [  967.675042] [c000000043c1bda0] [c0000000001b7a04] kthread+0x184/0x190
  19:31:59 DEBUG| [stdout] [  967.675046] [c000000043c1be10] [c00000000000cf64] ret_from_kernel_thread+0x5c/0x64
  19:31:59 DEBUG| [stdout] [  967.675051] INFO: task stress-ng:179021 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675080]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675142] task:stress-ng       state:D stack:    0 pid:179021 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675146] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675146] [c000000011cbb850] [c0000000ffea2100] 0xc0000000ffea2100 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675150] [c000000011cbba40] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675153] [c000000011cbbaa0] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675156] [c000000011cbbb60] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675159] [c000000011cbbbd0] [c00000000061b5d4] wb_wait_for_completion+0xa4/0xf0
  19:31:59 DEBUG| [stdout] [  967.675163] [c000000011cbbc30] [c00000000061bd88] sync_inodes_sb+0xd8/0x140
  19:31:59 DEBUG| [stdout] [  967.675166] [c000000011cbbcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675170] [c000000011cbbce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675173] [c000000011cbbd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675177] [c000000011cbbd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675181] [c000000011cbbdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675184] [c000000011cbbe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675188] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675190] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675192] REGS: c000000011cbbe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675194] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675204] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d84328 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675238] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675240] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675241] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675243] INFO: task stress-ng:179023 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675272]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675334] task:stress-ng       state:D stack:    0 pid:179023 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675337] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675338] [c000000011c977c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675342] [c000000011c979b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675345] [c000000011c97a10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675348] [c000000011c97ad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675351] [c000000011c97b40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675355] [c000000011c97c30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675359] [c000000011c97cc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675362] [c000000011c97ce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675366] [c000000011c97d50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675369] [c000000011c97d90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675373] [c000000011c97db0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675376] [c000000011c97e10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675380] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675382] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675383] REGS: c000000011c97e80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675385] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675394] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d86fe8 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675427] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675429] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675430] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675431] INFO: task stress-ng:179024 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675460]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675491] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675522] task:stress-ng       state:D stack:    0 pid:179024 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675525] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675526] [c000000011cbf7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675529] [c000000011cbf9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675533] [c000000011cbfa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675536] [c000000011cbfad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675539] [c000000011cbfb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675542] [c000000011cbfc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675546] [c000000011cbfcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675550] [c000000011cbfce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675553] [c000000011cbfd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675557] [c000000011cbfd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675560] [c000000011cbfdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675563] [c000000011cbfe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675567] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675569] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675571] REGS: c000000011cbfe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675572] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675581] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d88648 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675614] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675615] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675617] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675619] INFO: task stress-ng:179037 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675647]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675709] task:stress-ng       state:D stack:    0 pid:179037 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675712] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675714] [c000000076d9f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675717] [c000000076d9f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675720] [c000000076d9fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675723] [c000000076d9fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675726] [c000000076d9fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675730] [c000000076d9fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675733] [c000000076d9fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675737] [c000000076d9fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675740] [c000000076d9fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675744] [c000000076d9fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675747] [c000000076d9fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675750] [c000000076d9fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675754] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675756] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675757] REGS: c000000076d9fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675759] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 48044444  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675768] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041cb70 0000733933066e00 ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000010 0000000000004800 0000000000000000 00000a493b6d12e0
  19:31:59 DEBUG| [stdout]                GPR08: 00000a493b6d0010 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d9a928 0000000000000024
  19:31:59 DEBUG| [stdout]                GPR16: 0000733932d99348 00000000000186a0 00007fffe041db96 00007fffe041db95
  19:31:59 DEBUG| [stdout]                GPR20: 00007fffe041db94 00007fffe041cc08 000000000000002e 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000001 000000000000002e 00007fffe041d008 000000000000001d
  19:31:59 DEBUG| [stdout]                GPR28: 0000000000100001 0000733932d9a928 00007fffe041cc78 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675801] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675802] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675804] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675805] INFO: task stress-ng:179044 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675834]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675896] task:stress-ng       state:D stack:    0 pid:179044 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675899] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675900] [c000000043c1f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675903] [c000000043c1f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675907] [c000000043c1fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675910] [c000000043c1fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675913] [c000000043c1fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675916] [c000000043c1fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675920] [c000000043c1fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675924] [c000000043c1fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675927] [c000000043c1fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675930] [c000000043c1fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675934] [c000000043c1fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675937] [c000000043c1fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675941] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675942] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675944] REGS: c000000043c1fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675945] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675954] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da45c8 000000000000002b
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 00000000000005a2 0000000000000616
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb64
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1688 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675987] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675989] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675990] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675991] INFO: task stress-ng:179045 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676020]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676082] task:stress-ng       state:D stack:    0 pid:179045 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676085] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676086] [c000000043c0f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676089] [c000000043c0f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676093] [c000000043c0fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676095] [c000000043c0fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676098] [c000000043c0fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676102] [c000000043c0fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676105] [c000000043c0fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676109] [c000000043c0fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676112] [c000000043c0fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676116] [c000000043c0fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676119] [c000000043c0fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676122] [c000000043c0fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676126] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676128] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676129] REGS: c000000043c0fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676131] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676140] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da5c28 000000000000002c
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000665 00000000000006ed
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb65
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1994 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676172] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676174] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676175] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.676176] INFO: task stress-ng:179046 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676204]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676235] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676266] task:stress-ng       state:D stack:    0 pid:179046 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676269] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676270] [c000000043c2b7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676273] [c000000043c2b9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676277] [c000000043c2ba10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676280] [c000000043c2bad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676283] [c000000043c2bb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676287] [c000000043c2bc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676290] [c000000043c2bcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676294] [c000000043c2bce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676297] [c000000043c2bd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676301] [c000000043c2bd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676304] [c000000043c2bdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676307] [c000000043c2be10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676311] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676313] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676314] REGS: c000000043c2be80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676316] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676324] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da7288 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000655 00000000000006d3
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb66
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1954 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676357] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676359] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676361] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.676362] INFO: task stress-ng:179047 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676390]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676452] task:stress-ng       state:D stack:    0 pid:179047 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676455] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676456] [c000000051e037c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676459] [c000000051e039b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676463] [c000000051e03a10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676466] [c000000051e03ad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676469] [c000000051e03b40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676472] [c000000051e03c30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676476] [c000000051e03cc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676479] [c000000051e03ce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676482] [c000000051e03d50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676486] [c000000051e03d90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676490] [c000000051e03db0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676493] [c000000051e03e10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676496] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676498] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676499] REGS: c000000051e03e80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676501] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676510] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da88e8 000000000000002e
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000443 000000000000049a
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb67
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b110c 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676543] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676545] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676546] --- interrupt: 3000

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2109317

Title:
  ubuntu_zfs_stress locks up and kdumps on small ppc64el jammy vms

Status in ubuntu-kernel-tests:
  New

Bug description:
  When run on small powerpc64 openstack vm (builder-ppc64el-
  cpu2-ram4-disk20) ubuntu_zfs_stress test locks op workqueues which
  prints kernel dump. This happens with jammy:linux 5.15.0-140.150
  (2025.04.14) and 5.15.0-138.148 (2025.03.17). Doesn't seem to occur
  when I increase the RAM to 8GBs (builder-ppc64el-cpu2-ram8-disk20).

  Failure is also present on 5.15.0-135 (kernel version in current jammy
  image) however it is not as frequent as 5.15.0-138 and 5.15.0-140.

  zfs version: zfs-zed (2.1.5-1ubuntu6~22.04.5) zfs-utils
  2.1.5-1ubuntu6~22.04.5

  19:28:40 DEBUG| [stdout] --------------------------------------------------------------------------------
  19:28:40 DEBUG| [stdout] ZFS options:   recordsize=4096
  19:28:40 DEBUG| [stdout] Stress test:   /home/ubuntu/autotest/client/tmp/ubuntu_zfs_stress/src/stress-ng/stress-ng --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:28:40 DEBUG| [stdout] VDEV path:
  19:28:40 DEBUG| [stdout] Mount point:   /testpool/test
  19:28:40 DEBUG| [stdout] Date:          Thu Apr 24 19:28:40 UTC 2025
  19:28:40 DEBUG| [stdout] Host:          basaran-ppc64el
  19:28:40 DEBUG| [stdout] Kernel:        5.15.0-138-generic #148-Ubuntu SMP Fri Mar 14 19:05:06 UTC 2025
  19:28:40 DEBUG| [stdout] Machine:       basaran-ppc64el ppc64le ppc64le
  19:28:40 DEBUG| [stdout] CPUs online:   2
  19:28:40 DEBUG| [stdout] CPUs total:    2
  19:28:40 DEBUG| [stdout] Page size:     65536
  19:28:40 DEBUG| [stdout] Pages avail:   49278
  19:28:40 DEBUG| [stdout] Pages total:   64466
  19:28:40 DEBUG| [stdout] --------------------------------------------------------------------------------
  19:28:40 DEBUG| [stdout]
  19:28:40 DEBUG| [stdout] zfs set recordsize=4096 testpool/test
  19:28:40 DEBUG| [stdout]
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] aio stressor will be skipped, it is not implemented on this system: ppc64le Linux 5.15.0-138-generic gcc 11.4.0 (built without aio.h)
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] setting to a 5 secs run per stressor
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] dispatching hogs: 4 hdd, 4 link, 4 symlink, 4 lockf, 4 seek, 4 dentry, 4 dir, 4 fallocate, 4 fstat, 1 io, 4 lease, 2 mmap, 4 open, 4 rename, 4 chdir, 4 chmod, 4 filename, 4 rename
  19:28:40 DEBUG| [stdout] stress-ng: info:  [178991] note: /proc/sys/kernel/sched_autogroup_enabled is 1 and this can impact scheduling throughput for processes not attached to a tty. Setting this to 0 may improve performance metrics
  19:28:40 DEBUG| [stdout] stress-ng: info:  [179044] open: using a maximum of 1048576 file descriptors
  19:28:40 DEBUG| [stdout] stress-ng: info:  [179037] io: this is a legacy I/O sync stressor, consider using iomix instead
  19:28:46 DEBUG| [stdout] stress-ng: info:  [179052] chdir: removing 8192 directories
  19:30:45 DEBUG| [stdout] stress-ng: warn:  [179042] cannot terminate process 179070, gave up after 120 seconds
  19:30:45 DEBUG| [stdout] stress-ng: warn:  [179043] cannot terminate process 179069, gave up after 120 seconds
  19:31:59 DEBUG| [stdout] Found kernel warning and/or call trace:
  19:31:59 DEBUG| [stdout]
  19:31:59 DEBUG| [stdout] [  768.603970] TESTING: --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:31:59 DEBUG| [stdout] [  769.222093] ubuntu_zfs_stre (14186): drop_caches: 1
  19:31:59 DEBUG| [stdout] [  769.222974] ubuntu_zfs_stre (14186): drop_caches: 2
  19:31:59 DEBUG| [stdout] [  769.224857] ubuntu_zfs_stre (14186): drop_caches: 3
  19:31:59 DEBUG| [stdout] [  769.251803] ZFS options:   recordsize=4096
  19:31:59 DEBUG| [stdout] [  769.251854] Stress test:   /home/ubuntu/autotest/client/tmp/ubuntu_zfs_stress/src/stress-ng/stress-ng --verify --times --metrics-brief --syslog --keep-name -t 5s --hdd 4 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --link 4 --symlink 4 --lockf 4 --seek 4 --aio 4 --aio-requests 32 --dentry 4 --dir 4 --dentry-order stride --fallocate 4 --fstat 4 --dentries 65536 --io 1 --lease 4 --mmap 0 --mmap-file --mmap-async --open 4 --rename 4 --hdd-bytes 128M --fallocate-bytes 128M --chdir 4 --chmod 4 --filename 4 --rename 4 --mmap-bytes 128M --hdd-write-size 512 --ionice-class besteffort --ionice-level 0
  19:31:59 DEBUG| [stdout] [  769.251872] Mount point:   /testpool/test
  19:31:59 DEBUG| [stdout] [  967.673892] INFO: task kworker/u4:3:20388 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.674222]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.674253] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.674285] task:kworker/u4:3    state:D stack:    0 pid:20388 ppid:     2 flags:0x00000800
  19:31:59 DEBUG| [stdout] [  967.674291] Workqueue: writeback wb_workfn (flush-zfs-62)
  19:31:59 DEBUG| [stdout] [  967.674319] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.674320] [c000000056066fe0] [c00000006348cd80] 0xc00000006348cd80 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.674333] [c0000000560671d0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.674337] [c000000056067230] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.674341] [c0000000560672f0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674344] [c000000056067360] [c008000002720724] cv_wait_common+0x17c/0x200 [spl]
  19:31:59 DEBUG| [stdout] [  967.674354] [c000000056067400] [c00800000446bc1c] dmu_tx_wait+0x184/0x2d0 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674426] [c000000056067460] [c0080000045d30e8] zfs_putpage+0x5c0/0x5f0 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674517] [c0000000560675e0] [c0080000045e2a7c] zpl_putpage+0x54/0x90 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674607] [c000000056067620] [c000000000472490] write_cache_pages+0x220/0x530
  19:31:59 DEBUG| [stdout] [  967.674612] [c000000056067750] [c0080000045e289c] zpl_writepages+0xc4/0x250 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674700] [c0000000560677b0] [c0000000004752e0] do_writepages+0xc0/0x2e0
  19:31:59 DEBUG| [stdout] [  967.674704] [c000000056067870] [c000000000619f68] __writeback_single_inode+0x68/0x4c0
  19:31:59 DEBUG| [stdout] [  967.674708] [c0000000560678c0] [c00000000061aac4] writeback_sb_inodes+0x2f4/0x730
  19:31:59 DEBUG| [stdout] [  967.674711] [c0000000560679f0] [c00000000061af84] __writeback_inodes_wb+0x84/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674715] [c000000056067a60] [c00000000061b40c] wb_writeback+0x36c/0x490
  19:31:59 DEBUG| [stdout] [  967.674718] [c000000056067b20] [c00000000061ce0c] wb_workfn+0x51c/0x700
  19:31:59 DEBUG| [stdout] [  967.674722] [c000000056067c70] [c0000000001aa634] process_one_work+0x2f4/0x5e0
  19:31:59 DEBUG| [stdout] [  967.674725] [c000000056067d10] [c0000000001aaf08] worker_thread+0xa8/0x610
  19:31:59 DEBUG| [stdout] [  967.674728] [c000000056067da0] [c0000000001b7a04] kthread+0x184/0x190
  19:31:59 DEBUG| [stdout] [  967.674732] [c000000056067e10] [c00000000000cf64] ret_from_kernel_thread+0x5c/0x64
  19:31:59 DEBUG| [stdout] [  967.674738] INFO: task txg_quiesce:178678 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.674773]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.674804] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.674835] task:txg_quiesce     state:D stack:    0 pid:178678 ppid:     2 flags:0x00000800
  19:31:59 DEBUG| [stdout] [  967.674838] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.674839] [c000000043c1b850] [c0000000ffea2100] 0xc0000000ffea2100 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.674843] [c000000043c1ba40] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.674847] [c000000043c1baa0] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.674850] [c000000043c1bb60] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.674853] [c000000043c1bbd0] [c008000002720724] cv_wait_common+0x17c/0x200 [spl]
  19:31:59 DEBUG| [stdout] [  967.674861] [c000000043c1bc70] [c00800000450d1a0] txg_quiesce+0x1e8/0x300 [zfs]
  19:31:59 DEBUG| [stdout] [  967.674947] [c000000043c1bce0] [c00800000450d398] txg_quiesce_thread+0xe0/0x180 [zfs]
  19:31:59 DEBUG| [stdout] [  967.675033] [c000000043c1bd60] [c00800000272e28c] thread_generic_wrapper+0x94/0xb8 [spl]
  19:31:59 DEBUG| [stdout] [  967.675042] [c000000043c1bda0] [c0000000001b7a04] kthread+0x184/0x190
  19:31:59 DEBUG| [stdout] [  967.675046] [c000000043c1be10] [c00000000000cf64] ret_from_kernel_thread+0x5c/0x64
  19:31:59 DEBUG| [stdout] [  967.675051] INFO: task stress-ng:179021 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675080]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675112] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675142] task:stress-ng       state:D stack:    0 pid:179021 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675146] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675146] [c000000011cbb850] [c0000000ffea2100] 0xc0000000ffea2100 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675150] [c000000011cbba40] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675153] [c000000011cbbaa0] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675156] [c000000011cbbb60] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675159] [c000000011cbbbd0] [c00000000061b5d4] wb_wait_for_completion+0xa4/0xf0
  19:31:59 DEBUG| [stdout] [  967.675163] [c000000011cbbc30] [c00000000061bd88] sync_inodes_sb+0xd8/0x140
  19:31:59 DEBUG| [stdout] [  967.675166] [c000000011cbbcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675170] [c000000011cbbce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675173] [c000000011cbbd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675177] [c000000011cbbd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675181] [c000000011cbbdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675184] [c000000011cbbe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675188] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675190] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675192] REGS: c000000011cbbe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675194] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675204] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d84328 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675238] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675240] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675241] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675243] INFO: task stress-ng:179023 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675272]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675334] task:stress-ng       state:D stack:    0 pid:179023 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675337] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675338] [c000000011c977c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675342] [c000000011c979b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675345] [c000000011c97a10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675348] [c000000011c97ad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675351] [c000000011c97b40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675355] [c000000011c97c30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675359] [c000000011c97cc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675362] [c000000011c97ce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675366] [c000000011c97d50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675369] [c000000011c97d90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675373] [c000000011c97db0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675376] [c000000011c97e10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675380] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675382] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675383] REGS: c000000011c97e80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675385] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675394] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d86fe8 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675427] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675429] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675430] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675431] INFO: task stress-ng:179024 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675460]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675491] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675522] task:stress-ng       state:D stack:    0 pid:179024 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675525] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675526] [c000000011cbf7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675529] [c000000011cbf9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675533] [c000000011cbfa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675536] [c000000011cbfad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675539] [c000000011cbfb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675542] [c000000011cbfc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675546] [c000000011cbfcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675550] [c000000011cbfce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675553] [c000000011cbfd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675557] [c000000011cbfd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675560] [c000000011cbfdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675563] [c000000011cbfe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675567] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675569] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675571] REGS: c000000011cbfe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675572] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44004442  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675581] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041b6d0 0000733933066e00 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000003 00007fffe041b1b8 0000000000000000 00007fffe041b108
  19:31:59 DEBUG| [stdout]                GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d88648 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR16: 00000a491df46600 0000000000000001 00007fffe041b788 00000a491df55438
  19:31:59 DEBUG| [stdout]                GPR20: 00000a491df465e0 0000000000000001 00007fffe041b780 0000000000000800
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000000 0000000000000001 0000000000000000 00000a491e036af8
  19:31:59 DEBUG| [stdout]                GPR28: 00007fffe041c788 0000000000000800 0000000000000000 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675614] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675615] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675617] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675619] INFO: task stress-ng:179037 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675647]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675709] task:stress-ng       state:D stack:    0 pid:179037 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675712] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675714] [c000000076d9f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675717] [c000000076d9f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675720] [c000000076d9fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675723] [c000000076d9fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675726] [c000000076d9fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675730] [c000000076d9fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675733] [c000000076d9fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675737] [c000000076d9fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675740] [c000000076d9fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675744] [c000000076d9fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675747] [c000000076d9fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675750] [c000000076d9fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675754] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675756] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675757] REGS: c000000076d9fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675759] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 48044444  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675768] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041cb70 0000733933066e00 ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR04: 0000000000000010 0000000000004800 0000000000000000 00000a493b6d12e0
  19:31:59 DEBUG| [stdout]                GPR08: 00000a493b6d0010 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932d9a928 0000000000000024
  19:31:59 DEBUG| [stdout]                GPR16: 0000733932d99348 00000000000186a0 00007fffe041db96 00007fffe041db95
  19:31:59 DEBUG| [stdout]                GPR20: 00007fffe041db94 00007fffe041cc08 000000000000002e 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR24: 0000000000000001 000000000000002e 00007fffe041d008 000000000000001d
  19:31:59 DEBUG| [stdout]                GPR28: 0000000000100001 0000733932d9a928 00007fffe041cc78 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675801] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675802] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675804] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675805] INFO: task stress-ng:179044 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.675834]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.675865] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.675896] task:stress-ng       state:D stack:    0 pid:179044 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.675899] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.675900] [c000000043c1f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.675903] [c000000043c1f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.675907] [c000000043c1fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.675910] [c000000043c1fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.675913] [c000000043c1fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.675916] [c000000043c1fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.675920] [c000000043c1fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.675924] [c000000043c1fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.675927] [c000000043c1fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.675930] [c000000043c1fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.675934] [c000000043c1fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.675937] [c000000043c1fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.675941] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675942] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.675944] REGS: c000000043c1fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.675945] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.675954] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da45c8 000000000000002b
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 00000000000005a2 0000000000000616
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb64
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1688 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.675987] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675989] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.675990] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.675991] INFO: task stress-ng:179045 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676020]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676082] task:stress-ng       state:D stack:    0 pid:179045 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676085] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676086] [c000000043c0f7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676089] [c000000043c0f9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676093] [c000000043c0fa10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676095] [c000000043c0fad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676098] [c000000043c0fb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676102] [c000000043c0fc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676105] [c000000043c0fcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676109] [c000000043c0fce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676112] [c000000043c0fd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676116] [c000000043c0fd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676119] [c000000043c0fdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676122] [c000000043c0fe10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676126] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676128] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676129] REGS: c000000043c0fe80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676131] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676140] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da5c28 000000000000002c
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000665 00000000000006ed
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb65
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1994 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676172] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676174] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676175] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.676176] INFO: task stress-ng:179046 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676204]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676235] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676266] task:stress-ng       state:D stack:    0 pid:179046 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676269] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676270] [c000000043c2b7c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676273] [c000000043c2b9b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676277] [c000000043c2ba10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676280] [c000000043c2bad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676283] [c000000043c2bb40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676287] [c000000043c2bc30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676290] [c000000043c2bcc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676294] [c000000043c2bce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676297] [c000000043c2bd50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676301] [c000000043c2bd90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676304] [c000000043c2bdb0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676307] [c000000043c2be10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676311] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676313] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676314] REGS: c000000043c2be80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676316] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676324] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da7288 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000655 00000000000006d3
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb66
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b1954 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676357] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676359] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676361] --- interrupt: 3000
  19:31:59 DEBUG| [stdout] [  967.676362] INFO: task stress-ng:179047 blocked for more than 120 seconds.
  19:31:59 DEBUG| [stdout] [  967.676390]       Tainted: P           O      5.15.0-138-generic #148-Ubuntu
  19:31:59 DEBUG| [stdout] [  967.676421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  19:31:59 DEBUG| [stdout] [  967.676452] task:stress-ng       state:D stack:    0 pid:179047 ppid:178991 flags:0x00042002
  19:31:59 DEBUG| [stdout] [  967.676455] Call Trace:
  19:31:59 DEBUG| [stdout] [  967.676456] [c000000051e037c0] [c000000017500000] 0xc000000017500000 (unreliable)
  19:31:59 DEBUG| [stdout] [  967.676459] [c000000051e039b0] [c00000000001dcf4] __switch_to+0x134/0x230
  19:31:59 DEBUG| [stdout] [  967.676463] [c000000051e03a10] [c0000000012da8f8] __schedule+0x228/0x700
  19:31:59 DEBUG| [stdout] [  967.676466] [c000000051e03ad0] [c0000000012dae5c] schedule+0x8c/0x1a0
  19:31:59 DEBUG| [stdout] [  967.676469] [c000000051e03b40] [c000000000223f00] rwsem_down_write_slowpath+0x360/0x6e0
  19:31:59 DEBUG| [stdout] [  967.676472] [c000000051e03c30] [c00000000061bd6c] sync_inodes_sb+0xbc/0x140
  19:31:59 DEBUG| [stdout] [  967.676476] [c000000051e03cc0] [c000000000621fec] sync_inodes_one_sb+0x2c/0x40
  19:31:59 DEBUG| [stdout] [  967.676479] [c000000051e03ce0] [c0000000005c21e4] iterate_supers+0x144/0x240
  19:31:59 DEBUG| [stdout] [  967.676482] [c000000051e03d50] [c000000000622810] ksys_sync+0x60/0x100
  19:31:59 DEBUG| [stdout] [  967.676486] [c000000051e03d90] [c0000000006228d0] sys_sync+0x20/0x40
  19:31:59 DEBUG| [stdout] [  967.676490] [c000000051e03db0] [c00000000002f954] system_call_exception+0x114/0x2c0
  19:31:59 DEBUG| [stdout] [  967.676493] [c000000051e03e10] [c00000000000c1e8] system_call_vectored_common+0xe8/0x278
  19:31:59 DEBUG| [stdout] [  967.676496] --- interrupt: 3000 at 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676498] NIP:  0000733932f41044 LR: 0000733932f41044 CTR: 0000000000000000
  19:31:59 DEBUG| [stdout] [  967.676499] REGS: c000000051e03e80 TRAP: 3000   Tainted: P           O       (5.15.0-138-generic)
  19:31:59 DEBUG| [stdout] [  967.676501] MSR:  800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE>  CR: 44008448  XER: 00000000
  19:31:59 DEBUG| [stdout] [  967.676510] IRQMASK: 0
  19:31:59 DEBUG| [stdout]                GPR00: 0000000000000024 00007fffe041a690 0000733933066e00 000000000000002d
  19:31:59 DEBUG| [stdout]                GPR04: 00007fffe04196a8 0000000000001000 0000000000000000 000000000000001a
  19:31:59 DEBUG| [stdout]                GPR08: 00000000000001a8 0000000000000000 0000000000000000 0000000000000000
  19:31:59 DEBUG| [stdout]                GPR12: 0000000000000000 000073393331bd60 0000733932da88e8 000000000000002e
  19:31:59 DEBUG| [stdout]                GPR16: 0000000000000000 0000000000000001 ffffffffffffffff ffffffffffffffff
  19:31:59 DEBUG| [stdout]                GPR20: 0000000000400000 00007339328b0000 0000000000000443 000000000000049a
  19:31:59 DEBUG| [stdout]                GPR24: 00007fffe041a740 00007fffe041a738 00007fffe041b7d8 000000000002bb67
  19:31:59 DEBUG| [stdout]                GPR28: 0000733933314d70 00007339328b110c 00000a491e036af8 0000000000100000
  19:31:59 DEBUG| [stdout] [  967.676543] NIP [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676545] LR [0000733932f41044] 0x733932f41044
  19:31:59 DEBUG| [stdout] [  967.676546] --- interrupt: 3000

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2109317/+subscriptions
References

[Bug 2109317] [NEW] ubuntu_zfs_stress locks up and kdumps on small ppc64el jammy vms
From: Mehmet Basaran, 2025-04-25