canonical-ubuntu-qa team mailing list archive

Thread
Date
[Bug 2046327] [NEW] cpuacct_100_100 in ubuntu_ltp_controllers failed wth "fopen failed: Cannot allocate memory" on B-4.15 PowerPC

To: canonical-ubuntu-qa@xxxxxxxxxxxxxxxxxxx
From: Po-Hsu Lin <2046327@xxxxxxxxxxxxxxxxxx>
Date: Wed, 13 Dec 2023 07:11:23 -0000
Reply-to: Bug 2046327 <2046327@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Issue found on Openstack PowerPC with 4.15.0-220.231

The last B-4.15 test report I can find is 4.15.0-213.224, back then we
don't run this ubuntu_ltp_controllers test on openstack PowerPC.

It only affects B-4.15 PowerPC, for 5.4 it will fail on PowerPC with bug
2039515

The test will fail with:
  cpuacct_task: Cannot allocate memory


Test log:
<<<test_start>>>
tag=cpuacct_100_100 stime=1702442504
cmdline="cpuacct.sh 100 100"
contacts=""
analysis=exit
<<<test_output>>>
incrementing stop
cpuacct 1 TINFO: timeout per run is 0h 5m 0s
tst_pid.c:84: TINFO: Cannot read session user limits from '/sys/fs/cgroup/user.slice/user-1000.slice/pids.max'
tst_pid.c:94: TINFO: Found limit of processes 10387 (from /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max)
cpuacct 1 TINFO: task limit fulfilled (approximate need 10000, limit 10205)
cpuacct 1 TINFO: memory requirements fulfilled (approximate need 1638400 kB, free 3247872 kB)
cpuacct 1 TINFO: cpuacct: /sys/fs/cgroup/cpu,cpuacct
cpuacct 1 TINFO: Creating 100 subgroups each with 100 processes
/opt/ltp/testcases/bin/cpuacct.sh: 0: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: 
cpuacct_task: Cannot allocate memorycpuacct_task: Cannot allocate memory

/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
cpuacct_task: Cannot allocate memory
Cannot fork
<<<execution_status>>>
initiation_status="ok"
duration=10 termination_type=exited termination_id=2 corefile=no
cutime=17 cstime=47
<<<test_end>>>

You will see warning trace logged in dmesg:
[ 5545.557884] WARNING: CPU: 0 PID: 12470 at /build/linux-zeyvK6/linux-4.15.0/arch/powerpc/mm/mmu_context_book3s64.c:221 destroy_context+0x58/0x170
[ 5545.557887] Modules linked in: nfsd auth_rpcgss nfs_acl lockd grace sunrpc sctp vmx_crypto binfmt_misc sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_net virtio_scsi
[ 5545.557911] CPU: 0 PID: 12470 Comm: cpuacct.sh Not tainted 4.15.0-220-generic #231-Ubuntu
[ 5545.557912] NIP:  c00000000007c6d8 LR: c000000000113be4 CTR: c0000000005ca8a0
[ 5545.557914] REGS: c0000000f7b9f9f0 TRAP: 0700   Not tainted  (4.15.0-220-generic)
[ 5545.557914] MSR:  8000000000029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24004842  XER: 20040000
[ 5545.557919] CFAR: c000000000113be0 SOFTE: 1 
               GPR00: c000000000113be4 c0000000f7b9fc70 c000000001783d00 c0000000f7b3f300 
               GPR04: c00a00000038eae0 ffffffffffffff80 0000000000000110 00000000fecf0000 
               GPR08: c0000000fffd3c80 0000000000000001 ad00abe300000040 0000000000000000 
               GPR12: 0000000024004842 c00000000fa80000 0000000000000000 000001e2eaa400d8 
               GPR16: 0000000000000000 000001e2eaa42cd0 000001e2eaa42bd8 00007ffffd07f825 
               GPR20: 0000000000000000 0000000000000002 0000000000000000 c000000008d9c700 
               GPR24: c0000000ef6f8000 0000000000000000 000001e31dc52938 000001e31dc52920 
               GPR28: 0000000000000000 c0000000f7b3f300 c0000000f7b3f300 c0000000f7b3f300 
[ 5545.557930] NIP [c00000000007c6d8] destroy_context+0x58/0x170
[ 5545.557931] LR [c000000000113be4] __mmdrop+0x64/0x210
[ 5545.557932] Call Trace:
[ 5545.557934] [c0000000f7b9fc70] [c0000000f7b9fd00] 0xc0000000f7b9fd00 (unreliable)
[ 5545.557936] [c0000000f7b9fca0] [c000000000113be4] __mmdrop+0x64/0x210
[ 5545.557962] [c0000000f7b9fd20] [c0000000003fbd68] do_execveat_common.isra.13+0x858/0x910
[ 5545.557964] [c0000000f7b9fe00] [c0000000003fc154] SyS_execve+0x54/0x70
[ 5545.557966] [c0000000f7b9fe30] [c00000000000b388] system_call+0x5c/0x70
[ 5545.557967] Instruction dump:
[ 5545.557968] e94307f8 7fa95000 409e0130 60000000 3d42001b 394aa578 e93f0490 e94a0000 
[ 5545.557971] 792926e4 7d4a482a 312affff 7d295110 <0b090000> e93f07f0 2fa90000 419e0070 
[ 5545.557974] ---[ end trace f1400cf6eb33eb34 ]---

The issue can be found with LTP 20230516 and after commit 776b57984
"rwtest: Remove df symlink check"

** Affects: ubuntu-kernel-tests
     Importance: Undecided
         Status: New


** Tags: 4.15 bionic ppc64el ubuntu-ltp-controllers

-- 
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2046327

Title:
  cpuacct_100_100 in ubuntu_ltp_controllers failed wth "fopen failed:
  Cannot allocate memory" on B-4.15 PowerPC

Status in ubuntu-kernel-tests:
  New

Bug description:
  Issue found on Openstack PowerPC with 4.15.0-220.231

  The last B-4.15 test report I can find is 4.15.0-213.224, back then we
  don't run this ubuntu_ltp_controllers test on openstack PowerPC.

  It only affects B-4.15 PowerPC, for 5.4 it will fail on PowerPC with
  bug 2039515

  The test will fail with:
    cpuacct_task: Cannot allocate memory

  
  Test log:
  <<<test_start>>>
  tag=cpuacct_100_100 stime=1702442504
  cmdline="cpuacct.sh 100 100"
  contacts=""
  analysis=exit
  <<<test_output>>>
  incrementing stop
  cpuacct 1 TINFO: timeout per run is 0h 5m 0s
  tst_pid.c:84: TINFO: Cannot read session user limits from '/sys/fs/cgroup/user.slice/user-1000.slice/pids.max'
  tst_pid.c:94: TINFO: Found limit of processes 10387 (from /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max)
  cpuacct 1 TINFO: task limit fulfilled (approximate need 10000, limit 10205)
  cpuacct 1 TINFO: memory requirements fulfilled (approximate need 1638400 kB, free 3247872 kB)
  cpuacct 1 TINFO: cpuacct: /sys/fs/cgroup/cpu,cpuacct
  cpuacct 1 TINFO: Creating 100 subgroups each with 100 processes
  /opt/ltp/testcases/bin/cpuacct.sh: 0: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory/opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: 
  cpuacct_task: Cannot allocate memorycpuacct_task: Cannot allocate memory

  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  /opt/ltp/testcases/bin/cpuacct.sh: 145: /opt/ltp/testcases/bin/cpuacct.sh: cpuacct_task: Cannot allocate memory
  cpuacct_task: Cannot allocate memory
  Cannot fork
  <<<execution_status>>>
  initiation_status="ok"
  duration=10 termination_type=exited termination_id=2 corefile=no
  cutime=17 cstime=47
  <<<test_end>>>

  You will see warning trace logged in dmesg:
  [ 5545.557884] WARNING: CPU: 0 PID: 12470 at /build/linux-zeyvK6/linux-4.15.0/arch/powerpc/mm/mmu_context_book3s64.c:221 destroy_context+0x58/0x170
  [ 5545.557887] Modules linked in: nfsd auth_rpcgss nfs_acl lockd grace sunrpc sctp vmx_crypto binfmt_misc sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum virtio_net virtio_scsi
  [ 5545.557911] CPU: 0 PID: 12470 Comm: cpuacct.sh Not tainted 4.15.0-220-generic #231-Ubuntu
  [ 5545.557912] NIP:  c00000000007c6d8 LR: c000000000113be4 CTR: c0000000005ca8a0
  [ 5545.557914] REGS: c0000000f7b9f9f0 TRAP: 0700   Not tainted  (4.15.0-220-generic)
  [ 5545.557914] MSR:  8000000000029033 <SF,EE,ME,IR,DR,RI,LE>  CR: 24004842  XER: 20040000
  [ 5545.557919] CFAR: c000000000113be0 SOFTE: 1 
                 GPR00: c000000000113be4 c0000000f7b9fc70 c000000001783d00 c0000000f7b3f300 
                 GPR04: c00a00000038eae0 ffffffffffffff80 0000000000000110 00000000fecf0000 
                 GPR08: c0000000fffd3c80 0000000000000001 ad00abe300000040 0000000000000000 
                 GPR12: 0000000024004842 c00000000fa80000 0000000000000000 000001e2eaa400d8 
                 GPR16: 0000000000000000 000001e2eaa42cd0 000001e2eaa42bd8 00007ffffd07f825 
                 GPR20: 0000000000000000 0000000000000002 0000000000000000 c000000008d9c700 
                 GPR24: c0000000ef6f8000 0000000000000000 000001e31dc52938 000001e31dc52920 
                 GPR28: 0000000000000000 c0000000f7b3f300 c0000000f7b3f300 c0000000f7b3f300 
  [ 5545.557930] NIP [c00000000007c6d8] destroy_context+0x58/0x170
  [ 5545.557931] LR [c000000000113be4] __mmdrop+0x64/0x210
  [ 5545.557932] Call Trace:
  [ 5545.557934] [c0000000f7b9fc70] [c0000000f7b9fd00] 0xc0000000f7b9fd00 (unreliable)
  [ 5545.557936] [c0000000f7b9fca0] [c000000000113be4] __mmdrop+0x64/0x210
  [ 5545.557962] [c0000000f7b9fd20] [c0000000003fbd68] do_execveat_common.isra.13+0x858/0x910
  [ 5545.557964] [c0000000f7b9fe00] [c0000000003fc154] SyS_execve+0x54/0x70
  [ 5545.557966] [c0000000f7b9fe30] [c00000000000b388] system_call+0x5c/0x70
  [ 5545.557967] Instruction dump:
  [ 5545.557968] e94307f8 7fa95000 409e0130 60000000 3d42001b 394aa578 e93f0490 e94a0000 
  [ 5545.557971] 792926e4 7d4a482a 312affff 7d295110 <0b090000> e93f07f0 2fa90000 419e0070 
  [ 5545.557974] ---[ end trace f1400cf6eb33eb34 ]---

  The issue can be found with LTP 20230516 and after commit 776b57984
  "rwtest: Remove df symlink check"

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2046327/+subscriptions