canonical-ubuntu-qa team mailing list archive
-
canonical-ubuntu-qa team
-
Mailing list archive
-
Message #01792
[Bug 2041512] [NEW] shiftfs-lxd.sh test in ubuntu_overlayfs_smoke_test failed because of OOM on 64k AWS a1.medium
Public bug reported:
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G
ram)
Trying to run the test will cause OOM and unexpected failure:
[13076.574301] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-88-lowlatency-64k #98-Ubuntu
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+0x0/0x204
[13076.574328] show_stack+0x20/0x2c
[13076.574331] dump_stack_lvl+0x68/0x84
[13076.574335] dump_stack+0x18/0x34
[13076.574337] dump_header+0x54/0x220
[13076.574341] oom_kill_process+0x22c/0x230
[13076.574346] out_of_memory+0xe4/0x364
[13076.574350] __alloc_pages_may_oom+0x118/0x19c
[13076.574353] __alloc_pages_slowpath.constprop.0+0x590/0x840
[13076.574356] __alloc_pages+0x200/0x270
[13076.574358] alloc_pages+0xb4/0x1bc
[13076.574363] __page_cache_alloc+0xd4/0xe4
[13076.574366] pagecache_get_page+0x178/0x6c0
[13076.574369] filemap_fault+0x57c/0x944
[13076.574371] __do_fault+0x44/0x170
[13076.574375] do_read_fault+0xe4/0x1e0
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_pte_fault+0x5c/0x1d0
[13076.574385] __handle_mm_fault+0x104/0x350
[13076.574388] handle_mm_fault+0xf8/0x274
[13076.574391] do_page_fault+0x160/0x560
[13076.574395] do_translation_fault+0x98/0xf0
[13076.574399] do_mem_abort+0x4c/0xc0
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_sync_handler+0x124/0x130
[13076.574409] el0t_64_sync+0x1a4/0x1a8
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
active_file:4 inactive_file:19 isolated_file:0
unevictable:454 dirty:0 writeback:0
slab_reclaimable:450 slab_unreclaimable:3151
mapped:131 shmem:261 pagetables:290 bounce:0
kernel_misc_reclaimable:0
free:9608 free_pcp:2 free_cma:8180
[13076.574421] Node 0 active_anon:16576kB inactive_anon:653312kB active_file:256kB inactive_file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:16512kB pagetables:18560kB all_unreclaimable? yes
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_highatomic:0KB active_anon:1088kB inactive_anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_highatomic:0KB active_anon:15488kB inactive_anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/,task=unsquashfs,pid=6063,uid=0
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(1698390302.185:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd_archive-var-snap-lxd-common-lxd-storage-pools-zfs-images-2291196f955ef93ccfcf536069b999f72a1bf96770ef1752720312584fb34553-rootfs" pid=10037 comm="apparmor_parser"
** Affects: ubuntu-kernel-tests
Importance: Undecided
Status: New
** Tags: 64k arm64 aws jammy lunar sru-20230904 sru-20231002 ubuntu-overlayfs-smoke-test
** Tags added: sru-20230904 sru-20231002
--
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2041512
Title:
shiftfs-lxd.sh test in ubuntu_overlayfs_smoke_test failed because of
OOM on 64k AWS a1.medium
Status in ubuntu-kernel-tests:
New
Bug description:
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G
ram)
Trying to run the test will cause OOM and unexpected failure:
[13076.574301] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-88-lowlatency-64k #98-Ubuntu
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+0x0/0x204
[13076.574328] show_stack+0x20/0x2c
[13076.574331] dump_stack_lvl+0x68/0x84
[13076.574335] dump_stack+0x18/0x34
[13076.574337] dump_header+0x54/0x220
[13076.574341] oom_kill_process+0x22c/0x230
[13076.574346] out_of_memory+0xe4/0x364
[13076.574350] __alloc_pages_may_oom+0x118/0x19c
[13076.574353] __alloc_pages_slowpath.constprop.0+0x590/0x840
[13076.574356] __alloc_pages+0x200/0x270
[13076.574358] alloc_pages+0xb4/0x1bc
[13076.574363] __page_cache_alloc+0xd4/0xe4
[13076.574366] pagecache_get_page+0x178/0x6c0
[13076.574369] filemap_fault+0x57c/0x944
[13076.574371] __do_fault+0x44/0x170
[13076.574375] do_read_fault+0xe4/0x1e0
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_pte_fault+0x5c/0x1d0
[13076.574385] __handle_mm_fault+0x104/0x350
[13076.574388] handle_mm_fault+0xf8/0x274
[13076.574391] do_page_fault+0x160/0x560
[13076.574395] do_translation_fault+0x98/0xf0
[13076.574399] do_mem_abort+0x4c/0xc0
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_sync_handler+0x124/0x130
[13076.574409] el0t_64_sync+0x1a4/0x1a8
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
active_file:4 inactive_file:19 isolated_file:0
unevictable:454 dirty:0 writeback:0
slab_reclaimable:450 slab_unreclaimable:3151
mapped:131 shmem:261 pagetables:290 bounce:0
kernel_misc_reclaimable:0
free:9608 free_pcp:2 free_cma:8180
[13076.574421] Node 0 active_anon:16576kB inactive_anon:653312kB active_file:256kB inactive_file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:16512kB pagetables:18560kB all_unreclaimable? yes
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_highatomic:0KB active_anon:1088kB inactive_anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_highatomic:0KB active_anon:15488kB inactive_anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/,task=unsquashfs,pid=6063,uid=0
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(1698390302.185:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd_archive-var-snap-lxd-common-lxd-storage-pools-zfs-images-2291196f955ef93ccfcf536069b999f72a1bf96770ef1752720312584fb34553-rootfs" pid=10037 comm="apparmor_parser"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2041512/+subscriptions
Follow ups
