canonical-ubuntu-qa team mailing list archive
-
canonical-ubuntu-qa team
-
Mailing list archive
-
Message #01795
[Bug 2041512] Re: shiftfs-lxd.sh test in ubuntu_overlayfs_smoke_test failed because of OOM on 64k AWS a1.medium
** Description changed:
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G
ram)
The test can either timeout with the 600 second threshold , or fail with instance creation:
- Error: Failed instance creation: websocket: close 1006 (abnormal closure): unexpected EOF
+ * Error: Failed instance creation: websocket: close 1006 (abnormal closure): unexpected EOF
+ * Error: Failed instance creation: Failed creating instance from image: Unpack failed: Failed to run: unsquashfs -f -d /var/snap/lxd/common/lxd/storage-pools/zfs/images/999d86db6c50e9de715f4b00e7f95523d33c376880d0cf1212acb5d9c76ffca8/rootfs -n -da 179 -fr 179 -p 1 /var/snap/lxd/common/lxd/images/999d86db6c50e9de715f4b00e7f95523d33c376880d0cf1212acb5d9c76ffca8.rootfs: signal: killed
Or even being killed and therefore leads to an incomplete test report.
OOM can be observed in dmesg, it's probably the reason why we are seeing this kind of unexpected failures:
[13076.574301] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-88-lowlatency-64k #98-Ubuntu
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+0x0/0x204
[13076.574328] show_stack+0x20/0x2c
[13076.574331] dump_stack_lvl+0x68/0x84
[13076.574335] dump_stack+0x18/0x34
[13076.574337] dump_header+0x54/0x220
[13076.574341] oom_kill_process+0x22c/0x230
[13076.574346] out_of_memory+0xe4/0x364
[13076.574350] __alloc_pages_may_oom+0x118/0x19c
[13076.574353] __alloc_pages_slowpath.constprop.0+0x590/0x840
[13076.574356] __alloc_pages+0x200/0x270
[13076.574358] alloc_pages+0xb4/0x1bc
[13076.574363] __page_cache_alloc+0xd4/0xe4
[13076.574366] pagecache_get_page+0x178/0x6c0
[13076.574369] filemap_fault+0x57c/0x944
[13076.574371] __do_fault+0x44/0x170
[13076.574375] do_read_fault+0xe4/0x1e0
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_pte_fault+0x5c/0x1d0
[13076.574385] __handle_mm_fault+0x104/0x350
[13076.574388] handle_mm_fault+0xf8/0x274
[13076.574391] do_page_fault+0x160/0x560
[13076.574395] do_translation_fault+0x98/0xf0
[13076.574399] do_mem_abort+0x4c/0xc0
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_sync_handler+0x124/0x130
[13076.574409] el0t_64_sync+0x1a4/0x1a8
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
active_file:4 inactive_file:19 isolated_file:0
unevictable:454 dirty:0 writeback:0
slab_reclaimable:450 slab_unreclaimable:3151
mapped:131 shmem:261 pagetables:290 bounce:0
kernel_misc_reclaimable:0
free:9608 free_pcp:2 free_cma:8180
[13076.574421] Node 0 active_anon:16576kB inactive_anon:653312kB active_file:256kB inactive_file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:16512kB pagetables:18560kB all_unreclaimable? yes
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_highatomic:0KB active_anon:1088kB inactive_anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_highatomic:0KB active_anon:15488kB inactive_anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/,task=unsquashfs,pid=6063,uid=0
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(1698390302.185:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd_archive-var-snap-lxd-common-lxd-storage-pools-zfs-images-2291196f955ef93ccfcf536069b999f72a1bf96770ef1752720312584fb34553-rootfs" pid=10037 comm="apparmor_parser"
--
You received this bug notification because you are a member of Canonical
Platform QA Team, which is subscribed to ubuntu-kernel-tests.
https://bugs.launchpad.net/bugs/2041512
Title:
shiftfs-lxd.sh test in ubuntu_overlayfs_smoke_test failed because of
OOM on 64k AWS a1.medium
Status in ubuntu-kernel-tests:
New
Bug description:
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G
ram)
The test can either timeout with the 600 second threshold , or fail with instance creation:
* Error: Failed instance creation: websocket: close 1006 (abnormal closure): unexpected EOF
* Error: Failed instance creation: Failed creating instance from image: Unpack failed: Failed to run: unsquashfs -f -d /var/snap/lxd/common/lxd/storage-pools/zfs/images/999d86db6c50e9de715f4b00e7f95523d33c376880d0cf1212acb5d9c76ffca8/rootfs -n -da 179 -fr 179 -p 1 /var/snap/lxd/common/lxd/images/999d86db6c50e9de715f4b00e7f95523d33c376880d0cf1212acb5d9c76ffca8.rootfs: signal: killed
Or even being killed and therefore leads to an incomplete test report.
OOM can be observed in dmesg, it's probably the reason why we are seeing this kind of unexpected failures:
[13076.574301] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-88-lowlatency-64k #98-Ubuntu
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+0x0/0x204
[13076.574328] show_stack+0x20/0x2c
[13076.574331] dump_stack_lvl+0x68/0x84
[13076.574335] dump_stack+0x18/0x34
[13076.574337] dump_header+0x54/0x220
[13076.574341] oom_kill_process+0x22c/0x230
[13076.574346] out_of_memory+0xe4/0x364
[13076.574350] __alloc_pages_may_oom+0x118/0x19c
[13076.574353] __alloc_pages_slowpath.constprop.0+0x590/0x840
[13076.574356] __alloc_pages+0x200/0x270
[13076.574358] alloc_pages+0xb4/0x1bc
[13076.574363] __page_cache_alloc+0xd4/0xe4
[13076.574366] pagecache_get_page+0x178/0x6c0
[13076.574369] filemap_fault+0x57c/0x944
[13076.574371] __do_fault+0x44/0x170
[13076.574375] do_read_fault+0xe4/0x1e0
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_pte_fault+0x5c/0x1d0
[13076.574385] __handle_mm_fault+0x104/0x350
[13076.574388] handle_mm_fault+0xf8/0x274
[13076.574391] do_page_fault+0x160/0x560
[13076.574395] do_translation_fault+0x98/0xf0
[13076.574399] do_mem_abort+0x4c/0xc0
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_sync_handler+0x124/0x130
[13076.574409] el0t_64_sync+0x1a4/0x1a8
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
active_file:4 inactive_file:19 isolated_file:0
unevictable:454 dirty:0 writeback:0
slab_reclaimable:450 slab_unreclaimable:3151
mapped:131 shmem:261 pagetables:290 bounce:0
kernel_misc_reclaimable:0
free:9608 free_pcp:2 free_cma:8180
[13076.574421] Node 0 active_anon:16576kB inactive_anon:653312kB active_file:256kB inactive_file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:16512kB pagetables:18560kB all_unreclaimable? yes
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_highatomic:0KB active_anon:1088kB inactive_anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_highatomic:0KB active_anon:15488kB inactive_anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/,task=unsquashfs,pid=6063,uid=0
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(1698390302.185:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd_archive-var-snap-lxd-common-lxd-storage-pools-zfs-images-2291196f955ef93ccfcf536069b999f72a1bf96770ef1752720312584fb34553-rootfs" pid=10037 comm="apparmor_parser"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2041512/+subscriptions
References
