kernel-packages team mailing list archive

Thread
Date
[Bug 1319244] Re: iostat: Cannot open /proc/stat: Cannot allocate memory

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Stefan Bader <stefan.bader@xxxxxxxxxxxxx>
Date: Wed, 14 May 2014 11:35:25 -0000
Reply-to: Bug 1319244 <1319244@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
It looks like two things at play:
1. Something in the way the system is set up seems to the DMA/DMA32 pool under higher pressure
2. Reading /proc/stat ends up requesting a 64kB memory area (order-4) in the DMA32 pool on node0

While there are still two elements in the 64kB slot, they both are
emergency reserve, which open_stat did not ask for. So while there is
not much that can easily be done about the memory pressure in general,
it looks like open_stat could very well be more conservative about its
memory requirement. First the calculation reserves 128 bytes for each
possible cpu, while the actual output only produces one line of overall
stats, plus one for each online cpu. So it seems to make sense to use
the number of online cpus for the allocation. Second the upper limit for
the buffer is set to the maximum that kmalloc can do. However it is
known that anything beyond order-3 allocations (32kB) are costly and
prone to fail due to memory pressure. And having a smaller buffer is not
world ending for the stat output either. Since the proc file uses the
seq_file functions it only would cause more read calls instead of
filling all stats into one single buffer.

** Changed in: linux (Ubuntu)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu)
     Assignee: (unassigned) => Stefan Bader (smb)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1319244

Title:
  iostat: Cannot open /proc/stat: Cannot allocate memory

Status in “linux” package in Ubuntu:
  Confirmed

Bug description:
  Intermitently I get errors stating that various programs cannot
  allocate memory when reading /proc/stat

  I have 2 servers that are affected by the issue.
  Probably a duplicate of 1226172

  [1764593.455329] top: page allocation failure: order:4, mode:0x1040d0
  [1764593.455335] CPU: 3 PID: 10323 Comm: top Tainted: GF          O 3.11.0-18-generic #32-Ubuntu
  [1764593.455338] Hardware name: Intel Corporation S2400SC/S2400SC, BIOS SE5C600.86B.02.02.0002.122320131210 12/23/2013
  [1764593.455340]  0000000000000000 ffff88042670b9d0 ffffffff816e8571 00000000001040d0
  [1764593.455345]  ffff88042670ba58 ffffffff81144ae0 00000000ffffffff ffff880a3fffcb48
  [1764593.455348]  000000000000000e ffff88042670ba28 ffffffff00000040 0000000000000007
  [1764593.455351] Call Trace:
  [1764593.455361]  [<ffffffff816e8571>] dump_stack+0x45/0x56
  [1764593.455367]  [<ffffffff81144ae0>] warn_alloc_failed+0xf0/0x140
  [1764593.455371]  [<ffffffff81148a15>] __alloc_pages_nodemask+0x725/0x920
  [1764593.455376]  [<ffffffff81184589>] alloc_pages_current+0xa9/0x160
  [1764593.455379]  [<ffffffff81143bfe>] __get_free_pages+0xe/0x50
  [1764593.455384]  [<ffffffff8118d75e>] kmalloc_order_trace+0x2e/0xa0
  [1764593.455388]  [<ffffffff811c9960>] ? single_open+0xb0/0xb0
  [1764593.455391]  [<ffffffff8118ee19>] __kmalloc+0x159/0x180
  [1764593.455393]  [<ffffffff811c9960>] ? single_open+0xb0/0xb0
  [1764593.455398]  [<ffffffff81213c10>] stat_open+0x50/0xc0
  [1764593.455401]  [<ffffffff8120a700>] proc_reg_open+0x70/0x110
  [1764593.455405]  [<ffffffff811a5d4b>] do_dentry_open+0x1eb/0x280
  [1764593.455408]  [<ffffffff8120a690>] ? proc_reg_release+0x70/0x70
  [1764593.455411]  [<ffffffff811a5e69>] vfs_open+0x49/0x50
  [1764593.455415]  [<ffffffff811b5e13>] do_last+0x1c3/0xe50
  [1764593.455418]  [<ffffffff8118e5d8>] ? kmem_cache_alloc_trace+0x108/0x130
  [1764593.455421]  [<ffffffff811b6b61>] path_openat+0xc1/0x630
  [1764593.455426]  [<ffffffff811689d9>] ? handle_mm_fault+0x299/0x670
  [1764593.455429]  [<ffffffff811b7eda>] do_filp_open+0x3a/0x90
  [1764593.455433]  [<ffffffff816ef78e>] ? _raw_spin_lock+0xe/0x20
  [1764593.455437]  [<ffffffff811c4847>] ? __alloc_fd+0xa7/0x130
  [1764593.455440]  [<ffffffff811a704c>] do_sys_open+0x12c/0x270
  [1764593.455445]  [<ffffffff81074989>] ? __set_current_blocked+0x49/0x60
  [1764593.455448]  [<ffffffff811a71ae>] SyS_open+0x1e/0x20
  [1764593.455452]  [<ffffffff816f841d>] system_call_fastpath+0x1a/0x1f
  [1764593.455454] Mem-Info:
  [1764593.455455] Node 0 DMA per-cpu:
  [1764593.455458] CPU    0: hi:    0, btch:   1 usd:   0
  [1764593.455459] CPU    1: hi:    0, btch:   1 usd:   0
  [1764593.455461] CPU    2: hi:    0, btch:   1 usd:   0
  [1764593.455462] CPU    3: hi:    0, btch:   1 usd:   0
  [1764593.455463] CPU    4: hi:    0, btch:   1 usd:   0
  [1764593.455465] CPU    5: hi:    0, btch:   1 usd:   0
  [1764593.455466] CPU    6: hi:    0, btch:   1 usd:   0
  [1764593.455468] CPU    7: hi:    0, btch:   1 usd:   0
  [1764593.455469] Node 0 DMA32 per-cpu:
  [1764593.455471] CPU    0: hi:  186, btch:  31 usd:   0
  [1764593.455472] CPU    1: hi:  186, btch:  31 usd:   0
  [1764593.455474] CPU    2: hi:  186, btch:  31 usd:   0
  [1764593.455475] CPU    3: hi:  186, btch:  31 usd:   0
  [1764593.455477] CPU    4: hi:  186, btch:  31 usd:   0
  [1764593.455478] CPU    5: hi:  186, btch:  31 usd:   0
  [1764593.455480] CPU    6: hi:  186, btch:  31 usd:   0
  [1764593.455481] CPU    7: hi:  186, btch:  31 usd:   0
  [1764593.455482] Node 0 Normal per-cpu:
  [1764593.455484] CPU    0: hi:  186, btch:  31 usd:   0
  [1764593.455485] CPU    1: hi:  186, btch:  31 usd:   0
  [1764593.455487] CPU    2: hi:  186, btch:  31 usd:   0
  [1764593.455488] CPU    3: hi:  186, btch:  31 usd:   0
  [1764593.455490] CPU    4: hi:  186, btch:  31 usd:   0
  [1764593.455491] CPU    5: hi:  186, btch:  31 usd:   0
  [1764593.455492] CPU    6: hi:  186, btch:  31 usd:   0
  [1764593.455494] CPU    7: hi:  186, btch:  31 usd:   0
  [1764593.455495] Node 1 Normal per-cpu:
  [1764593.455497] CPU    0: hi:  186, btch:  31 usd:   0
  [1764593.455498] CPU    1: hi:  186, btch:  31 usd:   0
  [1764593.455500] CPU    2: hi:  186, btch:  31 usd:   0
  [1764593.455501] CPU    3: hi:  186, btch:  31 usd:   0
  [1764593.455503] CPU    4: hi:  186, btch:  31 usd:   0
  [1764593.455504] CPU    5: hi:  186, btch:  31 usd:  30
  [1764593.455505] CPU    6: hi:  186, btch:  31 usd:   0
  [1764593.455507] CPU    7: hi:  186, btch:  31 usd:   0
  [1764593.455511] active_anon:4739030 inactive_anon:2482840 isolated_anon:0
  [1764593.455511]  active_file:6228818 inactive_file:5945293 isolated_file:0
  [1764593.455511]  unevictable:1459 dirty:61 writeback:0 unstable:0
  [1764593.455511]  free:318692 slab_reclaimable:462577 slab_unreclaimable:149699
  [1764593.455511]  mapped:25851 shmem:2934 pagetables:27313 bounce:0
  [1764593.455511]  free_cma:0
  [1764593.455514] Node 0 DMA free:15848kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15932kB managed:15848kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
  [1764593.455519] lowmem_reserve[]: 0 2897 40132 40132
  [1764593.455523] Node 0 DMA32 free:505356kB min:3244kB low:4052kB high:4864kB active_anon:682760kB inactive_anon:959288kB active_file:336884kB inactive_file:205164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3049268kB managed:2970684kB mlocked:0kB dirty:0kB writeback:0kB mapped:2720kB shmem:8kB slab_reclaimable:261396kB slab_unreclaimable:16056kB kernel_stack:120kB pagetables:952kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
  [1764593.455527] lowmem_reserve[]: 0 0 37234 37234
  [1764593.455530] Node 0 Normal free:615816kB min:41704kB low:52128kB high:62556kB active_anon:11679740kB inactive_anon:6085128kB active_file:9771528kB inactive_file:8399428kB unevictable:3380kB isolated(anon):0kB isolated(file):0kB present:38797312kB managed:38127692kB mlocked:3380kB dirty:192kB writeback:0kB mapped:29824kB shmem:5620kB slab_reclaimable:724768kB slab_unreclaimable:205508kB kernel_stack:8296kB pagetables:66784kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
  [1764593.455535] lowmem_reserve[]: 0 0 0 0
  [1764593.455538] Node 1 Normal free:137748kB min:45144kB low:56428kB high:67716kB active_anon:6593620kB inactive_anon:2886944kB active_file:14806860kB inactive_file:15176580kB unevictable:2456kB isolated(anon):0kB isolated(file):0kB present:41943040kB managed:41273172kB mlocked:2456kB dirty:52kB writeback:0kB mapped:70860kB shmem:6108kB slab_reclaimable:864144kB slab_unreclaimable:377232kB kernel_stack:2912kB pagetables:41516kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
  [1764593.455543] lowmem_reserve[]: 0 0 0 0
  [1764593.455545] Node 0 DMA: 0*4kB 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15848kB
  [1764593.455557] Node 0 DMA32: 70707*4kB (UEMR) 17882*8kB (UEMR) 4937*16kB (UEMR) 13*32kB (UEMR) 2*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 505420kB
  [1764593.455567] Node 0 Normal: 137539*4kB (UEM) 7098*8kB (UEM) 252*16kB (UEMR) 23*32kB (UMR) 10*64kB (UMR) 13*128kB (UER) 5*256kB (UER) 1*512kB (R) 1*1024kB (R) 0*2048kB 0*4096kB = 616828kB
  [1764593.455578] Node 1 Normal: 33118*4kB (UEM) 505*8kB (UEM) 39*16kB (UEM) 27*32kB (UEM) 7*64kB (M) 2*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 138704kB
  [1764593.455589] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  [1764593.455591] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
  [1764593.455592] 13157020 total pagecache pages
  [1764593.455593] 979097 pages in swap cache
  [1764593.455595] Swap cache stats: add 13258848, delete 12279729, find 15382022/16560210
  [1764593.455596] Free swap  = 45316860kB
  [1764593.455597] Total swap = 56722668kB
  [1764593.597232] wcaxx 0000:08:00.0: Underrun detected by hardware. Latency at max of 20ms.
  [1764593.766212] wcaxx 0000:08:00.0: Underrun detected by hardware. Latency at max of 20ms.
  [1764593.766286] 20971519 pages RAM
  [1764593.766287] 374670 pages reserved
  [1764593.766289] 10669900 pages shared
  [1764593.766290] 11731430 pages non-shared

  ProblemType: Bug
  DistroRelease: Ubuntu 13.10
  Package: linux-image 3.11.0.20.21
  ProcVersionSignature: Ubuntu 3.11.0-18.32-generic 3.11.10.4
  Uname: Linux 3.11.0-18-generic x86_64
  ApportVersion: 2.12.5-0ubuntu2.2
  Architecture: amd64
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory: 'iw'
  Date: Wed May 14 00:12:36 2014
  HibernationDevice: RESUME=UUID=d678f2d9-0912-4fe6-97ca-e1685a5b5c8e
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  MachineType: Intel Corporation S2400SC
  MarkForUpload: True
  ProcFB: 0 VESA VGA
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.11.0-18-generic root=UUID=0e0470c8-fe13-4b7f-ace7-e59d91e04b07 ro
  RelatedPackageVersions:
   linux-restricted-modules-3.11.0-18-generic N/A
   linux-backports-modules-3.11.0-18-generic  N/A
   linux-firmware                             1.116.2
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UpgradeStatus: Upgraded to saucy on 2013-11-12 (182 days ago)
  dmi.bios.date: 12/23/2013
  dmi.bios.vendor: Intel Corp.
  dmi.bios.version: SE5C600.86B.02.02.0002.122320131210
  dmi.board.asset.tag: ....................
  dmi.board.name: S2400SC
  dmi.board.vendor: Intel Corporation
  dmi.board.version: G18552-403
  dmi.chassis.asset.tag: ....................
  dmi.chassis.type: 23
  dmi.chassis.vendor: ...............................
  dmi.chassis.version: p4308cpmhen
  dmi.modalias: dmi:bvnIntelCorp.:bvrSE5C600.86B.02.02.0002.122320131210:bd12/23/2013:svnIntelCorporation:pnS2400SC:pvr....................:rvnIntelCorporation:rnS2400SC:rvrG18552-403:cvn...............................:ct23:cvrp4308cpmhen:
  dmi.product.name: S2400SC
  dmi.product.version: ....................
  dmi.sys.vendor: Intel Corporation

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319244/+subscriptions
References

[Bug 1319244] [NEW] iostat: Cannot open /proc/stat: Cannot allocate memory
From: Stephane Bakhos, 2014-05-14