kernel-packages team mailing list archive

Thread
Date
[Bug 1350522] Re: EC2 kernel crash due to vmalloc

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: Stefan Bader <stefan.bader@xxxxxxxxxxxxx>
Date: Fri, 01 Aug 2014 07:53:33 -0000
Reply-to: Bug 1350522 <1350522@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
So I guess the next question would be whether in all 95% which do work
(deducted from the ~5% failing), are all Xen version != 4.2 or a mix of
versions, including 4.2? At least the fact that this persists while only
rebooting make some hypervisor code involvement likely.

The log looks to me like further prove that it is systemd-udevd itself at the bottom of the problem. Which probably should not surprise me with the usual tendency of taking over the world that systemd seems to have. So the systemd-udevd is loading a module, when it gets this. We can see it is a vmalloc. So the size of the area is not that critical.
All the cases I looked at had: vmalloc: "allocation failure, allocated 20480 of 24576 bytes" after the first trace. The way I understand that first warning is: we got some space to store a PTE (page table entry) but that slot seems to already (or still) contain a valid entry. This would be returned as -EBUSY.

After the warning about the PTE, the two lines about the allocation failure are printed:
  vmalloc: allocation failure, allocated 20480 of 24576 bytes
  systemd-udevd: page allocation failure: order:0, mode:0xd2

The order-0 allocation is a bit odd. The flags/mode I would decode as:
  GFP_KERNEL(__GFP_WAIT|__GFP_IO|__GFP_FS)|__GFP_HIGHMEM

So a normal kernel page allocation (that should wait) trying for highmem
(which should fallback for normal->dma32->dma). Not sure about what
exactly the memory printouts tell us. Will try to chat with Andy about
those. But more or less this sequence seems to repeat with varying
amounts of requested/returned memory. So a bit weird that a) a normal
order-0 allocation fails and this only happens in a few cases.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1350522

Title:
  EC2 kernel crash due to vmalloc

Status in “linux” package in Ubuntu:
  Confirmed
Status in “linux” source package in Utopic:
  Confirmed

Bug description:
  During Alpha-2 automated testing, I saw the following in a log:

  19:35:02  [    2.475810] systemd-udevd[95]: starting version 204
  19:35:02  [    2.547049] ------------[ cut here ]------------
  19:35:02  [    2.547065] WARNING: CPU: 0 PID: 97 at /build/buildd/linux-3.16.0/mm/vmalloc.c:128 vmap_page_range_noflush+0x2d1/0x370()
  19:35:02  [    2.547069] Modules linked in:
  19:35:02  [    2.547073] CPU: 0 PID: 97 Comm: systemd-udevd Not tainted 3.16.0-6-generic #11-Ubuntu
  19:35:02  [    2.547077]  0000000000000009 ffff880002defb98 ffffffff81755538 0000000000000000
  19:35:02  [    2.547082]  ffff880002defbd0 ffffffff8106bb0d ffff88000400ec88 0000000000000001
  19:35:02  [    2.547086]  ffff880002fcfb00 ffffffffc0391000 0000000000000000 ffff880002defbe0
  19:35:02  [    2.547090] Call Trace:
  19:35:02  [    2.547096]  [<ffffffff81755538>] dump_stack+0x45/0x56
  19:35:02  [    2.547101]  [<ffffffff8106bb0d>] warn_slowpath_common+0x7d/0xa0
  19:35:02  [    2.547104]  [<ffffffff8106bbea>] warn_slowpath_null+0x1a/0x20
  19:35:02  [    2.547108]  [<ffffffff81197c31>] vmap_page_range_noflush+0x2d1/0x370
  19:35:02  [    2.547112]  [<ffffffff81197cfe>] map_vm_area+0x2e/0x40
  19:35:02  [    2.547115]  [<ffffffff8119a058>] __vmalloc_node_range+0x188/0x280
  19:35:02  [    2.547120]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547124]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547129]  [<ffffffff8104f294>] module_alloc+0x74/0xd0
  19:35:02  [    2.547132]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547135]  [<ffffffff810e92b4>] module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547146]  [<ffffffff810e9a6c>] layout_and_allocate+0x74c/0xc70
  19:35:02  [    2.547149]  [<ffffffff810ea063>] load_module+0xd3/0x1b70
  19:35:02  [    2.547154]  [<ffffffff811cfeb1>] ? vfs_read+0xf1/0x170
  19:35:02  [    2.547157]  [<ffffffff810e7aa1>] ? copy_module_from_fd.isra.46+0x121/0x180
  19:35:02  [    2.547161]  [<ffffffff810ebc76>] SyS_finit_module+0x86/0xb0
  19:35:02  [    2.547167]  [<ffffffff8175de7f>] tracesys+0xe1/0xe6
  19:35:02  [    2.547169] ---[ end trace 8a5de7fc66e75fe4 ]---
  19:35:02  [    2.547172] vmalloc: allocation failure, allocated 20480 of 24576 bytes
  19:35:02  [    2.547175] systemd-udevd: page allocation failure: order:0, mode:0xd2
  19:35:02  [    2.547180] CPU: 0 PID: 97 Comm: systemd-udevd Tainted: G        W     3.16.0-6-generic #11-Ubuntu
  19:35:02  [    2.547183]  ffffffff81a88bc0 ffff880002defc08 ffffffff81755538 00000000000000d2
  19:35:02  [    2.547187]  ffff880002defc90 ffffffff811642bf ffffffff81a88bc0 ffff880002defc28
  19:35:02  [    2.547191]  00003fff00000018 ffff880002defca0 ffff880002defc40 0000000000000163
  19:35:02  [    2.547195] Call Trace:
  19:35:02  [    2.547197]  [<ffffffff81755538>] dump_stack+0x45/0x56
  19:35:02  [    2.547202]  [<ffffffff811642bf>] warn_alloc_failed+0xdf/0x130
  19:35:02  [    2.547207]  [<ffffffff8119a118>] __vmalloc_node_range+0x248/0x280
  19:35:02  [    2.547210]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547214]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547217]  [<ffffffff8104f294>] module_alloc+0x74/0xd0
  19:35:02  [    2.547220]  [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547224]  [<ffffffff810e92b4>] module_alloc_update_bounds+0x14/0x70
  19:35:02  [    2.547229]  [<ffffffff810e9a6c>] layout_and_allocate+0x74c/0xc70
  19:35:02  [    2.547232]  [<ffffffff810ea063>] load_module+0xd3/0x1b70
  19:35:02  [    2.547235]  [<ffffffff811cfeb1>] ? vfs_read+0xf1/0x170
  19:35:02  [    2.547238]  [<ffffffff810e7aa1>] ? copy_module_from_fd.isra.46+0x121/0x180
  19:35:02  [    2.547242]  [<ffffffff810ebc76>] SyS_finit_module+0x86/0xb0
  19:35:02  [    2.547246]  [<ffffffff8175de7f>] tracesys+0xe1/0xe6
  19:35:02  [    2.547248] Mem-Info:
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jul 30 20:19 seq
   crw-rw---- 1 root audio 116, 33 Jul 30 20:19 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.4-0ubuntu2
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  CurrentDmesg:
   [   42.901579] systemd-logind[1378]: New seat seat0.
   [   42.932887] systemd-logind[1378]: New session 1 of user ubuntu.
  DistroRelease: Ubuntu 14.10
  Ec2AMI: ami-c571d8d8
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: sa-east-1b
  Ec2InstanceType: m3.medium
  Ec2Kernel: aki-5553f448
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory
  Lspci:
   
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  Package: linux-image-virtual 3.16.0.5.6
  PackageArchitecture: amd64
  PciMultimedia:
   
  ProcFB:
   
  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
  ProcModules:
   
  ProcVersionSignature: User Name 3.16.0-5.10-generic 3.16.0-rc6
  RelatedPackageVersions:
   linux-restricted-modules-3.16.0-5-generic N/A
   linux-backports-modules-3.16.0-5-generic  N/A
   linux-firmware                            N/A
  RfKill: Error: [Errno 2] No such file or directory
  Tags: utopic ec2-images third-party-packages
  Uname: Linux 3.16.0-5-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
  _MarkForUpload: True
  --- 
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jul 31 20:13 seq
   crw-rw---- 1 root audio 116, 33 Jul 31 20:13 timer
  AplayDevices: Error: [Errno 2] No such file or directory
  ApportVersion: 2.14.5-0ubuntu1
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: Error: [Errno 2] No such file or directory
  CurrentDmesg:
   [   17.975584] systemd-logind[1248]: New seat seat0.
   [   17.993902] systemd-logind[1248]: New session 1 of user ubuntu.
  DistroRelease: Ubuntu 14.10
  Ec2AMI: ami-b7ce1dc0
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: eu-west-1c
  Ec2InstanceType: m3.medium
  Ec2Kernel: aki-52a34525
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory
  Lspci:
   
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  Package: linux-image-virtual 3.16.0.6.7
  PackageArchitecture: amd64
  PciMultimedia:
   
  ProcFB:
   
  ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 initcall_debug debug ignore_loglevel LOGLEVEL=8
  ProcModules:
   
  ProcVersionSignature: Ubuntu 3.16.0-6.11-generic 3.16.0-rc7
  RelatedPackageVersions:
   linux-restricted-modules-3.16.0-6-generic N/A
   linux-backports-modules-3.16.0-6-generic  N/A
   linux-firmware                            N/A
  RfKill: Error: [Errno 2] No such file or directory
  Tags:  utopic ec2-images
  Uname: Linux 3.16.0-6-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups:
   
  _MarkForUpload: True

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350522/+subscriptions
References

[Bug 1350522] [NEW] EC2 kernel crash due to vmalloc
From: Ben Howard, 2014-07-30