kernel-packages team mailing list archive

[Gregg King <greggking@xxxxxxxxx>]

Thank you so much Rasmus!

Your solution worked for me:
sudo touch /etc/udev/rules.d/40-vm-hotadd.rules
reboot

I could always trigger the CPU usage bug with this:
stress --cpu 8 --io 4 --vm 7 --vm-bytes 128M --vm-hang 3 --timeout 60s

If running it once didn't work, a second run would do it. Now I've run it
over and over and kswap hits 1-2% during the stress test and drops back
down as soon as it ends.

Like Rasmus says, it's an AWS instance which runs on Xen.


On 17 June 2016 at 07:37, Rasmus Larsen <rla@xxxxxxxxxx> wrote:

> I had this issue too on AWS.
>
> In my case, it was the udev rule for vm-hotadd and the fix as mentioned
> previously basically came down to "touch /etc/udev/rules.d/40-vm-
> hotadd.rules" which effectively disables the /lib/udev/rules.d/40-vm-
> hotadd.rules file (after a reboot).
>
> The udev rule basically seems to only be active for Xen or Hyper-V and
> while it seems the Hyper-V stuff was also present in previous versions,
> the Xen stuff seems to be introduced in 15.10 or newer.
>
> So if you're seeing this issue on anything running on Xen, including
> AWS, try:
>
> touch /etc/udev/rules.d/40-vm-hotadd.rules
> reboot
>
> This is probably a bug in Xen or a bug in the kernel.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1518457
>
> Title:
>   kswapd0 100% CPU usage
>
> Status in Linux:
>   Unknown
> Status in linux package in Ubuntu:
>   Confirmed
>
> Bug description:
>   As per bug 721896 and various others:
>
>   I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
>   Occasionally (about once a day), kswapd0 falls into a busy loop and
>   spins on 100% CPU usage indefinitely. This can be provoked by
>   copying/writing large files (e.g. dding a 256MB file), but it happens
>   occasionally otherwise. System memory usage (not including
>   buffers/caches) currently sits at 36%, which is typical[1]. Initially
>   I had no swap space configured; I've since tried enabling a 256MB swap
>   file, but the problem continues to occur and no swap space is used.
>   The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.
>
>   Happy to provide further information/take further debugging actions.
>
>
>   [1] Full output from `free`:
>                total       used       free     shared    buffers     cached
>   Mem:       1014936     483448     531488      28556       9756     112700
>   -/+ buffers/cache:     360992     653944
>   Swap:       262140          0     262140
>
>   ProblemType: Bug
>   DistroRelease: Ubuntu 15.10
>   Package: linux-image-4.2.0-18-generic 4.2.0-18.22
>   ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
>   Uname: Linux 4.2.0-18-generic x86_64
>   AlsaDevices:
>    total 0
>    crw-rw---- 1 root audio 116,  1 Nov 19 19:40 seq
>    crw-rw---- 1 root audio 116, 33 Nov 19 19:40 timer
>   AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
>   ApportVersion: 2.19.1-0ubuntu5
>   Architecture: amd64
>   ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
>   AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
> '/dev/snd/timer'] failed with exit code 1:
>   CRDA: N/A
>   Date: Fri Nov 20 20:44:30 2015
>   Ec2AMI: ami-1c552a76
>   Ec2AMIManifest: (unknown)
>   Ec2AvailabilityZone: us-east-1d
>   Ec2InstanceType: t2.micro
>   Ec2Kernel: unavailable
>   Ec2Ramdisk: unavailable
>   IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
>   Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to
> initialize libusb: -99
>   MachineType: Xen HVM domU
>   PciMultimedia:
>
>   ProcEnviron:
>    TERM=screen
>    PATH=(custom, no user)
>    LANG=en_US.UTF-8
>    SHELL=/bin/bash
>   ProcFB: 0 xen
>   ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic
> root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1
> console=ttyS0 net.ifnames=0
>   RelatedPackageVersions:
>    linux-restricted-modules-4.2.0-18-generic N/A
>    linux-backports-modules-4.2.0-18-generic  N/A
>    linux-firmware                            N/A
>   RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
>   SourcePackage: linux
>   UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
>   UpgradeStatus: No upgrade log present (probably fresh install)
>   dmi.bios.date: 05/06/2015
>   dmi.bios.vendor: Xen
>   dmi.bios.version: 4.2.amazon
>   dmi.chassis.type: 1
>   dmi.chassis.vendor: Xen
>   dmi.modalias:
> dmi:bvnXen:bvr4.2.amazon:bd05/06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
>   dmi.product.name: HVM domU
>   dmi.product.version: 4.2.amazon
>   dmi.sys.vendor: Xen
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linux/+bug/1518457/+subscriptions
>

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1518457

Title:
  kswapd0 100% CPU usage

Status in Linux:
  Unknown
Status in linux package in Ubuntu:
  Confirmed

Bug description:
  As per bug 721896 and various others:

  I'm on an AWS t2.micro instance (Xeon E5-2670, 991MiB of memory).
  Occasionally (about once a day), kswapd0 falls into a busy loop and
  spins on 100% CPU usage indefinitely. This can be provoked by
  copying/writing large files (e.g. dding a 256MB file), but it happens
  occasionally otherwise. System memory usage (not including
  buffers/caches) currently sits at 36%, which is typical[1]. Initially
  I had no swap space configured; I've since tried enabling a 256MB swap
  file, but the problem continues to occur and no swap space is used.
  The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`.

  Happy to provide further information/take further debugging actions.

  
  [1] Full output from `free`:
               total       used       free     shared    buffers     cached
  Mem:       1014936     483448     531488      28556       9756     112700
  -/+ buffers/cache:     360992     653944
  Swap:       262140          0     262140

  ProblemType: Bug
  DistroRelease: Ubuntu 15.10
  Package: linux-image-4.2.0-18-generic 4.2.0-18.22
  ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
  Uname: Linux 4.2.0-18-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Nov 19 19:40 seq
   crw-rw---- 1 root audio 116, 33 Nov 19 19:40 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.19.1-0ubuntu5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  Date: Fri Nov 20 20:44:30 2015
  Ec2AMI: ami-1c552a76
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: us-east-1d
  Ec2InstanceType: t2.micro
  Ec2Kernel: unavailable
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  MachineType: Xen HVM domU
  PciMultimedia:
   
  ProcEnviron:
   TERM=screen
   PATH=(custom, no user)
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 xen
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-18-generic root=UUID=35bc01f4-4602-4823-976e-508edef899df ro console=tty1 console=ttyS0 net.ifnames=0
  RelatedPackageVersions:
   linux-restricted-modules-4.2.0-18-generic N/A
   linux-backports-modules-4.2.0-18-generic  N/A
   linux-firmware                            N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 05/06/2015
  dmi.bios.vendor: Xen
  dmi.bios.version: 4.2.amazon
  dmi.chassis.type: 1
  dmi.chassis.vendor: Xen
  dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd05/06/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
  dmi.product.name: HVM domU
  dmi.product.version: 4.2.amazon
  dmi.sys.vendor: Xen

To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1518457/+subscriptions