← Back to team overview

kernel-packages team mailing list archive

  1. kernel-packages team
  2. Mailing list archive
  3. Message #155113

[Bug 1532513] [NEW] kswapd 100% CPU usage and running out of memory without swap being used

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This bug closely resembles bug 1518457 and others, but is reported
separately as the affected kernel versions do not match, see comments
#45, #43 and #8 on 1518457.

The issue first occured on a production workload (JIRA running on
Tomcat) on an EC2 t2.small instance after upgrading to 15.10. The issue
happens when running out of memory, typicall during busy hours or when
nightly Python backup script runs. To reproduce the issue, I use
t2.micro (1GiB RAM) because the problem then arises just by starting the
JIRA application. I do not see the problem on 14.04 or 15.05 on the same
server.

Symptoms: kswapd0 falls into a busy loop and spins on 100% CPU usage and
swap not being used/barely being used. The system can be recovered with
`echo 1 > /proc/sys/vm/drop_caches`, but with high traffic/little free
memory, the problem returns immediately.

Swappiness has not been changed (vm.swappiness = 60). I can reproduce
this consitently on Ubuntu AMIs (https://cloud-
images.ubuntu.com/locator/ec2/). I've used AMI ami-c4140aa8 (eu-
central-1). I can provide SSH access to a machine on request and have
public Puppet scripts to configure a new server.

I've attached scripts and logs from an up to date 15.10  and 15.04
instance to compare results with and without the bug. Note that the CPU
numbers in the vmstat output doesn't specify which process, but top has
showed that the majority of user time is the java process and the
majority of system time is the kswapd0 process.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-23-generic 4.2.0-23.28
ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6
Uname: Linux 4.2.0-23-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116,  1 Jan 10 03:56 seq
 crw-rw---- 1 root audio 116, 33 Jan 10 03:56 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Sun Jan 10 04:07:30 2016
Ec2AMI: ami-c4140aa8
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-central-1b
Ec2InstanceType: t2.micro
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Xen HVM domU
PciMultimedia:
 
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 xen
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=56681ffc-93a4-47c9-b487-b67be12205d9 ro console=tty1 console=ttyS0 net.ifnames=0
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-23-generic N/A
 linux-backports-modules-4.2.0-23-generic  N/A
 linux-firmware                            N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/07/2015
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/07/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
dmi.product.name: HVM domU
dmi.product.version: 4.2.amazon
dmi.sys.vendor: Xen

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: amd64 apport-bug ec2-images wily

** Attachment added: "Commands to reproduce on Ubuntu 15.10 and vmstat log"
   https://bugs.launchpad.net/bugs/1532513/+attachment/4547222/+files/commands-15.10.log

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1532513

Title:
  kswapd 100% CPU usage and running out of memory without swap  being
  used

Status in linux package in Ubuntu:
  New

Bug description:
  This bug closely resembles bug 1518457 and others, but is reported
  separately as the affected kernel versions do not match, see comments
  #45, #43 and #8 on 1518457.

  The issue first occured on a production workload (JIRA running on
  Tomcat) on an EC2 t2.small instance after upgrading to 15.10. The
  issue happens when running out of memory, typicall during busy hours
  or when nightly Python backup script runs. To reproduce the issue, I
  use t2.micro (1GiB RAM) because the problem then arises just by
  starting the JIRA application. I do not see the problem on 14.04 or
  15.05 on the same server.

  Symptoms: kswapd0 falls into a busy loop and spins on 100% CPU usage
  and swap not being used/barely being used. The system can be recovered
  with `echo 1 > /proc/sys/vm/drop_caches`, but with high traffic/little
  free memory, the problem returns immediately.

  Swappiness has not been changed (vm.swappiness = 60). I can reproduce
  this consitently on Ubuntu AMIs (https://cloud-
  images.ubuntu.com/locator/ec2/). I've used AMI ami-c4140aa8 (eu-
  central-1). I can provide SSH access to a machine on request and have
  public Puppet scripts to configure a new server.

  I've attached scripts and logs from an up to date 15.10  and 15.04
  instance to compare results with and without the bug. Note that the
  CPU numbers in the vmstat output doesn't specify which process, but
  top has showed that the majority of user time is the java process and
  the majority of system time is the kswapd0 process.

  ProblemType: Bug
  DistroRelease: Ubuntu 15.10
  Package: linux-image-4.2.0-23-generic 4.2.0-23.28
  ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6
  Uname: Linux 4.2.0-23-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jan 10 03:56 seq
   crw-rw---- 1 root audio 116, 33 Jan 10 03:56 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.19.1-0ubuntu5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  Date: Sun Jan 10 04:07:30 2016
  Ec2AMI: ami-c4140aa8
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: eu-central-1b
  Ec2InstanceType: t2.micro
  Ec2Kernel: unavailable
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
  MachineType: Xen HVM domU
  PciMultimedia:
   
  ProcEnviron:
   TERM=xterm-256color
   PATH=(custom, no user)
   XDG_RUNTIME_DIR=<set>
   LANG=en_US.UTF-8
   SHELL=/bin/bash
  ProcFB: 0 xen
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=56681ffc-93a4-47c9-b487-b67be12205d9 ro console=tty1 console=ttyS0 net.ifnames=0
  RelatedPackageVersions:
   linux-restricted-modules-4.2.0-23-generic N/A
   linux-backports-modules-4.2.0-23-generic  N/A
   linux-firmware                            N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 12/07/2015
  dmi.bios.vendor: Xen
  dmi.bios.version: 4.2.amazon
  dmi.chassis.type: 1
  dmi.chassis.vendor: Xen
  dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/07/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
  dmi.product.name: HVM domU
  dmi.product.version: 4.2.amazon
  dmi.sys.vendor: Xen

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1532513/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next