kernel-packages team mailing list archive

Thread
Date

[Bug 1159258] Re: The IOSCHED default of deadline combined with 4k page size causes poor performance in lightly configured systems.

To: kernel-packages@xxxxxxxxxxxxxxxxxxx
From: "Christopher M. Penalver" <christopher.m.penalver@xxxxxxxxx>
Date: Fri, 03 Jan 2014 12:52:00 -0000
Reply-to: Bug 1159258 <1159258@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

** Tags removed: iosched kernel page performance size
** Tags added: latest-bios-a06 needs-upstream-testing regression-potential

** Tags added: bot-stop-nagging

** Description changed:

+ WORKAROUND: Set hugepage to use a 2M page size, and utilize kernel boot parameters:
+ levator=cfq transparent_hugepage=always
+
I have a Dell Latitude D-420 notebook which has behaved miserably since
sometime after I upgraded to Kubuntu 12.10.

I would be the first to admit that it's an older machine, however, the
drop in performance was far to great and too rapid to make sense. I
went so far as to take the machine apart thinking it was full of dust,
as it started running hot enough to run the fan at top speed. It was
almost dust free.

It is configured with an Intel dual core U2550, 1.2GHz processor, 2GB of
ram, and a 1.8" 4200RPM PATA ZIF HDD. I wondered if the weak link was
the hard drive. I borrowed a 128GB SSD and there was almost no
improvement.

Activity as minimal as opening Google Chrome with two or three simple
tabs and leaving it to sit with Akonadi and Nepomuk, to run in the back
ground was enough to make it impossible to watch a full screen MP4
video. This is a far cry from a few months ago when I could have a
dozen Chrome windows open, Kontact running, watching a full screen MP4
with no loss in quality, all while downloading at a speed which
saturated my DSL connection.

What was even stranger was it was only moderately swapping out.

From there, I decided to investigate. The load average was regularly
around 3. User load was only responsible for roughly 20% of an almost
constant 100% CPU load. Combining this with the fact that swap used
wasn't all that high, I started to dig a little deeper.

Interestingly, both regular memory usage AND buffer allocation were
constantly high, with buffer allocation around 300MB. Disk cache
hovered around a paltey 100MB. The cache didn't change by increasing
vm.swappiness to 100, either. On top of it all, the Intel iwl3945
wireless adapter was loosing packets and constantly having to
renegotiate WPA.

I ran vmstat for a while. Context switching was occurring roughly 1500
per second along with about 3000 software interrupts per second. From
here, I looked into IOSCHED. As it turns out, linux-
image-3.5.0-26-generic is configured to use the deadline scheduler, by
default, as shown in /boot/config-3.5.0-26-generic:

CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"

Switching to CFQ improved the situation immensely, however, I was still
getting far more kernel time using up the CPU, and the context switching
was far too high.

I happened to come across documentation on the transparent hugepage
system while I was reading up on IOSCHED and it hit me. For some reason
the kernel was probably bogged down in memory fragmentation and huge
page tables due to the 4k page side and the pressures of deadline
scheduling with a rather slow hard drive. The hugepage system seems to
do a wonderful job. I have it set to use a 2M page size. The system,
itself, uses almost know resources, with khugepaged clocking in an
insignificant 5:00m of CPU time in roughly six hours.

My poor old computer is now running as fast or faster then before this
issue occurred. Switching between intensive applications such as having
tens of Javascript heavy tabs like Facebook open in Chrome, bittorrent
maxed out at about 500kB/s while using only encrypted connections, and
watching a full screen MP4 flawlessly, rarely saturates CPU usage. The
CPU is no longer mostly all kernel and wait time, either. Context
switching has settled to a reasonable 500 to 700 switches per second.
Buffer allocations are rarely higher than 25MB. As a bonus, the memory
allocated for page tables has dropped from around 80MB to 20MB. Best of
all, my fan is now back on low, and I can use the notebook without the
risk of burns!

I realize that an aggressive scheduling approach is probably the way to
go with new and resource rich hardware. however, with lightly
provisioned systems it seems to create the perfect storm. After all,
this notebook is still quite a bit more powerful than a recent tablet.
If there is a wish to see Ubuntu running on hardware such as that,
something will have to be done.

It could be as easy as a small script to assess resources, perhaps with
each kernel update. and configure a more suitable default for these
lighter systems.

Cheers,
Jonathan Skanes
- ---
+ ---
ApportVersion: 2.6.1-0ubuntu10
Architecture: i386
AudioDevicesInUse:
- USER PID ACCESS COMMAND
- /dev/snd/controlC0: jon 2267 F.... pulseaudio
+ USER PID ACCESS COMMAND
+ /dev/snd/controlC0: jon 2267 F.... pulseaudio
DistroRelease: Ubuntu 12.10
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=d1980c70-8ee4-4f44-a4d3-fa1e12b2b123
InstallationDate: Installed on 2009-12-28 (1181 days ago)
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
MachineType: Dell Inc. Latitude D420
MarkForUpload: True
Package: linux (not installed)
PccardctlIdent:
- Socket 0:
- no product info available
+ Socket 0:
+ no product info available
PccardctlStatus:
- Socket 0:
- no card
+ Socket 0:
+ no card
ProcEnviron:
- LANGUAGE=en_CA
- TERM=xterm
- PATH=(custom, no user)
- LANG=en_CA.UTF-8
- SHELL=/bin/bash
+ LANGUAGE=en_CA
+ TERM=xterm
+ PATH=(custom, no user)
+ LANG=en_CA.UTF-8
+ SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-26-generic root=UUID=60cdd746-3be0-4d48-a03f-45ba7381db4f ro elevator=cfq transparent_hugepage=always quiet
ProcVersionSignature: Ubuntu 3.5.0-26.42-generic 3.5.7.6
RelatedPackageVersions:
- linux-restricted-modules-3.5.0-26-generic N/A
- linux-backports-modules-3.5.0-26-generic N/A
- linux-firmware 1.95
+ linux-restricted-modules-3.5.0-26-generic N/A
+ linux-backports-modules-3.5.0-26-generic N/A
+ linux-firmware 1.95
Tags: quantal
Uname: Linux 3.5.0-26-generic i686
UpgradeStatus: Upgraded to quantal on 2012-10-29 (145 days ago)
UserGroups: adm admin cdrom dialout kismet lp lpadmin plugdev pulse-rt sage sambashare tilp vboxusers wireshark
dmi.bios.date: 02/02/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A06
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA06:bd02/02/2008:svnDellInc.:pnLatitudeD420:pvr:rvnDellInc.:rn:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: Latitude D420
dmi.sys.vendor: Dell Inc.

** Tags removed: regression-potential
** Tags added: regression-release

** Summary changed:

- The IOSCHED default of deadline combined with 4k page size causes poor performance in lightly configured systems.
+ [Dell Latitude D420] The IOSCHED default of deadline combined with 4k page size causes poor performance

** Changed in: linux (Ubuntu)
Importance: Medium => Low

--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1159258

Title:
[Dell Latitude D420] The IOSCHED default of deadline combined with 4k
page size causes poor performance

Status in “linux” package in Ubuntu:
Confirmed

Bug description:
WORKAROUND: Set hugepage to use a 2M page size, and utilize kernel boot parameters:
levator=cfq transparent_hugepage=always

I have a Dell Latitude D-420 notebook which has behaved miserably
since sometime after I upgraded to Kubuntu 12.10.

I would be the first to admit that it's an older machine, however,
the drop in performance was far to great and too rapid to make sense.
I went so far as to take the machine apart thinking it was full of
dust, as it started running hot enough to run the fan at top speed.
It was almost dust free.

It is configured with an Intel dual core U2550, 1.2GHz processor, 2GB
of ram, and a 1.8" 4200RPM PATA ZIF HDD. I wondered if the weak link
was the hard drive. I borrowed a 128GB SSD and there was almost no
improvement.

Activity as minimal as opening Google Chrome with two or three simple
tabs and leaving it to sit with Akonadi and Nepomuk, to run in the
back ground was enough to make it impossible to watch a full screen
MP4 video. This is a far cry from a few months ago when I could have
a dozen Chrome windows open, Kontact running, watching a full screen
MP4 with no loss in quality, all while downloading at a speed which
saturated my DSL connection.

What was even stranger was it was only moderately swapping out.

I ran vmstat for a while. Context switching was occurring roughly
1500 per second along with about 3000 software interrupts per second.
From here, I looked into IOSCHED. As it turns out, linux-
image-3.5.0-26-generic is configured to use the deadline scheduler, by
default, as shown in /boot/config-3.5.0-26-generic:

Switching to CFQ improved the situation immensely, however, I was
still getting far more kernel time using up the CPU, and the context
switching was far too high.

I happened to come across documentation on the transparent hugepage
system while I was reading up on IOSCHED and it hit me. For some
reason the kernel was probably bogged down in memory fragmentation and
huge page tables due to the 4k page side and the pressures of deadline
scheduling with a rather slow hard drive. The hugepage system seems
to do a wonderful job. I have it set to use a 2M page size. The
system, itself, uses almost know resources, with khugepaged clocking
in an insignificant 5:00m of CPU time in roughly six hours.

My poor old computer is now running as fast or faster then before this
issue occurred. Switching between intensive applications such as
having tens of Javascript heavy tabs like Facebook open in Chrome,
bittorrent maxed out at about 500kB/s while using only encrypted
connections, and watching a full screen MP4 flawlessly, rarely
saturates CPU usage. The CPU is no longer mostly all kernel and wait
time, either. Context switching has settled to a reasonable 500 to
700 switches per second. Buffer allocations are rarely higher than
25MB. As a bonus, the memory allocated for page tables has dropped
from around 80MB to 20MB. Best of all, my fan is now back on low, and
I can use the notebook without the risk of burns!

I realize that an aggressive scheduling approach is probably the way
to go with new and resource rich hardware. however, with lightly
provisioned systems it seems to create the perfect storm. After all,
this notebook is still quite a bit more powerful than a recent tablet.
If there is a wish to see Ubuntu running on hardware such as that,
something will have to be done.

It could be as easy as a small script to assess resources, perhaps
with each kernel update. and configure a more suitable default for
these lighter systems.

Cheers,
Jonathan Skanes
---
ApportVersion: 2.6.1-0ubuntu10
Architecture: i386
AudioDevicesInUse:
USER PID ACCESS COMMAND
/dev/snd/controlC0: jon 2267 F.... pulseaudio
DistroRelease: Ubuntu 12.10
EcryptfsInUse: Yes
HibernationDevice: RESUME=UUID=d1980c70-8ee4-4f44-a4d3-fa1e12b2b123
InstallationDate: Installed on 2009-12-28 (1181 days ago)
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release i386 (20091028.5)
MachineType: Dell Inc. Latitude D420
MarkForUpload: True
Package: linux (not installed)
PccardctlIdent:
Socket 0:
no product info available
PccardctlStatus:
Socket 0:
no card
ProcEnviron:
LANGUAGE=en_CA
TERM=xterm
PATH=(custom, no user)
LANG=en_CA.UTF-8
SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-26-generic root=UUID=60cdd746-3be0-4d48-a03f-45ba7381db4f ro elevator=cfq transparent_hugepage=always quiet
ProcVersionSignature: Ubuntu 3.5.0-26.42-generic 3.5.7.6
RelatedPackageVersions:
linux-restricted-modules-3.5.0-26-generic N/A
linux-backports-modules-3.5.0-26-generic N/A
linux-firmware 1.95
Tags: quantal
Uname: Linux 3.5.0-26-generic i686
UpgradeStatus: Upgraded to quantal on 2012-10-29 (145 days ago)
UserGroups: adm admin cdrom dialout kismet lp lpadmin plugdev pulse-rt sage sambashare tilp vboxusers wireshark
dmi.bios.date: 02/02/2008
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A06
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA06:bd02/02/2008:svnDellInc.:pnLatitudeD420:pvr:rvnDellInc.:rn:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: Latitude D420
dmi.sys.vendor: Dell Inc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1159258/+subscriptions