kernel-packages team mailing list archive

[Launchpad Bug Tracker <731340@xxxxxxxxxxxxxxxxxx>]

[Expired for linux (Ubuntu) because there has been no activity for 60
days.]

** Changed in: linux (Ubuntu)
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/731340

Title:
  LUKS is extremely slow on amd64 builds but not on i386

Status in “linux” package in Ubuntu:
  Expired

Bug description:
  TL;DR: Disk I/O with LUKS is extremely slow on amd64 builds, however,
  it is much faster on i386 builds. This has been verified by testing on
  multiple hardware platforms and with multiple versions of Ubuntu. The
  possible cause of the slowness has been shown to somehow be related to
  the architecture.  --- seems to actually be 10.10, since it's slow in
  i386, too, just not quite -as- slow; see comment #5.

  Longer version:
  My friend and I have the same Lenovo T61p laptops. We have the same hardware specs:
   - 2.5 GHz Core 2 Duo
   - 4 GB RAM
   - 80 GB Intel SSD

  We even use the same OS, file systems, etc.:
   - Kubuntu
   - ext4
   - Whole disk encryption with LUKS
   - Disk set up with LVM

  We have one difference though: for a long time, I have been using the
  32 bit version of Kubuntu, my friend has always been using the 64 bit
  version.

  Since we bought our laptops and still used hard discs instead of SSDs, my friends laptop has always been noticable slower with I/O.
  In the beginning we just thought that he had gotten a bad disk. We both soon bought a new 80 GB Intel SSD in order to obtain better I/O performance.
  When we installed on the SSD we both saw a huge increase in I/O performance. However, my friends system was still much slower than mine.
  We thought that it might be due to alignment issues, so my friend researched this subject a lot and made sure that it was perfectly aligned.
  Still, his performance with much poorer than mine.

  We have really tried debugging this in countless ways. I will not even
  try to describe all of it here.

  We both work on some of the same software projects. Thus, we have the same SVN checkouts on our file systems.
  We took one of these checkouts (one with lots of files in it, about 13k files) and used it as reference.
  We then measured how long it took to do a find |wc -l inside the folder. Here are the results:
  On my friends system it took about 90 seconds. On my system it took less than 15 seconds.
  We have repeated this test on several different Ubuntu installations from circa 8.04 up until the most recent 10.04 and we always got the same results.

  Then a few months ago I upgraded to Ubuntu 10.10 and my friend did too. Now, suddenly, I began experiencing the same slowness.
  Now, my computer also takes about 90 seconds to do the find in the same folder.
  This was strange. We feared that maybe our SSDs were failing. We have heard stories about SSDs becoming slow before failing.

  During a discussion about possible causes for this behavior it struck me: the only difference between our systems was the architechture. My friend always use amd64, and I use i386.
  Recently, when I installed Kubuntu 10.10, it chose the 64 bit version, because I no longer had any legacy apps requiring 32 bit.
  It was after this install that my system suddenly began performing badly.

  So we have results like this:
  Time C: Time taken by the find command on my friend's system
  Time T: Time taken by the find command on my system
  Arch C: My friend's architecture
  Arch T: My architecture

  Ubuntu version |  Time C  | Time T |  Arch C |  Arch T
  ---------------+----------+--------+---------+---------
  8.04           |  90 s    |  <15 s |  amd64  |  i386
  8.10           |  90 s    |  <15 s |  amd64  |  i386
  9.04           |  90 s    |  <15 s |  amd64  |  i386
  9.10           |  90 s    |  <15 s |  amd64  |  i386
  10.04          |  90 s    |  <15 s |  amd64  |  i386
  10.10          |  90 s    |   90 s |  amd64  |  amd64

  (I hope this gets formatted correctly. If not, there is a correctly
  formatted version here: http://paste.adora.dk/P1977.html)

  Of course, we made sure to drop cache before testing with the command:
  sync ; echo 3 > /proc/sys/vm/drop_caches

  So the full command we ran was:
  sync ; echo 3 > /proc/sys/vm/drop_caches ; time find $PATH_TO_WC |wc -l

  We also did each test multiple times to make sure we got consistent
  results.

  (I am trying to provide as much data as possible so as to make it
  easier to find and fix the bug)

  In order to try to disprove our hypothesis about the architecture
  making the difference, I tried booting up on a 32 bit Knoppix Live CD
  on my laptop. I then mounted my LUKS encrypted rootfs from the Knoppix
  CD and did a find inside the same folder. Now it took about between 10
  and 15 seconds to do the find.

  OK, so we have found some correlation between the architechture of Ubuntu and the slow I/O.
  There were still several possible places that could be responsible for the problem:
   * LVM
   * LUKS
   * Ext4

  I then wanted to see which one of these that made a difference.
  My /boot partition is not encrypted with LUKS, and I had enough free space available to copy the reference folder to /boot and then did the test there.
  On the /boot partition, it took less than 2 seconds. I tried this both in the installed Kubuntu 10.10 (amd64) and from the 32 bit Knoppix live cd. Both systems took less than 2 seconds.

  Here is the data from the Knoppix tests from encrypted / and non-encrypted /boot:
  http://p.adora.dk/P1978.txt

  It is clear that non-encrypted /boot is much quicker than encrypted /.
  However, all of the tests from the encrypted / partition takes less
  than 15 sec.  This same test takes about 90 sec in amd64 version of
  Ubuntu.

  The results from 64 bit Ubuntu 10.10 are available here:
  http://paste.adora.dk/P1979.txt

  /boot is ext4, so we can eliminate ext4 as the cause of the problem.

  In order to test if LVM is the cause, I created a 100mb file on /boot. I made this file into a loop back device and set up an LVM volume on it.
  I then created a file system inside it and mounted. Then I copied the reference folder to this file system and did the tests again.
  All tests from this LVM volume took less than 2 seconds. So LVM is eliminated as the cause.

  I see the combination of LUKS/amd64 as the only possible cause left.

  I investigated further to check if it was a problem related to LUKS only or if it was also present on ecryptfs. For this test, I copied the reference folder to my ecryptfs encrypted homedir on my netbook (Lenovo S10 with Atom CPU, 2 GB RAM, and a 160 GB hard drive -- no SSD). The tests on the netbook took about 6-7 seconds.
  This was actually pretty fast for a hard drive. So fast that I guess that it is safe to conclude that the problem does not occur on ecryptfs.

  I have now layed out all the data and methology of the investigation. I really hope someone can take the next step and isolate the problem so it can be fixed.
  As it is, I/O performance on 64 bit Ubuntu with LUKS is almost unusably slow.

  My guess is that either the 64 bit or the 32 bit kernel has some kind
  of compile flag or other setting that makes the difference.

  ProblemType: Bug
  DistroRelease: Ubuntu 10.10
  Package: linux-image-2.6.35-25-generic 2.6.35-25.44
  Regression: No
  Reproducible: Yes
  ProcVersionSignature: Ubuntu 2.6.35-25.44-generic 2.6.35.10
  Uname: Linux 2.6.35-25-generic x86_64
  NonfreeKernelModules: nvidia
  AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
  Architecture: amd64
  ArecordDevices:
   **** List of CAPTURE Hardware Devices ****
   card 0: Intel [HDA Intel], device 0: AD198x Analog [AD198x Analog]
     Subdevices: 2/2
     Subdevice #0: subdevice #0
     Subdevice #1: subdevice #1
  AudioDevicesInUse:
   USER        PID ACCESS COMMAND
   /dev/snd/controlC0:  tdn        2187 F.... pulseaudio
  Card0.Amixer.info:
   Card hw:0 'Intel'/'HDA Intel at 0xfe220000 irq 49'
     Mixer name	: 'Analog Devices AD1984'
     Components	: 'HDA:11d41984,17aa20bb,00100400'
     Controls      : 31
     Simple ctrls  : 19
  Card29.Amixer.info:
   Card hw:29 'ThinkPadEC'/'ThinkPad Console Audio Control at EC reg 0x30, fw 7KHT24WW-1.08'
     Mixer name	: 'ThinkPad EC 7KHT24WW-1.08'
     Components	: ''
     Controls      : 1
     Simple ctrls  : 1
  Card29.Amixer.values:
   Simple mixer control 'Console',0
     Capabilities: pswitch pswitch-joined penum
     Playback channels: Mono
     Mono: Playback [on]
  Date: Tue Mar  8 14:10:10 2011
  HibernationDevice: RESUME=UUID=bd03b060-e813-403f-b6c0-dbd58049fbfa
  InstallationMedia: Kubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
  MachineType: LENOVO 6460D8G
  PccardctlIdent:
   Socket 0:
     no product info available
  PccardctlStatus:
   Socket 0:
     no card
  ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.35-25-generic root=/dev/mapper/hostname-root ro quiet splash
  ProcEnviron:
   LANGUAGE=
   PATH=(custom, user)
   LANG=en_DK.UTF-8
   SHELL=/bin/zsh
  RelatedPackageVersions: linux-firmware 1.38.4
  SourcePackage: linux
  WifiSyslog:

  dmi.bios.date: 11/14/2008
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7LETC5WW (2.25 )
  dmi.board.name: 6460D8G
  dmi.board.vendor: LENOVO
  dmi.board.version: Not Available
  dmi.chassis.asset.tag: No Asset Information
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Not Available
  dmi.modalias: dmi:bvnLENOVO:bvr7LETC5WW(2.25):bd11/14/2008:svnLENOVO:pn6460D8G:pvrThinkPadT61p:rvnLENOVO:rn6460D8G:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
  dmi.product.name: 6460D8G
  dmi.product.version: ThinkPad T61p
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/731340/+subscriptions