ac100 team mailing list archive

[Max Polk <maxpolk@xxxxxxxxx>]

A weekly fatal unable to fork error message from /etc/cron.weekly/apt-
xapian-index on a server with 512MB memory with a web server and
database server running leads to this bug. That cron job calls /usr/sbin
/update-apt-xapian-index, which is a Python script that imports axi and
axi.indexer.  Those Python packages are the heart of the problem.

On an idle system with 512M memory and 768M swap, while /etc/cron.weekly
/apt-xapian-index was running, I used free to see memory and swap used.
It rose to then peaked at this:

       total   used   free
Mem:  503528 497736   5792
Swap: 786428 231196 555232

Total used memory is 728M.

After ending, it immediately dropped back to this:

       total   used   free
Mem:  503528 259160 244368
Swap: 786428 224484 561944

Total used memory is 483M.

I conclude that apt-xapian-index consumes the difference, which is 245M.

Running "apt-cache stats" I see at the end "Total space accounted for:
26.0 M".

Therefore, it takes 245M to sort and index 26M of information. This
seems conclusive that the algorithms, containers, and/or functions
chosen are very inefficient. It should definitely not require 10 times
the memory space of what is being indexed.

The solution is to change the sorting algorithm in the python axi and
axi.indexer modules. The first priority is to switch to an algorithm
that consumes a whole lot less memory (i.e., each step of the algorithm
keeps less objects in memory), and it will stop crashing and stop
thrashing (memory swapping to disk).

The second priority is of lesser importance (because renice can solve a
lot of the effect), which is to switch to an algorithm that takes a lot
less time to run (i.e., takes fewer steps to complete), and it will stop
consuming so much CPU for so long.

-- 
You received this bug notification because you are a member of AC100
Team, which is subscribed to AC100_enablement.
https://bugs.launchpad.net/bugs/655831

Title:
  update-apt-xapian-index bogs down system

Status in Enablement project for the Toshiba AC100 NetBook:
  New
Status in Apt Xapian Index:
  New
Status in “apt-xapian-index” package in Ubuntu:
  Triaged

Bug description:
  Binary package hint: apt-xapian-index

  This is related to bug 363695 but is not a dupe - that has been
  "fixed".

  every time the update-apt-xapian-index occurs, the system crawls.
  Possibly related to the I/O responsiveness bug of bug 131094.

  The process is extremely intrusive, taking simple things such as flash
  video and throwing the performance down the toilet.

  While this apport bug was collected via a Lucid machine this also
  affects maverick.

  ProblemType: Bug
  DistroRelease: Ubuntu 10.04
  Package: apt-xapian-index 0.25ubuntu2
  ProcVersionSignature: Ubuntu 2.6.32-25.44-generic 2.6.32.21+drm33.7
  Uname: Linux 2.6.32-25-generic i686
  NonfreeKernelModules: nvidia
  Architecture: i386
  Date: Wed Oct  6 12:32:35 2010
  InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100429)
  PackageArchitecture: all
  ProcEnviron:
   LANG=en_US.utf8
   SHELL=/bin/bash
  SourcePackage: apt-xapian-index

To manage notifications about this bug go to:
https://bugs.launchpad.net/ac100/+bug/655831/+subscriptions