launchpad-dev team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

Hey.  Francis asked us to cc the dev list about this problem, so here we 
are. :-)

Gary

-------- Original Message --------
Subject: Translation statistics background
Date: Wed, 25 Jan 2012 10:02:36 -0500
From: Benji York <benji.york@xxxxxxxxxxxxx>
To: Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>
CC: Gary Poster <gary.poster@xxxxxxxxxxxxx>

The problem we're having is that translation statistics are
intermittently not being updated.  There have been problems with
translation statistics for a while, but this seems related to a recent
change I was tasked with.

The old way: when a translation was changed its statistics (number of
untranslated messages, number of suggestions awaiting approval, number
of translated messages, etc.) would be updated.  That update would often
take longer than the timeout window and would also not update the
statistics for shared translations.  Therefore there was a job to update
the statistics for recently changed translations once a day
(verify-pofile-stats-daily).

The new way: when a translation is changed a record is added to a job
table and every five minutes a job runs to update the statistics of any
translations (and their shared translations) listed in that table.

Here's Jeroen's thoughts on the current issue:

    It working "sometimes" makes me suspect one of these scenarios:

    (a) Routine recalculations for translations that haven't changed
    (background statistics scrubbing, really) never works, e.g. because
    a cron job is not running.  But whenever a translation gets touched,
    its statistics do get updated properly.

    (b) We're missing a database privilege for some loosely-related
    table that isn't always needed for statistics updates.  Something
    related to Distribution, perhaps.

    (c) Things are running, but much too slowly so that we never catch
    up.

    The most important thing is to have a look at logs: do these jobs
    actually run, and if so, successfully?

There have been many, interrelated bugs about translation statistics
recently:

Bug 877195 is the original bug about moving the statistics update out of
web requests.

Bug 894177 was caused by a missing database permission (and missing
test).  Jeroen thinks that something similar is a possible cause of the
current problem (see above).

Bug 903532 arose from the fact that verify-pofile-stats-daily had been
removed from the crontab (by whom I know not) and that I wasn't aware
that bug 886067 wasn't done.

Bug 903532 was about adding the shared translation statistics update
functionality to the new job (superseding bug 886067)

Bug 781274 contains some background information about a now unneeded job
that updated ed all translation statistics once a week, but got to the
point that it took more than a week to run.

These bugs have lots of useful information in them.

--
Benji York