yellow team mailing list archive

[Benji York <benji.york@xxxxxxxxxxxxx>]

Hi guys.  Today's installment in things I learned today that you will
want to know if I get abducted by aliens is all about translation
statistics.

Bug 903532 arises from the fact that verify-pofile-stats-daily has been
removed from the crontab (by whom I know not) and that I wasn't aware
that bug 886067 wasn't done.

To fix bug 903532 in the short term we can reinstate
verify-pofile-stats-daily and perhaps even run a tweaked version of it
with a bigger window to fix the translation statistics that have been
out of whack for more than a week (it's current window is 7 days).

I (or you, if the aliens have their way) will have a
LOSA, er WebOp(s) add it back tomorrow.  I might be nice to know why it was
removed so it's not removed again.  (We'll need it even after bug 886067
is done because of some unnamed database wind that I'm not totally clear
on).

As for how to actually fix bug 886067, here's the conversation jtv and I
had about that:

16:09 < jtv:#launchpad-dev> benji: that would only apply if the bug it
refers to had been fully fixed.
16:09 < jtv:#launchpad-dev> Instead, we now have a partial implementation.
16:10 < jtv:#launchpad-dev> What the bug specified (with a bit of
excessive detail that I would argue with, which I'll skip over for
now) is “do the statistics updates in a separate job, and have each
job update not just a single POFile but its entire sharing set.”
16:10 < jtv:#launchpad-dev> The latter part is not implemented AFAICS.
16:11 < jtv:#launchpad-dev> Which leaves us without the biggest thing
the statistics script did for us: patch up statistics for POFiles that
were affected by changes in shared translations.
16:11 < benji:#launchpad-dev> jtv: gotcha, that leads me to the second
half; I can't say that I completely understand the details of how the
shared translations need to be updated when an individual POFile is
changed; will you outline that for me?
16:12 < jtv:#launchpad-dev> It's not so much that shared translations
need to be updated, but that they _are_ updated and the statistics
need to reflect that.
16:12 < jtv:#launchpad-dev> A template can share translations with a
bunch of other templates.
16:13 < jtv:#launchpad-dev> Therefore, a POFile can share translations
with other POFiles: those whose templates share with its template,
translated to the same language.
16:13 < jtv:#launchpad-dev> Say POFile A shares with POFile B.
16:13 < jtv:#launchpad-dev> They are both completely untranslated.
16:14 < jtv:#launchpad-dev> Going through the UI for POFile  A, I
translate POTMsgSet x.
16:14 < jtv:#launchpad-dev> But x is also shared with B's template.
16:14 < jtv:#launchpad-dev> We always updated statistics for A at this point,
16:14 < jtv:#launchpad-dev> but B obviously also needs its statistics
updated because it is no longer completely untranslated.
16:15 < jtv:#launchpad-dev> There was no code to do that immediately;
too complex and too costly.
16:15 < jtv:#launchpad-dev> Instead, we relied on the statistics
updater that was already in place.
16:16 < jtv:#launchpad-dev> And so, in the current design, the
pofilestatsjob needs to look up a full set of sharing POFiles and
update all their statistics.
16:16 < jtv:#launchpad-dev> In my opinion the current design is not
that great for template imports, where we'd basically have to look up
the sharing set twice (and once in the critical path).  But that's
another topic.
16:17 < jtv:#launchpad-dev> benji: still with me?
16:18 < benji:#launchpad-dev> jtv: I have a glimmer of understanding
of the words you are saying.  Do you know of a place in the code which
we already "full set of sharing POFiles"?
16:18 < jtv:#launchpad-dev> You're hitting a sore spot there.
16:18 < jtv:#launchpad-dev> No, we don't have that.
16:18 < jtv:#launchpad-dev> What we have is code to look up a full set
of sharing templates.
16:19 < jtv:#launchpad-dev> So you'd have to go over the full set of
sharing templates (POTemplateSharingSubset IIRC), and for each, look
up their POFile for the language you're interested in.
16:20 < jtv:#launchpad-dev> Also, the template import code needs to do
this for all languages that any of the sharing templates are
translated to.  Which is why I would have preferred for POFileStatsJob
to refer to a POTemplate and an optional language, rather than a
POFile.
16:20 < jtv:#launchpad-dev> A POFile is basically a tuple of a
POTemplate and a language.  But who's to say there's always a POFile?
16:23  * benji considers buying a very large whiteboard.
16:26 < benji:#launchpad-dev> jtv: so... it looks like I need to make
the job use POTemplateSharingSubset.getSharingPOTemplates(POFile_this_job_is_about.template)
to get a set of templates and then call updateStatistics on each of
those templates
16:27  * benji crosses his fingers.
16:28 < jtv:#launchpad-dev> benji: you'll be happy to hear that that
is almost entirely correct.  The only small missing detail is in what
you call updateStatistics on:
16:28 < jtv:#launchpad-dev> for each of those sharing templates you
need to find the POFile (if any) that translates it to
POFile_this_job_is_about.language.  And if there is one, call
updateStatistics on that POFile.
16:28 < benji:#launchpad-dev> oh! yeah; I need to go from the
templates to... /some/ set of POFiles, right?
16:29 < jtv:#launchpad-dev> benji: yes, and tragically there's no
direct support for that.  Not that it'll be _very_ hard, but still.
16:32 < benji:#launchpad-dev> jtv: thanks much

-- 
Benji York