launchpad-reviewers team mailing list archive

["Jeroen T. Vermeulen" <jtv@xxxxxxxxxxxxx>]

The proposal to merge lp:~jtv/launchpad/bug-994650-scrub-faster into lp:launchpad has been updated.

Description changed to:

The garbo job I'm building to update POFileTranslator records for us was painfully slow.  William pointed out the main reason: the query that fetched the POFiles that the outer loop iterates over.  Each time the outer loop takes little slices from that query's result, Storm re-executes the query.

So we needed to keep that list in memory, but again as William pointed out, there are rather too many POFiles to keep comfortably in memory.  This branch loads the POFiles' ids up front, and then postpones the loading of further details for as long as possible.  It does mean handing ids around for some things that were objects before

In fact, you see three levels of POFile retrieval here:

1. Setup fetches only ids, so that only the ids have to be kept in memory.

2. Per batch, the loop loads just the details needed to find out if the POFile needs updating: its POTemplate id, its Language id, and its POTMsgSet ids.

3. Any POFile that actually needs fixing (hopefully a minority) gets loaded as a proper object.

This kind of thing is never fast enough, so there are obvious optimizations that I may pursue next: cache the POTMsgSet ids as we iterate over POFiles of the same POTemplate, to help the no-change case even more.  And to speed up the case where there are changes, such as on the initial run, first gather those POFiles in the batch that need fixing, and batch-load them.  The batch-loading can also cover templates, languages, distroseries, distributions, productseries, and products.  Most of these are needed only for the purpose of logging which POFile is being updated.

To test:
{{{
# Get database schema set up for this branch.
bzr branch lp:~jtv/launchpad/db-994410
cd db-994410
make schema

cd ..

# Get the prerequisite branch.
bzr branch lp:~jtv/launchpad/bug-994650-scrub-pofiletranslator jtv-scrubber
cd jtv-scrubber

# Merge in the branch under review here.
bzr merge lp:~jtv/launchpad/bug-994650-scrub-faster
make

# Test.
./bin/test -vvc lp.translations.scripts.tests.test_scrub_pofiletranslator
}}}


Alternatively, get the branch that combines all these changes:

{{{
bzr branch lp:~jtv/launchpad/combined-async-pofiletranslator
cd combined-async-pofiletranslator
make
make schema
./bin/test -vvc lp.translations.scripts.tests.test_scrub_pofiletranslator
}}}


Jeroen

For more details, see:
https://code.launchpad.net/~jtv/launchpad/bug-994650-scrub-faster/+merge/105169
-- 
https://code.launchpad.net/~jtv/launchpad/bug-994650-scrub-faster/+merge/105169
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/bug-994650-scrub-faster into lp:launchpad.