launchpad-reviewers team mailing list archive

[Ines Almeida <mp+475757@xxxxxxxxxxxxxxxxxx>]

Ines Almeida has proposed merging ~ines-almeida/launchpad:fix-garbo-daily-not-finishing-run into launchpad:master.

Commit message:
Reduce the max chunk size for ScrubPOFileTranslator garbo daily
    
This job has been consantly having long running transactions that take hours. Reducing the size of the chunk to see if it makes transaction.commit() more manageable

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~ines-almeida/launchpad/+git/launchpad/+merge/475757

See https://warthogs.atlassian.net/browse/LP-2045 for more details.

Logs `scripts.lp.internal::lp-logs/garbo-daily.log` show constant long running transactions in this daily garbo job.

We might still need more investigation to understand why, but as a simple first stab at it:
 - reducing the chunk size so that we commit smaller transactions
 - adding logs to see how many items we are processing each night (might give a clue about whether we process large chunks each evening, or if there is some other issue going on if not)
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~ines-almeida/launchpad:fix-garbo-daily-not-finishing-run into launchpad:master.

diff --git a/lib/lp/translations/scripts/scrub_pofiletranslator.py b/lib/lp/translations/scripts/scrub_pofiletranslator.py
index ee45697..34f8b91 100644
--- a/lib/lp/translations/scripts/scrub_pofiletranslator.py
+++ b/lib/lp/translations/scripts/scrub_pofiletranslator.py
@@ -275,11 +275,15 @@ def process_work_items(logger, work_items, pofiles):
 class ScrubPOFileTranslator(TunableLoop):
     """Tunable loop, meant for running from inside Garbo."""
 
-    maximum_chunk_size = 5000
+    maximum_chunk_size = 2500
 
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.pofile_ids = tuple(get_pofile_ids())
+        self.log.info(
+            f"[ScrubPOFileTranslator] Found {len(self.pofile_ids)} "
+            "PO Files to process."
+        )
         self.next_offset = 0
 
     def __call__(self, chunk_size):