launchpad-reviewers team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

Aaron Bentley has proposed merging lp:~abentley/launchpad/translation-packaging-resource into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  Bug #781038 in Launchpad itself: "run_jobs.py packaging_translations taking > 2hours and consuming 40% of available RAM"
  https://bugs.launchpad.net/launchpad/+bug/781038

For more details, see:
https://code.launchpad.net/~abentley/launchpad/translation-packaging-resource/+merge/60832

= Summary =
Fix bug #781038: run_jobs.py packaging_translations taking > 2hours and consuming 40% of available RAM

== Proposed fix ==
Periodically commit the transaction to allow cached objects to be released.

== Pre-implementation notes ==
Discussed with deryck.

== Implementation details ==
For TranslationMergeJob, it was simply a matter of restoring the existing functionality.  TransactionManager(None) prevented commits.  Using TransactionManager(transaction.manager) enabled committing.

For TranslationSplitJob, added new code to commit transaction every 100 POTMsgSets.

In both cases, added logging to make future debugging easier.

== Tests ==
None

== Demo and Q/A ==
None


= Launchpad lint =

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/translations/utilities/translationsplitter.py
  lib/lp/translations/model/translationpackagingjob.py
-- 
https://code.launchpad.net/~abentley/launchpad/translation-packaging-resource/+merge/60832
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~abentley/launchpad/translation-packaging-resource into lp:launchpad.

=== modified file 'lib/lp/translations/model/translationpackagingjob.py'
--- lib/lp/translations/model/translationpackagingjob.py	2011-03-23 16:28:51 +0000
+++ lib/lp/translations/model/translationpackagingjob.py	2011-05-12 20:33:36 +0000
@@ -12,11 +12,13 @@
     'TranslationSplitJob',
     ]
 
+import logging
 
 from lazr.lifecycle.interfaces import (
     IObjectCreatedEvent,
     IObjectDeletedEvent,
     )
+import transaction
 from zope.interface import (
     classProvides,
     implements,
@@ -85,9 +87,16 @@
 
     def run(self):
         """See `IRunnableJob`."""
+        logger = logging.getLogger()
         if not self.distroseries.distribution.full_functionality:
+            logger.warning(
+                'Skipping merge for unsupported distroseries "%s".' %
+                self.distroseries.displayname)
             return
-        tm = TransactionManager(None, False)
+        logger.info(
+            'Merging %s and %s', self.productseries.displayname,
+            self.sourcepackage.displayname)
+        tm = TransactionManager(transaction.manager, False)
         TranslationMerger.mergePackagingTemplates(
             self.productseries, self.sourcepackagename, self.distroseries, tm)
 
@@ -103,4 +112,8 @@
 
     def run(self):
         """See `IRunnableJob`."""
+        logger = logging.getLogger()
+        logger.info(
+            'Splitting %s and %s', self.productseries.displayname,
+            self.sourcepackage.displayname)
         TranslationSplitter(self.productseries, self.sourcepackage).split()

=== modified file 'lib/lp/translations/utilities/translationsplitter.py'
--- lib/lp/translations/utilities/translationsplitter.py	2011-02-23 16:23:12 +0000
+++ lib/lp/translations/utilities/translationsplitter.py	2011-05-12 20:33:36 +0000
@@ -4,7 +4,10 @@
 __metaclass__ = type
 
 
+import logging
+
 from storm.locals import ClassAlias, Store
+import transaction
 
 from lp.translations.model.potemplate import POTemplate
 from lp.translations.model.translationtemplateitem import (
@@ -81,6 +84,11 @@
 
     def split(self):
         """Split the translations for the ProductSeries and SourcePackage."""
-        for upstream_item, ubuntu_item in self.findShared():
+        logger = logging.getLogger()
+        shared = enumerate(self.findShared(), 1)
+        for num, (upstream_item, ubuntu_item) in shared:
             self.splitPOTMsgSet(ubuntu_item)
             self.migrateTranslations(upstream_item.potmsgset, ubuntu_item)
+            if num % 100 == 0:
+                logger.info('%d entries split.  Committing...', num)
+                transaction.commit()