launchpad-reviewers team mailing list archive

["Jeroen T. Vermeulen" <jtv@xxxxxxxxxxxxx>]

Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/import-queue-cleanup-blocked into lp:launchpad/devel.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers): code
Related bugs:
  #609129 Clean up old blocked Ubuntu PO files
  https://bugs.launchpad.net/bugs/609129


= Bug 609129 =

The Translations import queue is big: well over 100K entries.  It's been larger; we're slowly getting it under control while actually importing more and more files.

This branch whittles down the queue by a further 40% or so.  It drops off the queue any uploads for Ubuntu that contain translations (not templates) in the gettext format and that haven't seen any changes in the past year.  These uploads were generally blocked automatically because their corresponding template uploads were manually blocked first.  Since I'm not touching the templates, any future uploads of the same translations are likely to get blocked again—hence there is little or no value to keeping these entries around.

That's for Ubuntu.  I discussed this change with the stakeholders.  The considerations may be different for project uploads, so I'm not touching those.  Similarly, there can be other non-template uploads (only XPI at the moment) but those are few enough that we probably don't have any blocked entries for them.  Even if we did, the extra complexity of blocking those as well would only be worth it if there were a noticeable number of them.

To test,
{{{
./bin/test -vvc -m lp.translations.tests.test_autoapproval -t cleanUp
}}}

You'll notice a few drive-by lint cleanups as well.

Jeroen
-- 
https://code.launchpad.net/~jtv/launchpad/import-queue-cleanup-blocked/+merge/30774
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/import-queue-cleanup-blocked into lp:launchpad/devel.

=== modified file 'lib/lp/translations/model/translationimportqueue.py'
--- lib/lp/translations/model/translationimportqueue.py	2010-07-02 14:23:47 +0000
+++ lib/lp/translations/model/translationimportqueue.py	2010-07-23 13:41:16 +0000
@@ -23,7 +23,7 @@
 from zope.interface import implements
 from zope.component import getUtility
 from sqlobject import SQLObjectNotFound, StringCol, ForeignKey, BoolCol
-from storm.expr import And, Or
+from storm.expr import And, Like, Or
 from storm.locals import Int, Reference
 
 from canonical.database.sqlbase import (
@@ -1307,6 +1307,15 @@
                 TranslationImportQueueEntry.status == status,
                 TranslationImportQueueEntry.date_status_changed < cutoff))
 
+        # Also clean out Blocked PO files for Ubuntu that haven't been
+        # touched for a year.  Keep blocked templates because they may
+        # determine the blocking of future translation uploads.
+        blocked_cutoff = now - datetime.timedelta(days=365)
+        deletion_clauses.append(And(
+            TranslationImportQueueEntry.distroseries_id != None,
+            TranslationImportQueueEntry.date_status_changed < blocked_cutoff,
+            Like(TranslationImportQueueEntry.path, '%.po')))
+
         entries = store.find(
             TranslationImportQueueEntry, Or(*deletion_clauses))
 

=== modified file 'lib/lp/translations/tests/test_autoapproval.py'
--- lib/lp/translations/tests/test_autoapproval.py	2010-07-21 14:02:28 +0000
+++ lib/lp/translations/tests/test_autoapproval.py	2010-07-23 13:41:16 +0000
@@ -824,6 +824,65 @@
                     self._exists(entry_id),
                     "Queue entry in state '%s' was not removed." % status)
 
+    def test_cleanUpObsoleteEntries_blocked_ubuntu_po(self):
+        # _cleanUpObsoleteEntries deletes Ubuntu entries for gettext
+        # translations that are Blocked if they haven't been touched in
+        # a year.  These entries once made up about half the queue.  As
+        # far as we can tell all these PO files have been auto-blocked
+        # after their template uploads were blocked, so even if they
+        # were ever re-uploaded, they'd just get blocked again.
+        entry = self._makeDistroEntry()
+        entry.path = 'fo.po'
+        entry_id = entry.id
+        self._setStatus(entry, RosettaImportStatus.BLOCKED)
+        now = datetime.now(UTC)
+        entry.dateimported = now - timedelta(days=700)
+
+        # It hasn't been a year yet since the last status chang; the
+        # entry stays in place.
+        entry.date_status_changed = now - timedelta(days=300)
+        with self.beingTheGardener():
+            self.queue._cleanUpObsoleteEntries(self.store)
+        self.assertTrue(self._exists(entry_id))
+
+        # A year has passed; the entry gets cleaned up.
+        entry.date_status_changed = now - timedelta(days=400)
+        with self.beingTheGardener():
+            self.queue._cleanUpObsoleteEntries(self.store)
+        self.assertFalse(self._exists(entry_id))
+
+    def test_cleanUpObsoleteEntries_blocked_product_po(self):
+        # _cleanUpObsoleteEntries leaves blocked project uploads in
+        # place.
+        entry = self._makeProductEntry()
+        entry.path = 'fo.po'
+        entry_id = entry.id
+        self._setStatus(entry, RosettaImportStatus.BLOCKED)
+        now = datetime.now(UTC)
+        entry.dateimported = now - timedelta(days=700)
+        entry.date_status_changed = now - timedelta(days=400)
+
+        with self.beingTheGardener():
+            self.queue._cleanUpObsoleteEntries(self.store)
+
+        self.assertTrue(self._exists(entry_id))
+
+    def test_cleanUpObsoleteEntries_blocked_ubuntu_pot(self):
+        # _cleanUpObsoleteEntries leaves blocked Ubuntu templates in
+        # place.
+        entry = self._makeDistroEntry()
+        entry.path = 'foo.pot'
+        entry_id = entry.id
+        self._setStatus(entry, RosettaImportStatus.BLOCKED)
+        now = datetime.now(UTC)
+        entry.dateimported = now - timedelta(days=700)
+        entry.date_status_changed = now - timedelta(days=400)
+
+        with self.beingTheGardener():
+            self.queue._cleanUpObsoleteEntries(self.store)
+
+        self.assertTrue(self._exists(entry_id))
+
     def test_cleanUpInactiveProductEntries(self):
         # After a product is deactivated, _cleanUpInactiveProductEntries
         # will clean up any entries it may have on the queue.