launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad/publish-distro-many-ppas into lp:launchpad.

Commit message:
Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/publish-distro-many-ppas/+merge/289925

Make publish-distro use an iterator to walk over PPAs, rather than materialising them up-front.

I'd noticed that commits were very slow on each individual archive when operating on all PPAs.  This turns out to be because of accidentally quadratic behaviour: getTargetArchives materialised all archives into memory up-front, which meant that they were all alive as far as the Storm store was concerned, which meant that the pre-commit invalidate call had to walk over them all.

Turning this into an iterator makes things behave much better: commit still takes a variable amount of time depending on how many archives publish-distro inspected and skipped before it had to do any real work, but the total time spent is linear in len(PublishDistro.getPPAs()) rather than linear in len(PublishDistro.getPPAs()) * len(archives that need publishing work).  It's also less annoying when running publish-distro by hand because it starts producing output more quickly.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~cjwatson/launchpad/publish-distro-many-ppas into lp:launchpad.

=== modified file 'lib/lp/archivepublisher/scripts/publishdistro.py'
--- lib/lp/archivepublisher/scripts/publishdistro.py	2016-03-21 15:40:55 +0000
+++ lib/lp/archivepublisher/scripts/publishdistro.py	2016-03-23 15:20:08 +0000
@@ -7,6 +7,7 @@
     'PublishDistro',
     ]
 
+from itertools import ifilter
 from optparse import OptionValueError
 
 from zope.component import getUtility
@@ -251,9 +252,9 @@
         if self.options.partner:
             return [distribution.getArchiveByComponent('partner')]
         elif self.options.ppa:
-            return filter(is_ppa_public, self.getPPAs(distribution))
+            return ifilter(is_ppa_public, self.getPPAs(distribution))
         elif self.options.private_ppa:
-            return filter(is_ppa_private, self.getPPAs(distribution))
+            return ifilter(is_ppa_private, self.getPPAs(distribution))
         elif self.options.copy_archive:
             return self.getCopyArchives(distribution)
         else: