launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad/optimise-publish-a2 into lp:launchpad.

Commit message:
Optimise phase A2 of the publisher by searching for to-be-deleted source and binary publications in one query each, rather than one per active distro(arch)series and pocket.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/optimise-publish-a2/+merge/226832

Optimise phase A2 of the publisher by searching for to-be-deleted source and binary publications in one query each, rather than one per active distro(arch)series and pocket.

I haven't tried this all the way through on dogfood as yet, but I've run the main binary query here and it runs in a second or so rather than nearly a minute.
-- 
https://code.launchpad.net/~cjwatson/launchpad/optimise-publish-a2/+merge/226832
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~cjwatson/launchpad/optimise-publish-a2 into lp:launchpad.

=== modified file 'lib/lp/archivepublisher/publishing.py'
--- lib/lp/archivepublisher/publishing.py	2014-07-15 00:38:40 +0000
+++ lib/lp/archivepublisher/publishing.py	2014-07-15 14:05:25 +0000
@@ -449,43 +449,51 @@
         # publications that are waiting to be deleted.  Each tuple is
         # added to the dirty_pockets set.
 
-        # Loop for each pocket in each distroseries:
-        for distroseries in self.distro.series:
-            for pocket in self.archive.getPockets():
-                if (self.cannotModifySuite(distroseries, pocket)
-                    or not self.isAllowed(distroseries, pocket)):
-                    # We don't want to mark release pockets dirty in a
-                    # stable distroseries, no matter what other bugs
-                    # that precede here have dirtied it.
-                    continue
-                conditions = base_conditions(SourcePackagePublishingHistory)
-                conditions.extend([
-                    SourcePackagePublishingHistory.pocket == pocket,
-                    SourcePackagePublishingHistory.distroseries ==
-                        distroseries,
-                    ])
-
-                # Make the source publications query.
-                sources = IStore(SourcePackagePublishingHistory).find(
-                    SourcePackagePublishingHistory, *conditions)
-                if not sources.is_empty():
-                    self.markPocketDirty(distroseries, pocket)
-                    # No need to check binaries if the pocket is already
-                    # dirtied from a source.
-                    continue
-
-                # Make the binary publications query.
-                conditions = base_conditions(BinaryPackagePublishingHistory)
-                conditions.extend([
-                    BinaryPackagePublishingHistory.pocket == pocket,
-                    BinaryPackagePublishingHistory.distroarchseriesID ==
-                        DistroArchSeries.id,
-                    DistroArchSeries.distroseries == distroseries,
-                    ])
-                binaries = IStore(BinaryPackagePublishingHistory).find(
-                    BinaryPackagePublishingHistory, *conditions)
-                if not binaries.is_empty():
-                    self.markPocketDirty(distroseries, pocket)
+        # Make the source publications query.
+        all_sources = IStore(SourcePackagePublishingHistory).find(
+            SourcePackagePublishingHistory,
+            *base_conditions(SourcePackagePublishingHistory))
+        all_sources.order_by(
+            SourcePackagePublishingHistory.distroseriesID,
+            SourcePackagePublishingHistory.pocket)
+
+        # Make the binary publications query.
+        conditions = base_conditions(BinaryPackagePublishingHistory)
+        conditions.extend([
+            BinaryPackagePublishingHistory.distroarchseriesID ==
+                DistroArchSeries.id,
+            DistroArchSeries.distroseriesID == DistroSeries.id,
+            ])
+        all_binaries = IStore(BinaryPackagePublishingHistory).find(
+            BinaryPackagePublishingHistory, *conditions)
+        all_binaries.order_by(
+            DistroSeries.id,
+            BinaryPackagePublishingHistory.pocket,
+            DistroArchSeries.architecturetag)
+
+        for (distroseries, pocket), sources in groupby(
+                all_sources, attrgetter("distroseries", "pocket")):
+            if (self.cannotModifySuite(distroseries, pocket)
+                or not self.isAllowed(distroseries, pocket)):
+                # We don't want to mark release pockets dirty in a
+                # stable distroseries, no matter what other bugs
+                # that precede here have dirtied it.
+                continue
+            self.markPocketDirty(distroseries, pocket)
+
+        for (distroarchseries, pocket), binaries in groupby(
+                all_binaries, attrgetter("distroarchseries", "pocket")):
+            distroseries = distroarchseries.distroseries
+            if self.isDirty(distroseries, pocket):
+                # Already dirtied by sources.
+                continue
+            if (self.cannotModifySuite(distroseries, pocket)
+                or not self.isAllowed(distroseries, pocket)):
+                # We don't want to mark release pockets dirty in a
+                # stable distroseries, no matter what other bugs
+                # that precede here have dirtied it.
+                continue
+            self.markPocketDirty(distroseries, pocket)
 
     def B_dominate(self, force_domination):
         """Second step in publishing: domination."""