launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad-buildd/search-partial-log into lp:launchpad-buildd.

Commit message:
Stop searching sbuild logs when we see "Toolchain package versions:".

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad-buildd/search-partial-log/+merge/187700

When QAing https://code.launchpad.net/~cjwatson/launchpad-buildd/large-build-artifacts/+merge/186503, I discovered that in the very case at hand - a large build log - launchpad-buildd takes about three minutes to process the log during which time it isn't responding to RPC requests from the master, so the master times it out and cancels the build.

The log file searches we're doing here only need to care about the phase of the build up to the end of build-dependency installation.  Therefore, a reasonable fix is to stop searching when we see "Toolchain package versions:".  (I thought about doing this when preparing my previous branch, but the problem of failing to respond to RPC requests hadn't occurred to me, so I postponed it.)  On my system, this takes the search time down to 0.1 seconds in the worst case where there's no match for any of the relevant regexes.
-- 
https://code.launchpad.net/~cjwatson/launchpad-buildd/search-partial-log/+merge/187700
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~cjwatson/launchpad-buildd/search-partial-log into lp:launchpad-buildd.

=== modified file 'lpbuildd/binarypackage.py'
--- lpbuildd/binarypackage.py	2013-09-24 08:58:57 +0000
+++ lpbuildd/binarypackage.py	2013-09-26 09:44:30 +0000
@@ -90,6 +90,7 @@
         """Finished the sbuild run."""
         if success != SBuildExitCodes.OK:
             log_patterns = []
+            stop_patterns = [["^Toolchain package versions:", re.M]]
 
             if (success == SBuildExitCodes.DEPFAIL or
                 success == SBuildExitCodes.PACKAGEFAIL):
@@ -101,7 +102,7 @@
                     log_patterns.append([rx, re.M])
 
             if log_patterns:
-                rx, mo = self.searchLogContents(log_patterns)
+                rx, mo = self.searchLogContents(log_patterns, stop_patterns)
                 if mo:
                     if rx in BuildLogRegexes.GIVENBACK:
                         success = SBuildExitCodes.GIVENBACK

=== modified file 'lpbuildd/debian.py'
--- lpbuildd/debian.py	2013-09-19 11:01:10 +0000
+++ lpbuildd/debian.py	2013-09-26 09:44:30 +0000
@@ -173,13 +173,17 @@
                 self._state = DebianBuildState.UPDATE
                 self.doUpdateChroot()
 
-    def searchLogContents(self, patterns_and_flags):
+    def searchLogContents(self, patterns_and_flags,
+                          stop_patterns_and_flags=[]):
         """Search for any of a list of regex patterns in the build log.
 
         The build log is matched using a sliding window, which avoids having
         to read the whole file into memory at once but requires that matches
         be no longer than the chunk size (currently 256KiB).
 
+        If any of the stop patterns are matched, the search stops
+        immediately without reading the rest of the file.
+
         :return: A tuple of the regex pattern that matched and the match
             object, or (None, None).
         """
@@ -187,6 +191,9 @@
         regexes = [
             re.compile(pattern, flags)
             for pattern, flags in patterns_and_flags]
+        stop_regexes = [
+            re.compile(pattern, flags)
+            for pattern, flags in stop_patterns_and_flags]
         log = open(os.path.join(self._cachepath, "buildlog"))
         try:
             window = ""
@@ -197,6 +204,9 @@
                     match = regex.search(window)
                     if match is not None:
                         return regex.pattern, match
+                for regex in stop_regexes:
+                    if regex.search(window) is not None:
+                        return None, None
                 if len(window) > chunk_size:
                     window = window[chunk_size:]
                 chunk = log.read(chunk_size)