launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad-buildd/apt-lists into lp:launchpad-buildd.

Commit message:
lpbuildd/binarypackage.py: Use "apt-get indextargets" and "apt-helper cat-file" where they exist to read Packages files, rather than looking in /var/lib/apt/lists/ directly.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad-buildd/apt-lists/+merge/286751

The apt maintainers don't want other code poking around in /var/lib/apt/lists/ directly, not least because they might e.g. want to start lz4-compressing the package lists there in the near future.  Use the recommended helpers where available to read Packages files for dep-wait analysis.  We still need the old paths because these helpers are only available in rather recent versions of apt.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~cjwatson/launchpad-buildd/apt-lists into lp:launchpad-buildd.

=== modified file 'debian/changelog'
--- debian/changelog	2016-02-18 14:44:09 +0000
+++ debian/changelog	2016-02-21 18:26:03 +0000
@@ -5,6 +5,9 @@
     qemu-user-static.
   * Always raise exception instances rather than using the two-argument form
     of raise.
+  * lpbuildd/binarypackage.py: Use "apt-get indextargets" and "apt-helper
+    cat-file" where they exist to read Packages files, rather than looking
+    in /var/lib/apt/lists/ directly.
 
   [ Kit Randel ]
   * Add http/s proxy support for snap builds.

=== modified file 'lpbuildd/binarypackage.py'
--- lpbuildd/binarypackage.py	2016-02-18 14:44:09 +0000
+++ lpbuildd/binarypackage.py	2016-02-21 18:26:03 +0000
@@ -147,6 +147,34 @@
         args.append(self._dscfile)
         self.runSubProcess(self._sbuildpath, args)
 
+    def getAptLists(self):
+        """Yield each of apt's Packages files in turn as a file object."""
+        apt_helper = "/usr/lib/apt/apt-helper"
+        if os.path.exists(os.path.join(self.chroot_path, apt_helper[1:])):
+            paths = subprocess.check_output(
+                ["sudo", "chroot", self.chroot_path,
+                 "apt-get", "indextargets", "--format", "$(FILENAME)",
+                 "Created-By: Packages"],
+                universal_newlines=True).splitlines()
+            for path in paths:
+                helper = subprocess.Popen(
+                    ["sudo", "chroot", self.chroot_path,
+                     apt_helper, "cat-file", path],
+                    stdout=subprocess.PIPE)
+                try:
+                    yield helper.stdout
+                finally:
+                    helper.stdout.read()
+                    helper.wait()
+        else:
+            apt_lists = os.path.join(
+                self.chroot_path, "var", "lib", "apt", "lists")
+            for name in sorted(os.listdir(apt_lists)):
+                if name.endswith("_Packages"):
+                    path = os.path.join(apt_lists, name)
+                    with open(path, "rb") as packages_file:
+                        yield packages_file
+
     def getAvailablePackages(self):
         """Return the available binary packages in the chroot.
 
@@ -154,27 +182,21 @@
             available versions of each package.
         """
         available = defaultdict(set)
-        apt_lists = os.path.join(
-            self.chroot_path, "var", "lib", "apt", "lists")
-        for name in sorted(os.listdir(apt_lists)):
-            if name.endswith("_Packages"):
-                path = os.path.join(apt_lists, name)
-                with open(path, "rb") as packages_file:
-                    for section in apt_pkg.TagFile(packages_file):
-                        available[section["package"]].add(section["version"])
-                        if "provides" in section:
-                            provides = apt_pkg.parse_depends(
-                                section["provides"])
-                            for provide in provides:
-                                # Disjunctions are currently undefined here.
-                                if len(provide) > 1:
-                                    continue
-                                # Virtual packages may only provide an exact
-                                # version or none.
-                                if provide[0][1] and provide[0][2] != "=":
-                                    continue
-                                available[provide[0][0]].add(
-                                    provide[0][1] if provide[0][1] else None)
+        for packages_file in self.getAptLists():
+            for section in apt_pkg.TagFile(packages_file):
+                available[section["package"]].add(section["version"])
+                if "provides" in section:
+                    provides = apt_pkg.parse_depends(section["provides"])
+                    for provide in provides:
+                        # Disjunctions are currently undefined here.
+                        if len(provide) > 1:
+                            continue
+                        # Virtual packages may only provide an exact version
+                        # or none.
+                        if provide[0][1] and provide[0][2] != "=":
+                            continue
+                        available[provide[0][0]].add(
+                            provide[0][1] if provide[0][1] else None)
         return available
 
     def getBuildDepends(self, dscpath, arch_indep):