launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad/dpkg-xz-support-619152 into lp:launchpad/devel.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  #619152 Add data.tar.xz support
  https://bugs.launchpad.net/bugs/619152


Summary:

Bug 619152 explains that newer versions of dpkg (in maverick) add support for xz compression of data.tar members in .deb packages.  In order to make this usable, support needs to be added to Launchpad in the same way that it was previously added for lzma compression.

Proposed fix:

Add data.tar.xz to the locations where data.tar.lzma is currently mentioned.

In order to make sure that upgrades work properly, we need to enforce a Pre-Depends on a new enough version of dpkg in packages using this, at least until the next LTS release.

Pre-implementation notes:

Discussed with Jelmer.  This can't land until we get a newer version of dpkg into hardy-cat for use on the Launchpad servers.

Tests:

This branch adds testXZDebUpload to lib/lp/archiveuploader/tests/test_uploadprocessor.py.

QA plan:

Upload a package using 'dh_builddeb -- -Zxz'.  We'll be doing this as part of the foundations-m-spring-cleaning specification (for maverick, if this change lands in time).
-- 
https://code.launchpad.net/~cjwatson/launchpad/dpkg-xz-support-619152/+merge/32868
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~cjwatson/launchpad/dpkg-xz-support-619152 into lp:launchpad/devel.

=== modified file 'lib/lp/archiveuploader/nascentuploadfile.py'
--- lib/lp/archiveuploader/nascentuploadfile.py	2010-07-21 11:13:19 +0000
+++ lib/lp/archiveuploader/nascentuploadfile.py	2010-08-17 12:14:45 +0000
@@ -647,10 +647,53 @@
             yield UploadError(
                 "%s: second chunk is %s, expected control.tar.gz." % (
                 self.filename, control_tar))
-        if data_tar not in ("data.tar.gz", "data.tar.bz2", "data.tar.lzma"):
+        if data_tar not in ("data.tar.gz", "data.tar.bz2", "data.tar.lzma",
+                            "data.tar.xz"):
             yield UploadError(
                 "%s: third chunk is %s, expected data.tar.gz, "
-                "data.tar.bz2 or data.tar.lzma." % (self.filename, data_tar))
+                "data.tar.bz2, data.tar.lzma or data.tar.xz." %
+                (self.filename, data_tar))
+
+        # xz-compressed debs must pre-depend on dpkg >= 1.15.6.
+        XZ_REQUIRED_DPKG_VER = '1.15.6'
+        if data_tar == "data.tar.xz":
+            parsed_deps = []
+            try:
+                parsed_deps = apt_pkg.ParseDepends(
+                    self.control['Pre-Depends'])
+            except (ValueError, TypeError):
+                yield UploadError(
+                    "Can't parse Pre-Depends in the control file.")
+                return
+            except KeyError:
+                # Go past the for loop and yield the error below.
+                pass
+
+            for token in parsed_deps:
+                try:
+                    name, version, relation = token[0]
+                except ValueError:
+                    yield("APT error processing token '%r' from Pre-Depends.")
+                    return
+
+                if name == 'dpkg':
+                    # VersionCompare returns values similar to cmp;
+                    # negative if first < second, zero if first ==
+                    # second and positive if first > second.
+                    if apt_pkg.VersionCompare(
+                        version, XZ_REQUIRED_DPKG_VER) >= 0:
+                        # Pre-Depends dpkg is fine.
+                        return
+                    else:
+                        yield UploadError(
+                            "Pre-Depends dpkg version should be >= %s "
+                            "when using xz compression." %
+                            XZ_REQUIRED_DPKG_VER)
+                        return
+
+            yield UploadError(
+                "Require Pre-Depends: dpkg (>= %s) when using xz "
+                "compression." % XZ_REQUIRED_DPKG_VER)
 
     def verifyDebTimestamp(self):
         """Check specific DEB format timestamp checks."""
@@ -668,7 +711,8 @@
                                 "control.tar.gz")
             # Only one of these files is present in the archive, so loop
             # until we find one of them, otherwise fail.
-            data_files = ("data.tar.gz", "data.tar.bz2", "data.tar.lzma")
+            data_files = ("data.tar.gz", "data.tar.bz2", "data.tar.lzma",
+                          "data.tar.xz")
             for file in data_files:
                 deb_file.seek(0)
                 try:

=== added directory 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz'
=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.diff.gz'
Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.diff.gz	1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.diff.gz	2010-08-17 12:14:45 +0000 differ
=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.dsc'
--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.dsc	1970-01-01 00:00:00 +0000
+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1.dsc	2010-08-17 12:14:45 +0000
@@ -0,0 +1,28 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+Format: 1.0
+Source: bar
+Binary: bar
+Architecture: any
+Version: 1.0-1
+Maintainer: Launchpad team <launchpad@xxxxxxxxxxxxxxxxxxx>
+Standards-Version: 3.6.2
+Checksums-Sha1: 
+ 73a04163fee97fd2257ab266bd48f1d3d528e012 164 bar_1.0.orig.tar.gz
+ 24b22aae5a9c983640ba14e2744422382a51e85c 573 bar_1.0-1.diff.gz
+Checksums-Sha256: 
+ f1ecff929899b567f45d6734b69d59a4f3c04dabce3cc8e6ed6d64073eda360e 164 bar_1.0.orig.tar.gz
+ 0a332563ea8f3f918f11d8f756c53f69cb6e555498d27bfaffa6f0b63b91080b 573 bar_1.0-1.diff.gz
+Files: 
+ fc1464e5985b962a042d5354452f361d 164 bar_1.0.orig.tar.gz
+ 9f5ca79e2655f81fd7ea4f5a3e68e307 573 bar_1.0-1.diff.gz
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.10 (GNU/Linux)
+Comment: Colin Watson <cjwatson@xxxxxxxxxx> -- Debian developer
+
+iD8DBQFManAkjn63CGxkqMURApSqAJ0TFHGGQ1YKayUp9+WAGtw3BvhukACdH+4U
+GTONJB6DgQkgET/TEHc4jfw=
+=4v08
+-----END PGP SIGNATURE-----

=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1_source.changes'
--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1_source.changes	1970-01-01 00:00:00 +0000
+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0-1_source.changes	2010-08-17 12:14:45 +0000
@@ -0,0 +1,40 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+Format: 1.8
+Date: Thu, 16 Feb 2006 15:34:09 +0000
+Source: bar
+Binary: bar
+Architecture: source
+Version: 1.0-1
+Distribution: breezy
+Urgency: low
+Maintainer: Launchpad team <launchpad@xxxxxxxxxxxxxxxxxxx>
+Changed-By: Daniel Silverstone <daniel.silverstone@xxxxxxxxxxxxx>
+Description: 
+ bar        - Stuff for testing
+Changes: 
+ bar (1.0-1) breezy; urgency=low
+ .
+   * Initial version
+Checksums-Sha1: 
+ 607e0e7f4853e22b6336a06955e5abac4553262c 921 bar_1.0-1.dsc
+ 73a04163fee97fd2257ab266bd48f1d3d528e012 164 bar_1.0.orig.tar.gz
+ 24b22aae5a9c983640ba14e2744422382a51e85c 573 bar_1.0-1.diff.gz
+Checksums-Sha256: 
+ fbba08406eff47c0e9be218620d9c56ddcaf8f36e85e8fce1e117dbe55c779ee 921 bar_1.0-1.dsc
+ f1ecff929899b567f45d6734b69d59a4f3c04dabce3cc8e6ed6d64073eda360e 164 bar_1.0.orig.tar.gz
+ 0a332563ea8f3f918f11d8f756c53f69cb6e555498d27bfaffa6f0b63b91080b 573 bar_1.0-1.diff.gz
+Files: 
+ 9a21d129444e84c51e8f81a0eb950834 921 devel optional bar_1.0-1.dsc
+ fc1464e5985b962a042d5354452f361d 164 devel optional bar_1.0.orig.tar.gz
+ 9f5ca79e2655f81fd7ea4f5a3e68e307 573 devel optional bar_1.0-1.diff.gz
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.10 (GNU/Linux)
+Comment: Colin Watson <cjwatson@xxxxxxxxxx> -- Debian developer
+
+iD8DBQFManAljn63CGxkqMURAgIDAJ9h62hYOQcezbCwB7ySGwvGZL7ipgCcCWJm
+GgaOcKpNNd2UyiPY0U+XVhE=
+=KiAo
+-----END PGP SIGNATURE-----

=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0.orig.tar.gz'
Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0.orig.tar.gz	1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz/bar_1.0.orig.tar.gz	2010-08-17 12:14:45 +0000 differ
=== added directory 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary'
=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.changes'
--- lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.changes	1970-01-01 00:00:00 +0000
+++ lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.changes	2010-08-17 12:14:45 +0000
@@ -0,0 +1,34 @@
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+Format: 1.8
+Date: Thu, 16 Feb 2006 15:34:09 +0000
+Source: bar
+Binary: bar
+Architecture: i386
+Version: 1.0-1
+Distribution: breezy
+Urgency: low
+Maintainer: Launchpad team <launchpad@xxxxxxxxxxxxxxxxxxx>
+Changed-By: Daniel Silverstone <daniel.silverstone@xxxxxxxxxxxxx>
+Description: 
+ bar        - Stuff for testing
+Changes: 
+ bar (1.0-1) breezy; urgency=low
+ .
+   * Initial version
+Checksums-Sha1: 
+ 29ac64be44f4cd6e8135662bd241e324ab243fdb 728 bar_1.0-1_i386.deb
+Checksums-Sha256: 
+ 73476463c92e5c6a22e3eec997199af6e80dd0275db667f1c9fe7049fee46cf0 728 bar_1.0-1_i386.deb
+Files: 
+ 9bc1955145a3fa3ff41e4a404cb1e21a 728 devel optional bar_1.0-1_i386.deb
+
+-----BEGIN PGP SIGNATURE-----
+Version: GnuPG v1.4.10 (GNU/Linux)
+Comment: Colin Watson <cjwatson@xxxxxxxxxx> -- Debian developer
+
+iD8DBQFManA0jn63CGxkqMURAlQ+AJ9GOOYLf7f8MpsUB9OWKAhfwSk96ACeJNh+
+2yGkbNYlEFDavze9JjHt+VY=
+=3ZiC
+-----END PGP SIGNATURE-----

=== added file 'lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.deb'
Binary files lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.deb	1970-01-01 00:00:00 +0000 and lib/lp/archiveuploader/tests/data/suite/bar_1.0-1_xz_binary/bar_1.0-1_i386.deb	2010-08-17 12:14:45 +0000 differ
=== modified file 'lib/lp/archiveuploader/tests/test_uploadprocessor.py'
--- lib/lp/archiveuploader/tests/test_uploadprocessor.py	2010-08-13 06:08:36 +0000
+++ lib/lp/archiveuploader/tests/test_uploadprocessor.py	2010-08-17 12:14:45 +0000
@@ -1348,8 +1348,8 @@
     def testLZMADebUpload(self):
         """Make sure that data files compressed with lzma in Debs work.
 
-        Each Deb contains a data.tar.xxx file where xxx is one of gz, bz2
-        or lzma.  Here we make sure that lzma works.
+        Each Deb contains a data.tar.xxx file where xxx is one of gz, bz2,
+        lzma or xz.  Here we make sure that lzma works.
         """
         # Setup the test.
         self.setupBreezy()
@@ -1389,6 +1389,50 @@
             "Expected one 'bar' item in the queue, actually got %d."
                 % queue_items.count())
 
+    def testXZDebUpload(self):
+        """Make sure that data files compressed with xz in Debs work.
+
+        Each Deb contains a data.tar.xxx file where xxx is one of gz, bz2,
+        lzma or xz.  Here we make sure that xz works.
+        """
+        # Setup the test.
+        self.setupBreezy()
+        self.layer.txn.commit()
+        self.options.context = 'absolutely-anything'
+        uploadprocessor = self.getUploadProcessor(self.layer.txn)
+
+        # Upload the source first to enable the binary later:
+        upload_dir = self.queueUpload("bar_1.0-1_xz")
+        self.processUpload(uploadprocessor, upload_dir)
+        # Make sure it went ok:
+        from_addr, to_addrs, raw_msg = stub.test_emails.pop()
+        self.assertTrue(
+            "rejected" not in raw_msg,
+            "Failed to upload bar source:\n%s" % raw_msg)
+        self.publishPackage("bar", "1.0-1")
+        # Clear out emails generated during upload.
+        ignore = pop_notifications()
+
+        # Upload a binary xz-compressed package.
+        upload_dir = self.queueUpload("bar_1.0-1_xz_binary")
+        self.processUpload(uploadprocessor, upload_dir)
+
+        # Successful binary uploads won't generate any email.
+        if len(stub.test_emails) != 0:
+            from_addr, to_addrs, raw_msg = stub.test_emails.pop()
+        self.assertEqual(
+            len(stub.test_emails), 0,
+            "Expected no emails!  Actually got:\n%s" % raw_msg)
+
+        # Check in the queue to see if it really made it:
+        queue_items = self.breezy.getQueueItems(
+            status=PackageUploadStatus.NEW, name="bar",
+            version="1.0-1", exact_match=True)
+        self.assertEqual(
+            queue_items.count(), 1,
+            "Expected one 'bar' item in the queue, actually got %d."
+                % queue_items.count())
+
     def testUploadResultingInNoBuilds(self):
         """Source uploads resulting in no builds.