launchpad-reviewers team mailing list archive

[Colin Watson <cjwatson@xxxxxxxxxxxxx>]

Colin Watson has proposed merging lp:~cjwatson/launchpad/contents-nanoseconds into lp:launchpad.

Commit message:
Work around Python bug that loses nanosecond precision in Contents file timestamps, causing them to be re-updated every time.

Requested reviews:
  Colin Watson (cjwatson)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/contents-nanoseconds/+merge/253678

The saga continues ...

I just noticed that the publisher is updating lots of Contents files on every run, which on closer investigation turns out to be because Python's os.utime truncates the timestamp to microsecond resolution, causing the copied file to appear older (fixed in Python 3.3, but we won't be on that for quite some time).  This works around that bug by rounding the timestamp up to the nearest second.
-- 
Your team Launchpad code reviewers is subscribed to branch lp:launchpad.

=== modified file 'lib/lp/archivepublisher/scripts/publish_ftpmaster.py'
--- lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2015-03-20 11:36:49 +0000
+++ lib/lp/archivepublisher/scripts/publish_ftpmaster.py	2015-03-20 14:09:39 +0000
@@ -9,6 +9,7 @@
     ]
 
 from datetime import datetime
+import math
 import os
 import shutil
 
@@ -579,6 +580,14 @@
                 "Installing new Contents file for %s/%s.", suite,
                 arch.architecturetag)
             shutil.copy2(new_contents, current_contents)
+            # Due to http://bugs.python.org/issue12904, shutil.copy2 doesn't
+            # copy timestamps precisely, and unfortunately it rounds down.
+            # If we must lose accuracy, we need to round up instead.  This
+            # can be removed once Launchpad runs on Python >= 3.3.
+            st = os.stat(new_contents)
+            os.utime(
+                current_contents,
+                (math.ceil(st.st_atime), math.ceil(st.st_mtime)))
             return True
         return False
 

=== modified file 'lib/lp/archivepublisher/tests/test_publish_ftpmaster.py'
--- lib/lp/archivepublisher/tests/test_publish_ftpmaster.py	2015-03-20 11:36:49 +0000
+++ lib/lp/archivepublisher/tests/test_publish_ftpmaster.py	2015-03-20 14:09:39 +0000
@@ -868,6 +868,32 @@
             "Contents",
             read_marker_file([backup_suite, "%s.gz" % contents_filename]))
 
+    def test_updateContentsFile_twice(self):
+        # If updateContentsFile is run twice in a row, it does not update
+        # the file the second time.
+        distro = self.makeDistroWithPublishDirectory()
+        distroseries = self.factory.makeDistroSeries(distribution=distro)
+        das = self.factory.makeDistroArchSeries(distroseries=distroseries)
+        script = self.makeScript(distro)
+        script.setUp()
+        script.setUpDirs()
+        archive_config = getPubConfig(distro.main_archive)
+        contents_filename = "Contents-%s" % das.architecturetag
+        backup_suite = os.path.join(
+            archive_config.archiveroot + "-distscopy", "dists",
+            distroseries.name)
+        os.makedirs(backup_suite)
+        content_suite = os.path.join(
+            archive_config.distroroot, "contents-generation", distro.name,
+            "dists", distroseries.name)
+        os.makedirs(content_suite)
+        write_marker_file(
+            [content_suite, "%s-staged.gz" % contents_filename], "Contents")
+        self.assertTrue(script.updateContentsFile(
+            distro.main_archive, distro, distroseries.name, das))
+        self.assertFalse(script.updateContentsFile(
+            distro.main_archive, distro, distroseries.name, das))
+
     def test_updateContentsFiles_updated_suites(self):
         # updateContentsFiles returns a list of suites for which it updated
         # Contents files.