launchpad-reviewers team mailing list archive

[Colin Watson <mp+419702@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:buildd-manager-download-to-subdirectory into launchpad:master.

Commit message:
buildd-manager: Fix downloading of files to subdirectories

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/419702

CI builds structure their output files using one subdirectory per job, so buildd-manager's downloader needs to cope with this.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:buildd-manager-download-to-subdirectory into launchpad:master.

diff --git a/lib/lp/buildmaster/downloader.py b/lib/lp/buildmaster/downloader.py
index ae795bd..1498b03 100644
--- a/lib/lp/buildmaster/downloader.py
+++ b/lib/lp/buildmaster/downloader.py
@@ -66,6 +66,10 @@ class RequestProcess(AMPChild):
         session.trust_env = False
         response = session.get(file_url, timeout=timeout, stream=True)
         response.raise_for_status()
+        try:
+            os.makedirs(os.path.dirname(path_to_write))
+        except FileExistsError:
+            pass
         f = tempfile.NamedTemporaryFile(
             mode="wb", prefix=os.path.basename(path_to_write) + "_",
             dir=os.path.dirname(path_to_write), delete=False)
diff --git a/lib/lp/buildmaster/tests/test_interactor.py b/lib/lp/buildmaster/tests/test_interactor.py
index d4f22fd..583fe1b 100644
--- a/lib/lp/buildmaster/tests/test_interactor.py
+++ b/lib/lp/buildmaster/tests/test_interactor.py
@@ -904,3 +904,17 @@ class TestWorkerWithLibrarian(TestCaseWithFactory):
         yield worker.getFiles([(empty_sha1, temp_name)])
         with open(temp_name, 'rb') as f:
             self.assertEqual(b'', f.read())
+
+    @defer.inlineCallbacks
+    def test_getFiles_to_subdirectory(self):
+        # getFiles works if asked to download files to a subdirectory.
+        # (This is used by CI builds.)
+        tachandler = self.worker_helper.getServerWorker()
+        worker = self.worker_helper.getClientWorker()
+        temp_dir = self.makeTemporaryDirectory()
+        temp_name = os.path.join(temp_dir, 'build:0', 'log')
+        empty_sha1 = hashlib.sha1(b'').hexdigest()
+        self.worker_helper.makeCacheFile(tachandler, empty_sha1, contents=b'')
+        yield worker.getFiles([(empty_sha1, temp_name)])
+        with open(temp_name, 'rb') as f:
+            self.assertEqual(b'', f.read())