launchpad-reviewers team mailing list archive

[Colin Watson <mp+455965@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:fix-swift-db-pruning into launchpad:master.

Commit message:
Fix pruning of DB branch builds from Swift

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/455965

It makes no sense to prune DB builds when they're older than what's on staging, and I don't know what I was thinking when I wrote that; staging is updated frequently, so in practice what this means is that we prune a bunch of builds that are newer than production.  What we actually need to do is to prune builds that are older than the DB revision deployed to production.

Figuring this out is unfortunately challenging, but I found what seems to be a viable approach of fetching the deployment bundle and picking it out from there.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:fix-swift-db-pruning into launchpad:master.

diff --git a/setup.cfg b/setup.cfg
index 51645e8..a08e2f0 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -36,6 +36,7 @@ install_requires =
     gunicorn
     importlib-resources; python_version < "3.7"
     ipython
+    Jinja2
     jsautobuild
     kombu
     launchpad-buildd
diff --git a/utilities/publish-to-swift b/utilities/publish-to-swift
index e031dea..566d111 100755
--- a/utilities/publish-to-swift
+++ b/utilities/publish-to-swift
@@ -13,6 +13,8 @@ from argparse import ArgumentParser
 
 import iso8601
 import requests
+import yaml
+from jinja2 import Environment, FunctionLoader
 from swiftclient.service import (
     SwiftService,
     SwiftUploadObject,
@@ -82,19 +84,35 @@ def prune_old_files_from_swift(options, container_name, object_dir, suffix):
     if suffix:
         suffix = "-" + suffix
     if suffix.endswith("-db"):
+        # It's difficult for the Launchpad web application to publish the
+        # currently-deployed revision of its database branch.  The easiest
+        # approach seems to be to pick it out of the Juju bundle we deploy.
         try:
-            response = requests.head("https://staging.launchpad.net/";)
+            response = requests.get(
+                "https://git.launchpad.net/launchpad-mojo-specs/plain/";
+                "lp/bundle.yaml"
+            )
             response.raise_for_status()
         except requests.RequestException:
-            # Staging is routinely down in order to restore its database
-            # from a recent production dump, so don't consider this an
-            # error; just skip pruning if we can't determine its revision.
-            print("staging.launchpad.net is down; not pruning.")
+            print("git.launchpad.net is down; not pruning.")
             return
+        template_env = Environment(
+            loader=FunctionLoader(lambda _: response.text)
+        )
+        template = template_env.get_template("bundle.yaml")
+        bundle = yaml.safe_load(
+            template.render(
+                {"charm_dir": "fake-charm-dir", "stage_name": "production"}
+            )
+        )
+        db_update = bundle["applications"]["launchpad-db-update"]
+        deployed_revision = db_update["options"]["build_label"]
+        if deployed_revision.endswith("-db"):
+            deployed_revision = deployed_revision[: -len("-db")]
     else:
         response = requests.head("https://launchpad.net/";)
         response.raise_for_status()
-    deployed_revision = response.headers["X-VCS-Revision"]
+        deployed_revision = response.headers["X-VCS-Revision"]
 
     with SwiftService(options=options) as swift:
         objs = {}