launchpad-reviewers team mailing list archive

[Colin Watson <mp+444120@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:charm-librarian-clear-storage-action into launchpad:master.

Commit message:
charm: Add a clear-storage action to the librarian

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/444120

This is useful immediately after resetting the associated database.  I expect to make use of it as part of automation for the staging environment, whose database is reset based on a production backup once a week.

Since this is destructive, I've included a couple of safety checks.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:charm-librarian-clear-storage-action into launchpad:master.

diff --git a/charm/launchpad-librarian/README.md b/charm/launchpad-librarian/README.md
index 52ab88c..5207d0a 100644
--- a/charm/launchpad-librarian/README.md
+++ b/charm/launchpad-librarian/README.md
@@ -94,3 +94,14 @@ especially careful when redeploying.  The general procedure is as follows:
 1. As `stg-launchpad@launchpad-bastion-ps5.internal`, run `lpndt
    service-start buildd-manager` to start `buildd-manager, then (after a
    minute) `lpndt service-start cron-fdt` to enable all cron jobs.
+
+## Clearing librarian storage
+
+If you have reset the associated database, then it may be useful to clear
+the librarian's storage as well, since the IDs used for stored files will no
+longer exist in the database.  To do this, run:
+
+    juju run-action --wait launchpad-librarian/leader clear-storage host=launchpadlibrarian.test
+
+The value passed to `host=` must match the hostname part of the
+`librarian_download_url` configuration option.
diff --git a/charm/launchpad-librarian/actions.yaml b/charm/launchpad-librarian/actions.yaml
new file mode 100644
index 0000000..54cd354
--- /dev/null
+++ b/charm/launchpad-librarian/actions.yaml
@@ -0,0 +1,15 @@
+clear-storage:
+  description: |
+    Clear local librarian storage.  This irreversibly deletes data, so it
+    requires the host name of this librarian instance to avoid accidentally
+    clearing the wrong instance, and it always refuses to clear storage for
+    the Launchpad production instance.  It is useful when the associated
+    database has been reset.
+  params:
+    host:
+      type: string
+      description: |
+        The public host name of this librarian instance, required as a
+        safety check.
+  required:
+    - host
diff --git a/charm/launchpad-librarian/actions/clear-storage b/charm/launchpad-librarian/actions/clear-storage
new file mode 100755
index 0000000..8e77570
--- /dev/null
+++ b/charm/launchpad-librarian/actions/clear-storage
@@ -0,0 +1,52 @@
+#! /usr/bin/python3
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import re
+import shutil
+import sys
+from pathlib import Path
+from urllib.parse import urlparse
+
+sys.path.append("lib")
+
+from charms.layer import basic  # noqa: E402
+
+basic.bootstrap_charm_deps()
+basic.init_config_states()
+
+from charmhelpers.core import hookenv  # noqa: E402
+from ols import base  # noqa: E402
+
+
+def clear_storage():
+    params = hookenv.action_get()
+    config = hookenv.config()
+    expected_host = urlparse(config["librarian_download_url"]).hostname
+    if expected_host == "launchpadlibrarian.net":
+        hookenv.action_fail("Refusing to clear storage for production.")
+        return
+    if expected_host != params["host"]:
+        hookenv.action_fail(
+            f"Requested clearing storage for {params['host']}, but this is "
+            f"{expected_host}."
+        )
+        return
+
+    librarian_dir = Path(base.base_dir()) / "librarian"
+    hookenv.log(f"Clearing {librarian_dir}")
+    # Only consider subdirectories created by
+    # lp.services.librarianserver.storage._relFileLocation.  In particular,
+    # this excludes "incoming" and "lost+found".
+    librarian_subdirs = [
+        path
+        for path in librarian_dir.iterdir()
+        if re.match(r"^[0-9a-f][0-9a-f]$", path.name)
+    ]
+    for path in librarian_subdirs:
+        shutil.rmtree(path)
+    hookenv.action_set({"result": "Completed"})
+
+
+if __name__ == "__main__":
+    clear_storage()