launchpad-reviewers team mailing list archive

[Colin Watson <mp+450472@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:charm-cron-control into launchpad:master.

Commit message:
charm: Add launchpad-cron-control

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/450472

One of the remaining odd pieces of our legacy frontend stack is "cron-control": a mechanism to centrally enable and disable cron jobs, checked by `lp.services.scripts.base.cronscript_enabled`.  This is an important part of some of our rollout procedures, such as database schema migrations.  It's worth having this mechanism in addition to things like `stop-services` and `start-services` actions on individual charms, because it gives us a very quick and easy central switch.

The actual machinery behind this amounts to a simple script to manipulate a `.ini` file which is served internally over HTTP, so it's pretty easy to get an equivalent up and running using charms.  While the original script can be found at https://bazaar.launchpad.net/+branch/canonical-mojo-specs/view/head:/launchpad-manual-servers/files/croncontrol-ctl.sh, I thought it would be more readable to reimplement similar logic in Python rather than incorporating that shell script directly.  Relying on Juju actions as the primary interface to this also means there's no particular need to worry about locking.

Using this in practice for a database deployment might look something like this:

  $ juju run-action --wait launchpad-cron-control/leader disable-cron job=publish-ftpmaster
  [...]
  $ curl http://cron-control.launchpad.test/cron.ini
  [DEFAULT]
  enabled = True

  [publish-ftpmaster]
  enabled = False

  # wait for slow Ubuntu archive publisher job to complete

  $ juju run-action --wait launchpad-cron-control/leader disable-cron-all
  [...]
  $ curl http://cron-control.launchpad.test/cron.ini
  [DEFAULT]
  enabled = False

  # wait for any remaining slow jobs to quiesce

  # run migration

  $ juju run-action --wait launchpad-cron-control/leader enable-cron-all
  [...]
  $ curl http://cron-control.launchpad.test/cron.ini
  [DEFAULT]
  enabled = True
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:charm-cron-control into launchpad:master.

diff --git a/charm/launchpad-cron-control/README.md b/charm/launchpad-cron-control/README.md
new file mode 100644
index 0000000..46e37cf
--- /dev/null
+++ b/charm/launchpad-cron-control/README.md
@@ -0,0 +1,30 @@
+# Launchpad cron controller
+
+This charm provides a centralized control mechanism for Launchpad's cron
+jobs.  It must be related to an `apache2` charm as follows:
+
+    juju relate launchpad-cron-control:apache-website frontend-cron-control:apache-website
+
+Once deployed, it should typically be pointed to by DNS using the same name
+as set in the `domain_cron_control` configuration option, and the
+`cron_control_url` option in other Launchpad charms should be set to
+`http://cron-control.launchpad.test/cron.ini` (substitute the appropriate
+domain name).
+
+## Actions
+
+To disable all cron jobs by default:
+
+    juju run-action --wait launchpad-cron-control/leader disable-cron-all
+
+To disable a particular job:
+
+    juju run-action --wait launchpad-cron-control/leader disable-cron job=publish-ftpmaster
+
+To enable all cron jobs:
+
+    juju run-action --wait launchpad-cron-control/leader enable-cron-all
+
+To enable a particular job, even if `disable-cron-all` is in effect:
+
+    juju run-action --wait launchpad-cron-control/leader enable-cron job=publish-ftpmaster
diff --git a/charm/launchpad-cron-control/actions.yaml b/charm/launchpad-cron-control/actions.yaml
new file mode 100644
index 0000000..734b826
--- /dev/null
+++ b/charm/launchpad-cron-control/actions.yaml
@@ -0,0 +1,20 @@
+disable-cron:
+  description: Disable a single cron job in this Launchpad environment.
+  params:
+    job:
+      type: string
+      description: The name of the cron job to disable.
+disable-cron-all:
+  description: >
+    Disable all cron jobs in this Launchpad environment by default.
+    (Individual jobs can be enabled using `enable-cron`.)
+enable-cron:
+  description: >
+    Enable a single cron job in this Launchpad environment.  (Only useful if
+    all cron jobs have been disabled by default using `disable-cron-all`.)
+  params:
+    job:
+      type: string
+      description: The name of the cron job to enable.
+enable-cron-all:
+  description: Enable all cron jobs in this Launchpad environment.
diff --git a/charm/launchpad-cron-control/actions/actions.py b/charm/launchpad-cron-control/actions/actions.py
new file mode 100755
index 0000000..1a0c298
--- /dev/null
+++ b/charm/launchpad-cron-control/actions/actions.py
@@ -0,0 +1,100 @@
+#! /usr/bin/python3
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os.path
+import sys
+import traceback
+from configparser import ConfigParser, NoSectionError
+from pathlib import Path
+from typing import Sequence
+
+sys.path.append("lib")
+
+from charms.layer import basic  # noqa: E402
+
+basic.bootstrap_charm_deps()
+basic.init_config_states()
+
+from charmhelpers.core import hookenv  # noqa: E402
+
+config_path = "/srv/launchpad/www/cron.ini"
+
+
+def read_config() -> ConfigParser:
+    config = ConfigParser({"enabled": "True"})
+    config.read(config_path)
+    return config
+
+
+def write_config(config: ConfigParser) -> None:
+    with open(f"{config_path}.new", "w") as f:
+        config.write(f)
+    os.replace(f"{config_path}.new", config_path)
+
+
+def set_config_option(
+    config: ConfigParser, section: str, option: str, value: str
+) -> None:
+    """Set a config option, ensuring that its section exists."""
+    if section != "DEFAULT" and not config.has_section(section):
+        config.add_section(section)
+    config.set(section, option, value)
+
+
+def enable_cron() -> None:
+    params = hookenv.action_get()
+    config = read_config()
+    if config.getboolean("DEFAULT", "enabled", fallback=False):
+        # The default is already enabled.  Just make sure that we aren't
+        # overriding the default.
+        try:
+            config.remove_option(params["job"], "enabled")
+        except NoSectionError:
+            pass
+    else:
+        set_config_option(config, params["job"], "enabled", "True")
+    write_config(config)
+
+
+def enable_cron_all() -> None:
+    config = ConfigParser({"enabled": "True"})
+    write_config(config)
+
+
+def disable_cron() -> None:
+    params = hookenv.action_get()
+    config = read_config()
+    set_config_option(config, params["job"], "enabled", "False")
+    write_config(config)
+
+
+def disable_cron_all() -> None:
+    config = ConfigParser({"enabled": "False"})
+    write_config(config)
+
+
+def main(argv: Sequence[str]) -> None:
+    action = Path(argv[0]).name
+    try:
+        if action == "disable-cron":
+            disable_cron()
+        elif action == "disable-cron-all":
+            disable_cron_all()
+        elif action == "enable-cron":
+            enable_cron()
+        elif action == "enable-cron-all":
+            enable_cron_all()
+        else:
+            hookenv.action_fail(f"Action {action} not implemented.")
+    except Exception:
+        hookenv.action_fail("Unhandled exception")
+        tb = traceback.format_exc()
+        hookenv.action_set(dict(traceback=tb))
+        hookenv.log(f"Unhandled exception in action {action}:")
+        for line in tb.splitlines():
+            hookenv.log(line)
+
+
+if __name__ == "__main__":
+    main(sys.argv)
diff --git a/charm/launchpad-cron-control/actions/disable-cron b/charm/launchpad-cron-control/actions/disable-cron
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-cron-control/actions/disable-cron
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-cron-control/actions/disable-cron-all b/charm/launchpad-cron-control/actions/disable-cron-all
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-cron-control/actions/disable-cron-all
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-cron-control/actions/enable-cron b/charm/launchpad-cron-control/actions/enable-cron
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-cron-control/actions/enable-cron
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-cron-control/actions/enable-cron-all b/charm/launchpad-cron-control/actions/enable-cron-all
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/charm/launchpad-cron-control/actions/enable-cron-all
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/charm/launchpad-cron-control/charmcraft.yaml b/charm/launchpad-cron-control/charmcraft.yaml
new file mode 100644
index 0000000..0187d46
--- /dev/null
+++ b/charm/launchpad-cron-control/charmcraft.yaml
@@ -0,0 +1,60 @@
+type: charm
+bases:
+  - build-on:
+    - name: ubuntu
+      channel: "20.04"
+      architectures: [amd64]
+    run-on:
+    - name: ubuntu
+      channel: "20.04"
+      architectures: [amd64]
+parts:
+  charm-wheels:
+    source: https://git.launchpad.net/~ubuntuone-hackers/ols-charm-deps/+git/wheels
+    source-commit: "42c89d9c66dbe137139b047fd54aed49b66d1a5e"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      "*": charm-wheels/
+    prime:
+      - "-charm-wheels"
+  ols-layers:
+    source: https://git.launchpad.net/ols-charm-deps
+    source-commit: "f63ae0386275bf9089b30c8abae252a0ea523633"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      "*": layers/
+    stage:
+      - layers
+    prime:
+      - "-layers"
+  interface-apache-website:
+    source: https://github.com/juju-solutions/interface-apache-website
+    source-commit: "2f736ebcc90d19ac142a2d898a2ec7e1aafaa13f"
+    source-submodules: []
+    source-type: git
+    plugin: dump
+    organize:
+      "*": layers/interface/apache-website/
+    stage:
+      - layers
+    prime:
+      - "-layers"
+  charm:
+    after:
+      - charm-wheels
+      - ols-layers
+      - interface-apache-website
+    source: .
+    plugin: reactive
+    build-snaps: [charm]
+    build-packages: [python3-dev]
+    build-environment:
+      - CHARM_LAYERS_DIR: $CRAFT_STAGE/layers/layer
+      - CHARM_INTERFACES_DIR: $CRAFT_STAGE/layers/interface
+      - PIP_NO_INDEX: "true"
+      - PIP_FIND_LINKS: $CRAFT_STAGE/charm-wheels
+    reactive-charm-build-arguments: [--binary-wheels-from-source]
diff --git a/charm/launchpad-cron-control/config.yaml b/charm/launchpad-cron-control/config.yaml
new file mode 100644
index 0000000..0b24408
--- /dev/null
+++ b/charm/launchpad-cron-control/config.yaml
@@ -0,0 +1,9 @@
+options:
+  domain_cron_control:
+    type: string
+    description: Domain name for this instance's cron-control service.
+    default: "cron-control.launchpad.test"
+  webmaster_email:
+    type: string
+    description: Webmaster contact address.
+    default: "webmaster@xxxxxxxxxxxxxx"
diff --git a/charm/launchpad-cron-control/icon.svg b/charm/launchpad-cron-control/icon.svg
new file mode 100644
index 0000000..b2889cc
--- /dev/null
+++ b/charm/launchpad-cron-control/icon.svg
@@ -0,0 +1 @@
+<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg"; viewBox="0 0 165.39062 165.39062"><defs><style>.cls-1{fill:#e9500e;}.cls-2{fill:#fff;}</style></defs><rect class="cls-1" width="165.39062" height="165.39062"/><path class="cls-2" d="M29.63876,57.97189C43.189,67.692,61.13456,69.25577,77.65457,62.15038c16.25576-6.87157,27.74036-21.43444,29.97828-38.0075.04663-.34331.11016-.81367-1.59861-1.24044l-10.10934-2.197c-.3254-.04494-.79136-.04967-1.15258,1.22455C91.37844,36.07384,84.34062,45.04243,72.6347,50.1123c-11.77316,5.10029-23.18748,4.05279-35.91893-3.29386-.58119-.27843-.91909-.26086-1.45568.52577l-5.77947,8.65163A1.34512,1.34512,0,0,0,29.63876,57.97189Z" transform="translate(0.39062 0.39062)"/><path class="cls-2" d="M79.86106,139.66026l10.3631.565c1.74155.03446,1.79122-.42981,1.83717-.77312,2.23826-16.5734-4.97222-33.66107-18.81739-44.59422C59.196,83.62132,41.47815,80.36935,25.83365,86.14747a1.33956,1.33956,0,0,0-.67918,1.85373l3.28,9.88226c.30952.90153.62816,1.011,1.26443.89409,14.22464-3.70543,25.50717-1.68748,35.50635,6.3512,9.94174,7.9934,14.34865,18.50754,13.86883,33.08867C79.08524,139.50144,79.53735,139.615,79.86106,139.66026Z" transform="translate(0.39062 0.39062)"/><path class="cls-2" d="M86.50488,70.59048a10.50817,10.50817,0,0,0-1.39587-.09461A9.35237,9.35237,0,0,0,79.39915,72.382a9.61981,9.61981,0,1,0,7.10573-1.79156Z" transform="translate(0.39062 0.39062)"/><path class="cls-2" d="M138.26869,53.18923,133.457,43.97736c-.68628-1.51583-1.22793-1.36985-1.79594-1.17657-15.382,6.63165-25.99848,21.22156-28.40434,39.03776-2.40755,17.82971,3.97169,34.72681,17.0647,45.19906a1.177,1.177,0,0,0,.90794.32844,1.48362,1.48362,0,0,0,.99546-.54l6.76175-8.11166c.62342-.78393.35783-1.18333.0321-1.52461-10.60639-10.44454-14.5764-20.81677-12.84905-33.60769,1.73682-12.86121,8.51918-22.08254,21.34457-29.019C138.52854,53.95289,138.36533,53.421,138.26869,53.18923Z" transform="translate(0.39062 0.39062)"/></svg>
\ No newline at end of file
diff --git a/charm/launchpad-cron-control/layer.yaml b/charm/launchpad-cron-control/layer.yaml
new file mode 100644
index 0000000..e3f44d9
--- /dev/null
+++ b/charm/launchpad-cron-control/layer.yaml
@@ -0,0 +1,4 @@
+includes:
+  - layer:basic
+  - interface:apache-website
+repo: https://git.launchpad.net/launchpad
diff --git a/charm/launchpad-cron-control/metadata.yaml b/charm/launchpad-cron-control/metadata.yaml
new file mode 100644
index 0000000..7be5b77
--- /dev/null
+++ b/charm/launchpad-cron-control/metadata.yaml
@@ -0,0 +1,15 @@
+name: launchpad-cron-control
+display-name: launchpad-cron-control
+summary: Launchpad cron controller
+maintainer: Launchpad Developers <launchpad-dev@xxxxxxxxxxxxxxxxxxx>
+description: |
+  Launchpad is an open source suite of tools that help people and teams
+  to work together on software projects.
+
+  This charm provides a centralized control mechanism for Launchpad's cron
+  jobs.
+subordinate: true
+requires:
+  apache-website:
+    interface: apache-website
+    scope: container
diff --git a/charm/launchpad-cron-control/reactive/launchpad-cron-control.py b/charm/launchpad-cron-control/reactive/launchpad-cron-control.py
new file mode 100644
index 0000000..d98a6c5
--- /dev/null
+++ b/charm/launchpad-cron-control/reactive/launchpad-cron-control.py
@@ -0,0 +1,68 @@
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os.path
+from configparser import ConfigParser
+
+from charmhelpers.core import hookenv, host, templating
+from charms.reactive import (
+    clear_flag,
+    endpoint_from_flag,
+    hook,
+    set_flag,
+    when,
+    when_not,
+    when_not_all,
+)
+
+
+def www_dir():
+    return "/srv/launchpad/www"
+
+
+@when_not("service.configured")
+def configure():
+    host.mkdir(www_dir())
+    config_path = os.path.join(www_dir(), "cron.ini")
+    if not os.path.exists(config_path):
+        with open(config_path, "w") as f:
+            ConfigParser({"enabled": "True"}).write(f)
+    set_flag("service.configured")
+
+
+@when("service.configured")
+def check_is_running():
+    hookenv.status_set("active", "Ready")
+
+
+@hook("upgrade-charm")
+def upgrade_charm():
+    clear_flag("service.configured")
+
+
+@when("config.changed")
+def config_changed():
+    clear_flag("service.configured")
+
+
+@when("apache-website.available", "service.configured")
+@when_not("service.apache-website.configured")
+def configure_apache_website():
+    apache_website = endpoint_from_flag("apache-website.available")
+    config = dict(hookenv.config())
+    config["www_dir"] = www_dir()
+    apache_website.set_remote(
+        domain=config["domain_cron_control"],
+        enabled="true",
+        ports="80",
+        site_config=templating.render(
+            "vhosts/cron-http.conf.j2", None, config
+        ),
+    )
+    set_flag("service.apache-website.configured")
+
+
+@when("service.apache-website.configured")
+@when_not_all("apache-website.available", "service.configured")
+def deconfigure_apache_website():
+    clear_flag("service.apache-website.configured")
diff --git a/charm/launchpad-cron-control/templates/vhosts/cron-http.conf.j2 b/charm/launchpad-cron-control/templates/vhosts/cron-http.conf.j2
new file mode 100644
index 0000000..1525ef2
--- /dev/null
+++ b/charm/launchpad-cron-control/templates/vhosts/cron-http.conf.j2
@@ -0,0 +1,13 @@
+<VirtualHost *:80>
+    ServerName {{ domain_cron_control }}
+    ServerAdmin {{ webmaster_email }}
+
+    CustomLog /var/log/apache2/{{ domain_cron_control }}-access.log combined
+    ErrorLog /var/log/apache2/{{ domain_cron_control }}-error.log
+
+    DocumentRoot "{{ www_dir }}"
+    <Directory "{{ www_dir }}/">
+        Require all granted
+    </Directory>
+</VirtualHost>
+