launchpad-reviewers team mailing list archive

[Colin Watson <mp+444200@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:charm-librarian-http-interface into launchpad:master.

Commit message:
charm/launchpad-librarian: Add http interface support

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/444200

This allows relating `launchpad-librarian` to a deployment of the `haproxy` charm to have it load-balance the various workers and expose them on the ports expected by other parts of Launchpad.

The code is based on the `ols-http` layer, but I had to write custom code for it because the librarian is rather special in a few ways: it has four different sets of ports with different purposes (download, upload, restricted download, and restricted upload), and multiple workers per set; and the current production `haproxy` configuration uses somewhat different options for the download vs. upload workers.

While this doesn't result in the exact same `haproxy` configuration that we have on production, I think it's equivalent.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:charm-librarian-http-interface into launchpad:master.

diff --git a/charm/launchpad-librarian/README.md b/charm/launchpad-librarian/README.md
index 52ab88c..a67512b 100644
--- a/charm/launchpad-librarian/README.md
+++ b/charm/launchpad-librarian/README.md
@@ -8,6 +8,11 @@ You will need the following relations:
     juju relate launchpad-librarian:session-db postgresql:db
     juju relate launchpad-librarian rabbitmq-server
 
+You can also relate it to a load balancer, which is especially useful if you
+set `workers` to something other than 1:
+
+    juju relate launchpad-librarian:loadbalancer haproxy:reverseproxy
+
 The librarian listens on four ports.  By default, these are:
 
 - Public download: 8000
diff --git a/charm/launchpad-librarian/config.yaml b/charm/launchpad-librarian/config.yaml
index 0a46990..937f30d 100644
--- a/charm/launchpad-librarian/config.yaml
+++ b/charm/launchpad-librarian/config.yaml
@@ -3,6 +3,28 @@ options:
     type: boolean
     default: true
     description: If true, enable jobs that may change the database.
+  haproxy_server_options:
+    type: string
+    description: Options to add to HAProxy "server" lines.
+    default: check inter 5000 rise 2 fall 5 maxconn 16
+  haproxy_service_options_download:
+    type: string
+    description: HAProxy options for download services.
+    default: |
+      - mode http
+      - option httplog
+      - option httpchk HEAD / HTTP/1.0
+      - balance leastconn
+  haproxy_service_options_upload:
+    type: string
+    description: HAProxy options for upload services.
+    default: |
+      - mode tcp
+      - option tcplog
+      - option httpchk HEAD / HTTP/1.0
+      - balance leastconn
+      - timeout client 600000
+      - timeout server 600000
   old_os_auth_url:
     type: string
     description: >
diff --git a/charm/launchpad-librarian/layer.yaml b/charm/launchpad-librarian/layer.yaml
index cdfe1c7..ce520c3 100644
--- a/charm/launchpad-librarian/layer.yaml
+++ b/charm/launchpad-librarian/layer.yaml
@@ -1,5 +1,6 @@
 includes:
   - layer:launchpad-db
+  - interface:http
 repo: https://git.launchpad.net/launchpad
 options:
   apt:
diff --git a/charm/launchpad-librarian/metadata.yaml b/charm/launchpad-librarian/metadata.yaml
index ed17f6e..5fb0de8 100644
--- a/charm/launchpad-librarian/metadata.yaml
+++ b/charm/launchpad-librarian/metadata.yaml
@@ -16,3 +16,6 @@ subordinate: false
 requires:
   session-db:
     interface: pgsql
+provides:
+  loadbalancer:
+    interface: http
diff --git a/charm/launchpad-librarian/reactive/launchpad-librarian.py b/charm/launchpad-librarian/reactive/launchpad-librarian.py
index 67cc398..2df140a 100644
--- a/charm/launchpad-librarian/reactive/launchpad-librarian.py
+++ b/charm/launchpad-librarian/reactive/launchpad-librarian.py
@@ -4,6 +4,7 @@
 import os.path
 import subprocess
 
+import yaml
 from charmhelpers.core import hookenv, host, templating
 from charms.launchpad.base import configure_email, get_service_config
 from charms.launchpad.db import (
@@ -167,3 +168,125 @@ def deconfigure():
 def session_db_changed():
     remove_state("service.configured")
     remove_state("session-db.database.changed")
+
+
+@when(
+    "config.set.librarian_download_port",
+    "config.set.librarian_restricted_download_port",
+    "config.set.librarian_restricted_upload_port",
+    "config.set.librarian_upload_port",
+    "loadbalancer.available",
+    "service.configured",
+)
+@when_not("launchpad.loadbalancer.configured")
+def configure_loadbalancer():
+    config = hookenv.config()
+
+    try:
+        service_options_download = yaml.safe_load(
+            config["haproxy_service_options_download"]
+        )
+    except Exception:
+        hookenv.log("Could not parse haproxy_service_options_download YAML")
+        hookenv.status_set(
+            "blocked",
+            "Bad haproxy_service_options_download YAML configuration",
+        )
+        return
+    try:
+        service_options_upload = yaml.safe_load(
+            config["haproxy_service_options_upload"]
+        )
+    except Exception:
+        hookenv.log("Could not parse haproxy_service_options_upload YAML")
+        hookenv.status_set(
+            "blocked", "Bad haproxy_service_options_upload YAML configuration"
+        )
+        return
+    server_options = config["haproxy_server_options"]
+
+    unit_name = hookenv.local_unit().replace("/", "-")
+    unit_ip = hookenv.unit_private_ip()
+    services = [
+        {
+            "service_name": "librarian-download",
+            "service_port": config["librarian_download_port"],
+            "service_host": "0.0.0.0",
+            "service_options": list(service_options_download),
+            "servers": [
+                [
+                    f"dl_{unit_name}_{i + 1}",
+                    unit_ip,
+                    config["port_download_base"] + i,
+                    server_options,
+                ]
+                for i in range(config["workers"])
+            ],
+        },
+        {
+            "service_name": "librarian-upload",
+            "service_port": config["librarian_upload_port"],
+            "service_host": "0.0.0.0",
+            "service_options": list(service_options_upload),
+            "servers": [
+                [
+                    f"ul_{unit_name}_{i + 1}",
+                    unit_ip,
+                    config["port_upload_base"] + i,
+                    f"port {config['port_download_base'] + i} "
+                    + server_options,
+                ]
+                for i in range(config["workers"])
+            ],
+        },
+        {
+            "service_name": "librarian-restricted-download",
+            "service_port": config["librarian_restricted_download_port"],
+            "service_host": "0.0.0.0",
+            "service_options": list(service_options_download),
+            "servers": [
+                [
+                    f"dl_restricted_{unit_name}_{i + 1}",
+                    unit_ip,
+                    config["port_restricted_download_base"] + i,
+                    server_options,
+                ]
+                for i in range(config["workers"])
+            ],
+        },
+        {
+            "service_name": "librarian-restricted-upload",
+            "service_port": config["librarian_restricted_upload_port"],
+            "service_host": "0.0.0.0",
+            "service_options": list(service_options_upload),
+            "servers": [
+                [
+                    f"ul_restricted_{unit_name}_{i + 1}",
+                    unit_ip,
+                    config["port_restricted_upload_base"] + i,
+                    f"port {config['port_restricted_download_base'] + i} "
+                    + server_options,
+                ]
+                for i in range(config["workers"])
+            ],
+        },
+    ]
+    services_yaml = yaml.dump(services)
+
+    for rel in hookenv.relations_of_type("loadbalancer"):
+        hookenv.relation_set(rel["__relid__"], services=services_yaml)
+
+    set_state("launchpad.loadbalancer.configured")
+
+
+@when("launchpad.loadbalancer.configured")
+@when_not_all(
+    "config.set.librarian_download_port",
+    "config.set.librarian_restricted_download_port",
+    "config.set.librarian_restricted_upload_port",
+    "config.set.librarian_upload_port",
+    "loadbalancer.available",
+    "service.configured",
+)
+def deconfigure_loadbalancer():
+    remove_state("launchpad.loadbalancer.configured")