launchpad-reviewers team mailing list archive

[Colin Watson <mp+447543@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:charm-librarian-frontend-relations into launchpad:master.

Commit message:
charm: Set up frontend relations for the librarian

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/447543

The Apache configuration files here are lifted fairly directly from production, although qastaging and staging are much the same.  The `service_name` settings are passed through the stack of Juju relations (with Squid and Apache) and can be used in Apache `balancer://` URLs with no extra configuration required.

At the moment, the public and restricted librarians are on different sets of IP addresses.  I _think_ that this was mostly due to historical issues around issuing TLS certificates that are no longer an issue, so I'm going to try to get things working with a single set of public IP addresses on (qa)staging; if that doesn't work out then I'll fall back to splitting the `vhost-config` relation into two and deploying two Apache applications, but it doesn't end up being all that fundamentally different.

This depends on https://code.launchpad.net/~cjwatson/launchpad-layers/+git/launchpad-layers/+merge/447537, and before I land this I'd expect to fix up `source` and `source-commit` and make sure that all our charms stay in sync.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:charm-librarian-frontend-relations into launchpad:master.

diff --git a/charm/launchpad-librarian/charmcraft.yaml b/charm/launchpad-librarian/charmcraft.yaml
index 77e67e8..569c452 100644
--- a/charm/launchpad-librarian/charmcraft.yaml
+++ b/charm/launchpad-librarian/charmcraft.yaml
@@ -34,12 +34,13 @@ parts:
   launchpad-layers:
     after:
       - ols-layers
-    source: https://git.launchpad.net/launchpad-layers
-    source-commit: "42a4b4c4f62936b1d050c775e84f7364dfb5efc0"
+    source: https://git.launchpad.net/~cjwatson/launchpad-layers
+    source-commit: "8b7d25a29b8297d491abe6a0c6b69f34e39dddba"
     source-submodules: []
     source-type: git
     plugin: dump
     organize:
+      apache-vhost-config: layers/interface/apache-vhost-config
       launchpad-base: layers/layer/launchpad-base
       launchpad-db: layers/layer/launchpad-db
       launchpad-payload: layers/layer/launchpad-payload
diff --git a/charm/launchpad-librarian/config.yaml b/charm/launchpad-librarian/config.yaml
index 937f30d..b48a704 100644
--- a/charm/launchpad-librarian/config.yaml
+++ b/charm/launchpad-librarian/config.yaml
@@ -3,6 +3,11 @@ options:
     type: boolean
     default: true
     description: If true, enable jobs that may change the database.
+  domain_librarian_aliases:
+    type: string
+    default: "[]"
+    description: >
+      A YAML-encoded list of aliases for the librarian's primary domain.
   haproxy_server_options:
     type: string
     description: Options to add to HAProxy "server" lines.
diff --git a/charm/launchpad-librarian/layer.yaml b/charm/launchpad-librarian/layer.yaml
index ce520c3..6efe850 100644
--- a/charm/launchpad-librarian/layer.yaml
+++ b/charm/launchpad-librarian/layer.yaml
@@ -1,5 +1,6 @@
 includes:
   - layer:launchpad-db
+  - interface:apache-vhost-config
   - interface:http
 repo: https://git.launchpad.net/launchpad
 options:
diff --git a/charm/launchpad-librarian/metadata.yaml b/charm/launchpad-librarian/metadata.yaml
index 5fb0de8..f114ffa 100644
--- a/charm/launchpad-librarian/metadata.yaml
+++ b/charm/launchpad-librarian/metadata.yaml
@@ -19,3 +19,5 @@ requires:
 provides:
   loadbalancer:
     interface: http
+  vhost-config:
+    interface: apache-vhost-config
diff --git a/charm/launchpad-librarian/reactive/launchpad-librarian.py b/charm/launchpad-librarian/reactive/launchpad-librarian.py
index 516e537..be74fe0 100644
--- a/charm/launchpad-librarian/reactive/launchpad-librarian.py
+++ b/charm/launchpad-librarian/reactive/launchpad-librarian.py
@@ -3,6 +3,7 @@
 
 import os.path
 import subprocess
+from urllib.parse import urlparse
 
 import yaml
 from charmhelpers.core import hookenv, host, templating
@@ -210,7 +211,7 @@ def configure_loadbalancer():
     unit_ip = hookenv.unit_private_ip()
     services = [
         {
-            "service_name": config["librarian_download_host"],
+            "service_name": "launchpad_librarian_download",
             "service_port": config["librarian_download_port"],
             "service_host": "0.0.0.0",
             "service_options": list(service_options_download),
@@ -225,7 +226,7 @@ def configure_loadbalancer():
             ],
         },
         {
-            "service_name": config["librarian_upload_host"],
+            "service_name": "launchpad_librarian_upload",
             "service_port": config["librarian_upload_port"],
             "service_host": "0.0.0.0",
             "service_options": list(service_options_upload),
@@ -241,7 +242,7 @@ def configure_loadbalancer():
             ],
         },
         {
-            "service_name": config["librarian_restricted_download_host"],
+            "service_name": "launchpad_librarian_restricted_download",
             "service_port": config["librarian_restricted_download_port"],
             "service_host": "0.0.0.0",
             "service_options": list(service_options_download),
@@ -256,7 +257,7 @@ def configure_loadbalancer():
             ],
         },
         {
-            "service_name": config["librarian_restricted_upload_host"],
+            "service_name": "launchpad_librarian_restricted_upload",
             "service_port": config["librarian_restricted_upload_port"],
             "service_host": "0.0.0.0",
             "service_options": list(service_options_upload),
@@ -291,3 +292,41 @@ def configure_loadbalancer():
 )
 def deconfigure_loadbalancer():
     remove_state("launchpad.loadbalancer.configured")
+
+
+@when(
+    "config.set.librarian_download_url",
+    "vhost-config.available",
+    "service.configured",
+)
+@when_not("launchpad.vhost.configured")
+def configure_vhost():
+    vhost_config = endpoint_from_flag("vhost-config.available")
+    config = dict(hookenv.config())
+    config["domain_librarian"] = urlparse(
+        config["librarian_download_url"]
+    ).hostname
+    config["domain_librarian_aliases"] = yaml.safe_load(
+        config["domain_librarian_aliases"]
+    )
+    vhost_config.publish_vhosts(
+        [
+            vhost_config.make_vhost(
+                80, templating.render("librarian-http.conf.j2", None, config)
+            ),
+            vhost_config.make_vhost(
+                443, templating.render("librarian-https.conf.j2", None, config)
+            ),
+        ]
+    )
+    set_state("launchpad.vhost.configured")
+
+
+@when("launchpad.vhost.configured")
+@when_not_all(
+    "config.set.librarian_download_url",
+    "vhost-config.available",
+    "service.configured",
+)
+def deconfigure_vhost():
+    remove_state("launchpad.vhost.configured")
diff --git a/charm/launchpad-librarian/templates/librarian-http.conf.j2 b/charm/launchpad-librarian/templates/librarian-http.conf.j2
new file mode 100644
index 0000000..2b18e2d
--- /dev/null
+++ b/charm/launchpad-librarian/templates/librarian-http.conf.j2
@@ -0,0 +1,36 @@
+<VirtualHost *:80>
+    ServerName {{ domain_librarian }}
+{%- for domain_librarian_alias in domain_librarian_aliases %}
+    ServerAlias {{ domain_librarian_alias }}
+{%- endfor %}
+
+    CustomLog /var/log/apache2/{{ domain_librarian }}-access.log combined
+    ErrorLog /var/log/apache2/{{ domain_librarian }}-error.log
+
+    ProxyRequests off
+    <Proxy *>
+        Require all granted
+    </Proxy>
+
+    SetEnv force-proxy-request-1.0 1
+
+    # Tokens may only be used on restricted librarian URLs.
+    ProxyPass "\?.*token=" !
+
+    ProxyPreserveHost on
+    ProxyPass / balancer://launchpad_librarian_download/
+    ProxyPassReverse / balancer://launchpad_librarian_download/
+</VirtualHost>
+
+<VirtualHost *:80>
+    ServerName wildcard.restricted.{{ domain_librarian }}
+    ServerAlias *.restricted.{{ domain_librarian }}
+
+    CustomLog /var/log/apache2/wildcard.restricted.{{ domain_librarian }}-access.log combined
+    ErrorLog /var/log/apache2/wildcard.restricted.{{ domain_librarian }}-error.log
+
+    # The restricted librarian is only available over HTTPS.
+    RewriteEngine on
+    RewriteRule ^/(.*)$ - [R=403,L]
+</VirtualHost>
+
diff --git a/charm/launchpad-librarian/templates/librarian-https.conf.j2 b/charm/launchpad-librarian/templates/librarian-https.conf.j2
new file mode 100644
index 0000000..12b22b1
--- /dev/null
+++ b/charm/launchpad-librarian/templates/librarian-https.conf.j2
@@ -0,0 +1,67 @@
+<VirtualHost *:443>
+    ServerName {{ domain_librarian }}
+{%- for domain_librarian_alias in domain_librarian_aliases %}
+    ServerAlias {{ domain_librarian_alias }}
+{%- endfor %}
+
+    SSLEngine on
+    SSLCertificateFile /etc/ssl/certs/{{ domain_librarian }}.crt
+    SSLCertificateKeyFile /etc/ssl/private/{{ domain_librarian }}.key
+
+    CustomLog /var/log/apache2/{{ domain_librarian }}-access.log combined
+    ErrorLog /var/log/apache2/{{ domain_librarian }}-error.log
+
+    # Make build log files auto-decompress and be viewable from the browser.
+    <Location ~ ".*/buildlog_[^/]*\.txt\.gz">
+        AddEncoding x-gzip gz
+    </Location>
+
+    SetEnv force-proxy-request-1.0 1
+
+    ProxyRequests off
+    <Proxy *>
+        Require all granted
+    </Proxy>
+
+    # Tokens may only be used on restricted librarian URLs.
+    ProxyPass "\?.*token=" !
+
+    ProxyPreserveHost on
+    ProxyPass / balancer://launchpad_librarian_download/
+    ProxyPassReverse / balancer://launchpad_librarian_download/
+</VirtualHost>
+
+<VirtualHost *:443>
+    ServerName wildcard.restricted.{{ domain_librarian }}
+    ServerAlias *.restricted.{{ domain_librarian }}
+
+    SSLEngine on
+    SSLCertificateFile /etc/ssl/certs/{{ domain_librarian }}.crt
+    SSLCertificateKeyFile /etc/ssl/private/{{ domain_librarian }}.key
+
+    CustomLog /var/log/apache2/{{ domain_librarian }}-access.log combined
+    ErrorLog /var/log/apache2/{{ domain_librarian }}-error.log
+
+    # Make build log files auto-decompress and be viewable from the browser.
+    <Location ~ ".*/buildlog_[^/]*\.txt\.gz">
+        AddEncoding x-gzip gz
+    </Location>
+
+    SetEnv force-proxy-request-1.0 1
+
+    ProxyRequests off
+    <Proxy *>
+        Require all granted
+    </Proxy>
+
+    ProxyPreserveHost on
+    # nocanon per https://portal.admin.canonical.com/C42560 to avoid
+    # problems with Launchpad's handling of e.g. %2B.
+    ProxyPass / balancer://launchpad_librarian_download/ nocanon
+    ProxyPassReverse / balancer://launchpad_librarian_download/
+
+    <Location />
+        Header set Cache-Control "max-age=604800"
+    </Location>
+</VirtualHost>
+