launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #30380
[Merge] ~ines-almeida/launchpad:add-ppa-publisher-charm into launchpad:master
Ines Almeida has proposed merging ~ines-almeida/launchpad:add-ppa-publisher-charm into launchpad:master.
Commit message:
Add ppa-publisher and ppa-uploader charms
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~ines-almeida/launchpad/+git/launchpad/+merge/447663
For context:
- The ppa-uploader has been tested and manages to accept a PPA through ftp, process it and create an entry in the DB for it
- The ppa-publisher charm is not 100% tested yet (I'm in the process of it), so it might still need some tweaks
--
Your team Launchpad code reviewers is requested to review the proposed merge of ~ines-almeida/launchpad:add-ppa-publisher-charm into launchpad:master.
diff --git a/charm/launchpad-ppa-publisher/README.md b/charm/launchpad-ppa-publisher/README.md
new file mode 100644
index 0000000..c4d13e8
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/README.md
@@ -0,0 +1,13 @@
+# Launchpad PPA Uploader
+
+This charm runs a Launchpad PPA publisher. Takes all PPAs that are pending
+in the database (added there by a launchpad-ppa-uploader unit) and arranges
+for contents of the disk to match.
+
+You will need the following relations:
+
+```
+ juju relate launchpad-ppa-publisher:db postgresql:db
+ juju relate launchpad-ppa-publisher rabbitmq-server
+ juju relate launchpad-ppa-publisher nrpe
+```
diff --git a/charm/launchpad-ppa-publisher/charmcraft.yaml b/charm/launchpad-ppa-publisher/charmcraft.yaml
new file mode 100644
index 0000000..9b3bb09
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/charmcraft.yaml
@@ -0,0 +1,62 @@
+type: charm
+bases:
+ - build-on:
+ - name: ubuntu
+ channel: "20.04"
+ architectures: [amd64]
+ run-on:
+ - name: ubuntu
+ channel: "20.04"
+ architectures: [amd64]
+parts:
+ charm-wheels:
+ source: https://git.launchpad.net/~ubuntuone-hackers/ols-charm-deps/+git/wheels
+ source-commit: "59b32ae07f98051385c96d6d8e7e02ca4f197fe5"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ "*": charm-wheels/
+ prime:
+ - "-charm-wheels"
+ ols-layers:
+ source: https://git.launchpad.net/ols-charm-deps
+ source-commit: "56d219f60a293a6c73759b8439ef5fdb11e19d1f"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ "*": layers/
+ stage:
+ - layers
+ prime:
+ - "-layers"
+ launchpad-layers:
+ after:
+ - ols-layers
+ source: https://git.launchpad.net/launchpad-layers
+ source-commit: "fe273e9f446f84243be2070334cacd03c09da946"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ launchpad-base: layers/layer/launchpad-base
+ launchpad-db: layers/layer/launchpad-db
+ launchpad-payload: layers/layer/launchpad-payload
+ stage:
+ - layers
+ prime:
+ - "-layers"
+ charm:
+ after:
+ - charm-wheels
+ - launchpad-layers
+ source: .
+ plugin: reactive
+ build-snaps: [charm]
+ build-packages: [libpq-dev, python3-dev]
+ build-environment:
+ - CHARM_LAYERS_DIR: $CRAFT_STAGE/layers/layer
+ - CHARM_INTERFACES_DIR: $CRAFT_STAGE/layers/interface
+ - PIP_NO_INDEX: "true"
+ - PIP_FIND_LINKS: $CRAFT_STAGE/charm-wheels
diff --git a/charm/launchpad-ppa-publisher/config.yaml b/charm/launchpad-ppa-publisher/config.yaml
new file mode 100644
index 0000000..41f2667
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/config.yaml
@@ -0,0 +1,25 @@
+options:
+ active:
+ type: boolean
+ default: true
+ description: If true, enable jobs that may change the database.
+ oval_data_rsync_endpoint:
+ type: string
+ default:
+ description: The rsync endpoint to use for synchronizing the OVAL data.
+ require_signing_keys:
+ type: boolean
+ description: If true, PPAs and copy archives will only be published once their signing-keys have been generated.
+ default: True
+ logparser_max_parsed_lines:
+ type: int
+ default: 10000000
+ description: The maximum number of lines that should be parsed by the launchpad log parser. The default value of None means there is no maximum.
+ signing_client_public_key:
+ type: string
+ description: Public key for encrypting communication between client and signing service.
+ default: ""
+ storm_cache_size:
+ type: int
+ default: 500000
+ description: The size of the Storm cache in objects.
diff --git a/charm/launchpad-ppa-publisher/layer.yaml b/charm/launchpad-ppa-publisher/layer.yaml
new file mode 100644
index 0000000..e7ab214
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/layer.yaml
@@ -0,0 +1,12 @@
+includes:
+ - layer:launchpad-db
+repo: https://git.launchpad.net/launchpad
+options:
+ ols-pg:
+ databases:
+ db:
+ name: launchpad_dev
+ roles:
+ - generate_contents_files
+ - process_death_row
+ - generateppahtaccess
diff --git a/charm/launchpad-ppa-publisher/metadata.yaml b/charm/launchpad-ppa-publisher/metadata.yaml
new file mode 100644
index 0000000..800054a
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/metadata.yaml
@@ -0,0 +1,17 @@
+name: launchpad-ppa-publisher
+display-name: launchpad-ppa-publisher
+summary: Launchpad PPA publisher
+maintainer: Launchpad Developers <launchpad-dev@xxxxxxxxxxxxxxxxxxx>
+description: |
+ Launchpad is an open source suite of tools that help people and teams
+ to work together on software projects.
+
+ This charm runs a Launchpad PPA publisher. Takes all PPAs that are pending
+ in the database (added there by a launchpad-ppa-uploader unit) and arranges
+ for contents of the disk to match.
+tags:
+ - network
+series:
+ - focal
+subordinate: false
+
diff --git a/charm/launchpad-ppa-publisher/reactive/launchpad-ppa-publisher.py b/charm/launchpad-ppa-publisher/reactive/launchpad-ppa-publisher.py
new file mode 100644
index 0000000..7c779d3
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/reactive/launchpad-ppa-publisher.py
@@ -0,0 +1,77 @@
+# Copyright 2023 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os
+
+from charmhelpers.core import hookenv, host, templating
+from charms.launchpad.base import configure_email, get_service_config
+from charms.launchpad.payload import configure_cron, configure_lazr
+from charms.reactive import (
+ remove_state,
+ set_state,
+ when,
+ when_not,
+ when_not_all,
+)
+from ols import base
+
+
+def configure_logrotate(config):
+ hookenv.log("Writing logrotate configuration.")
+ templating.render(
+ "logrotate.conf.j2",
+ "/etc/logrotate.d/launchpad",
+ config,
+ perms=0o644,
+ )
+
+
+@when(
+ "launchpad.base.configured",
+ "launchpad.db.configured",
+)
+@when_not("service.configured")
+def configure():
+ hookenv.log("Configuring ppa publisher")
+ config = get_service_config()
+ ppa_archive_root = os.path.join(base.base_dir(), "ppa-archive")
+ ppa_archive_private = os.path.join(base.base_dir(), "private-ppa-archive")
+ ppa_keys_root = os.path.join(base.base_dir(), "ppa-signing-keys")
+
+ config["ppa_archive_root"] = ppa_archive_root
+ config["ppa_archive_private_root"] = ppa_archive_private
+ config["ppa_signing_keys_root"] = ppa_keys_root
+
+ host.mkdir(
+ ppa_archive_root, owner=base.user(), group=base.user(), perms=0o775
+ )
+ host.mkdir(
+ ppa_archive_private, owner=base.user(), group=base.user(), perms=0o775
+ )
+ host.mkdir(
+ ppa_keys_root, owner=base.user(), group=base.user(), perms=0o775
+ )
+
+ configure_lazr(
+ config,
+ "launchpad-ppa-publisher-lazr.conf.j2",
+ "launchpad-ppa-publisher/launchpad-lazr.conf",
+ )
+ configure_email(config, "launchpad-ppa-publisher")
+ configure_logrotate(config)
+ configure_cron(config, "crontab.j2")
+ set_state("service.configured")
+
+
+@when("service.configured")
+def check_is_running():
+ hookenv.status_set("active", "Ready")
+
+
+@when("service.configured")
+@when_not_all(
+ "launchpad.base.configured",
+ "launchpad.db.configured",
+)
+def deconfigure():
+ remove_state("service.configured")
diff --git a/charm/launchpad-ppa-publisher/templates/crontab.j2 b/charm/launchpad-ppa-publisher/templates/crontab.j2
new file mode 100644
index 0000000..cc69fb6
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/templates/crontab.j2
@@ -0,0 +1,17 @@
+# m h dom mon dow command
+MAILTO={{ cron_mailto }}
+LPCONFIG=launchpad-ppa-publisher
+
+{% if active -%}
+* * * * * umask 022; nice -n 5 ionice -c 2 -n 7 {{ code_dir }}/cronscripts/publishing/cron.publish-ppa "-d ubuntu" "--all-derived" "-d debian" >> {{ logs_dir }}/cron.ppa.log 2>&1
+
+59 05 * * 0 {{ code_dir }}/cronscripts/publishing/cron.daily-ppa >> {{ logs_dir }}/cron.ppa.log 2>&1
+0 */6 * * * nice -n 17 {{ code_dir }}/scripts/process-death-row.py -d ubuntu --ppa -q --log-file=INFO:{{ logs_dir }}/process-death-row.log
+0 * * * * nice -n 17 {{ code_dir }}/scripts/process-death-row.py --all-derived --ppa -q --log-file=INFO:{{ logs_dir }}/derived-process-death-row.log
+*/20 * * * * nice -n 12 ionice -c 2 -n 7 {{ code_dir }}/cronscripts/ppa-generate-keys.py -q --log-file=INFO:{{ logs_dir }}/ppa-generate-keys.log
+* * * * * {{ code_dir }}/cronscripts/generate-ppa-htaccess.py >> {{ logs_dir }}/generate-ppa-htaccess-wrapper.log 2>&1
+
+# OOPS amqp
+*/15 * * * * {{ code_dir }}/bin/datedir2amqp --exchange oopses --host {{ rabbitmq_host }} --username {{ rabbitmq_username }} --password {{ rabbitmq_password }} --vhost {{ rabbitmq_vhost }} --repo {{ oopses_dir }} --key ""
+{% endif %}
+
diff --git a/charm/launchpad-ppa-publisher/templates/launchpad-ppa-publisher-lazr.conf.j2 b/charm/launchpad-ppa-publisher/templates/launchpad-ppa-publisher-lazr.conf.j2
new file mode 100644
index 0000000..24932e4
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/templates/launchpad-ppa-publisher-lazr.conf.j2
@@ -0,0 +1,34 @@
+# Public configuration data. The contents of this file may be freely shared
+# with developers if needed for debugging.
+
+# A schema's sections, keys, and values are automatically inherited,
+# except for '.optional' sections. Update this config to override key
+# values. Values are strings, except for numbers that look like ints.
+# The tokens true, false, and none are treated as True, False, and None.
+
+{% from "macros.j2" import opt -%}
+
+[meta]
+extends: ../launchpad-db-lazr.conf
+
+[archivepublisher]
+oval_data_rsync_endpoint: {{ oval_data_rsync_endpoint }}
+
+[personalpackagearchive]
+root: {{ ppa_archive_root }}
+private_root: {{ ppa_archive_private_root }}
+signing_keys_root: {{ ppa_signing_keys_root }}
+require_signing_keys: {{ require_signing_keys }}
+
+[ppa_apache_log_parser]
+logs_root: {{ logs_dir }}/apache
+
+[launchpad]
+{{- opt("logparser_max_parsed_lines", logparser_max_parsed_lines) }}
+
+[signing]
+client_public_key: {{ signing_client_public_key }}
+
+[database]
+storm_cache_size: {{ storm_cache_size }}
+
diff --git a/charm/launchpad-ppa-publisher/templates/logrotate.conf.j2 b/charm/launchpad-ppa-publisher/templates/logrotate.conf.j2
new file mode 100644
index 0000000..494f246
--- /dev/null
+++ b/charm/launchpad-ppa-publisher/templates/logrotate.conf.j2
@@ -0,0 +1,16 @@
+{{ logs_dir }}/launchpad.log
+{
+ rotate 21
+ daily
+ dateext
+ delaycompress
+ compress
+ notifempty
+ missingok
+ create 0644 syslog adm
+ sharedscripts
+ postrotate
+ invoke-rc.d rsyslog rotate >/dev/null
+ endscript
+}
+
diff --git a/charm/launchpad-ppa-uploader/README.md b/charm/launchpad-ppa-uploader/README.md
new file mode 100644
index 0000000..05b5046
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/README.md
@@ -0,0 +1,13 @@
+# Launchpad PPA Uploader
+
+Processes the PPA uploads requested by users, and temporarily stores them.
+
+You will need the following relations:
+
+```
+ juju relate launchpad-ppa-uploader:db postgresql:db
+ juju relate launchpad-ppa-uploader memcached
+ juju relate launchpad-ppa-uploader nrpe
+ juju relate launchpad-ppa-uploader rabbitmq-server
+ juju relate launchpad-ppa-uploader txpkgupload
+```
\ No newline at end of file
diff --git a/charm/launchpad-ppa-uploader/charmcraft.yaml b/charm/launchpad-ppa-uploader/charmcraft.yaml
new file mode 100644
index 0000000..0be4ca9
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/charmcraft.yaml
@@ -0,0 +1,63 @@
+type: charm
+bases:
+ - build-on:
+ - name: ubuntu
+ channel: "20.04"
+ architectures: [amd64]
+ run-on:
+ - name: ubuntu
+ channel: "20.04"
+ architectures: [amd64]
+parts:
+ charm-wheels:
+ source: https://git.launchpad.net/~ubuntuone-hackers/ols-charm-deps/+git/wheels
+ source-commit: "59b32ae07f98051385c96d6d8e7e02ca4f197fe5"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ "*": charm-wheels/
+ prime:
+ - "-charm-wheels"
+ ols-layers:
+ source: https://git.launchpad.net/ols-charm-deps
+ source-commit: "56d219f60a293a6c73759b8439ef5fdb11e19d1f"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ "*": layers/
+ stage:
+ - layers
+ prime:
+ - "-layers"
+ launchpad-layers:
+ after:
+ - ols-layers
+ source: https://git.launchpad.net/launchpad-layers
+ source-commit: "300b0d9fd332c055395fe209512335cea03c7af3"
+ source-submodules: []
+ source-type: git
+ plugin: dump
+ organize:
+ launchpad-base: layers/layer/launchpad-base
+ launchpad-db: layers/layer/launchpad-db
+ launchpad-payload: layers/layer/launchpad-payload
+ upload-queue-processor: layers/interface/upload-queue-processor
+ stage:
+ - layers
+ prime:
+ - "-layers"
+ charm:
+ after:
+ - charm-wheels
+ - launchpad-layers
+ source: .
+ plugin: reactive
+ build-snaps: [charm]
+ build-packages: [libpq-dev, python3-dev]
+ build-environment:
+ - CHARM_LAYERS_DIR: $CRAFT_STAGE/layers/layer
+ - CHARM_INTERFACES_DIR: $CRAFT_STAGE/layers/interface
+ - PIP_NO_INDEX: "true"
+ - PIP_FIND_LINKS: $CRAFT_STAGE/charm-wheels
diff --git a/charm/launchpad-ppa-uploader/config.yaml b/charm/launchpad-ppa-uploader/config.yaml
new file mode 100644
index 0000000..c5001b0
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/config.yaml
@@ -0,0 +1,33 @@
+options:
+ active:
+ type: boolean
+ default: true
+ description: If true, enable jobs that may change the database.
+ ppa_api_endpoint:
+ type: string
+ default: http://xmlrpc-private.launchpad.test:8087/archive
+ description: Private Package archive API endpoint
+ domain:
+ type: string
+ description: Domain name for this instance.
+ default: "launchpad.test"
+ oval_data_rsync_endpoint:
+ type: string
+ default:
+ description: The rsync endpoint to use for synchronizing the OVAL data.
+ require_signing_keys:
+ type: boolean
+ default: True
+ description: If true, PPAs and copy archives will only be published once their signing-keys have been generated.
+ logparser_max_parsed_lines:
+ type: int
+ default: 10000000
+ description: The maximum number of lines that should be parsed by the launchpad log parser. The default value of None means there is no maximum.
+ signing_endpoint:
+ type: string
+ default:
+ description: Endpoint for the signing service.
+ signing_client_public_key:
+ type: string
+ default:
+ description: Public key for encrypting communication between client and signing service.
diff --git a/charm/launchpad-ppa-uploader/layer.yaml b/charm/launchpad-ppa-uploader/layer.yaml
new file mode 100644
index 0000000..c146aae
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/layer.yaml
@@ -0,0 +1,15 @@
+includes:
+ - layer:launchpad-db
+ - interface:memcache
+ - interface:upload-queue-processor
+repo: https://git.launchpad.net/launchpad
+options:
+ ols-pg:
+ databases:
+ db:
+ name: launchpad_dev
+ roles:
+ - process_upload
+ - archivepublisher
+ - package_diff_job
+ - ppa-apache-log-parser
diff --git a/charm/launchpad-ppa-uploader/metadata.yaml b/charm/launchpad-ppa-uploader/metadata.yaml
new file mode 100644
index 0000000..30ee33f
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/metadata.yaml
@@ -0,0 +1,23 @@
+name: launchpad-ppa-uploader
+display-name: launchpad-ppa-uploader
+summary: Launchpad PPA uploader
+maintainer: Launchpad Developers <launchpad-dev@xxxxxxxxxxxxxxxxxxx>
+description: |
+ Launchpad is an open source suite of tools that help people and teams
+ to work together on software projects.
+
+ This charm runs a Launchpad PPA upload processor, which processes the PPA
+ uploads requested by users, temporarily stores them, and puts them into
+ the database.
+tags:
+ - network
+series:
+ - focal
+subordinate: false
+requires:
+ memcache:
+ interface: memcache
+provides:
+ upload-queue-processor:
+ interface: upload-queue-processor
+ scope: container
diff --git a/charm/launchpad-ppa-uploader/reactive/launchpad-ppa-uploader.py b/charm/launchpad-ppa-uploader/reactive/launchpad-ppa-uploader.py
new file mode 100644
index 0000000..50c9c8a
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/reactive/launchpad-ppa-uploader.py
@@ -0,0 +1,115 @@
+# Copyright 2023 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os
+
+from charmhelpers.core import hookenv, host, templating
+from charms.launchpad.base import configure_email, get_service_config
+from charms.launchpad.payload import configure_cron, configure_lazr
+from charms.reactive import (
+ clear_flag,
+ endpoint_from_flag,
+ hook,
+ remove_state,
+ set_flag,
+ when,
+ when_not,
+ when_not_all,
+)
+from ols import base
+
+
+def configure_logrotate(config):
+ hookenv.log("Writing logrotate configuration.")
+ templating.render(
+ "logrotate.conf.j2",
+ "/etc/logrotate.d/launchpad-ppa-uploader",
+ config,
+ perms=0o644,
+ )
+
+
+@when(
+ "launchpad.db.configured",
+ "memcache.available",
+)
+@when_not("service.configured")
+def configure():
+ hookenv.log("Configuring ppa uploader")
+ config = get_service_config()
+ config["ppa_archive_root"] = os.path.join(base.base_dir(), "ppa-archive")
+ config["ppa_archive_private_root"] = os.path.join(
+ base.base_dir(), "private-ppa-archive"
+ )
+ config["ppa_keys_root"] = os.path.join(base.base_dir(), "ppa-signing-keys")
+
+ # Create PPA file directories
+ ppa_queue_dir = os.path.join(base.base_dir(), "ppa-queue")
+ host.mkdir(
+ ppa_queue_dir, owner=base.user(), group=base.user(), perms=0o775
+ )
+ config["ppa_queue_dir"] = ppa_queue_dir
+ for folder in ["accepted", "failed", "rejected"]:
+ dir = os.path.join(ppa_queue_dir, folder)
+ host.mkdir(dir, owner=base.user(), group=base.user(), perms=0o755)
+
+ hookenv.log("Setting up memcache for ppa uploader")
+ memcache = endpoint_from_flag("memcache.available")
+ config["memcache_servers"] = ",".join(
+ sorted(
+ f"({host}:{port},1)"
+ for host, port in memcache.memcache_hosts_ports()
+ )
+ )
+ configure_lazr(
+ config,
+ "launchpad-ppa-uploader-lazr.conf.j2",
+ "launchpad-ppa-uploader/launchpad-lazr.conf",
+ )
+ configure_lazr(
+ config,
+ "launchpad-ppa-auth-lazr.conf.j2",
+ "launchpad-ppa-auth/launchpad-lazr.conf",
+ secret=True,
+ )
+ configure_email(config, "launchpad-ppa-uploader")
+ configure_logrotate(config)
+ configure_cron(config, "crontab.j2")
+ set_flag("service.configured")
+
+
+@when("service.configured", "upload-queue-processor.available")
+@when_not("service.txpkgupload-configured")
+def configure_txpkgupload():
+ hookenv.log("Configuring txpkgupload")
+
+ fsroot = os.path.join(base.base_dir(), "ppa-queue", "incoming")
+ host.mkdir(fsroot, owner=base.user(), group=base.user(), perms=0o777)
+ txpkgupload = endpoint_from_flag("upload-queue-processor.available")
+ txpkgupload.set_config(fsroot=fsroot)
+ host.add_user_to_group(base.user(), "txpkgupload")
+ host.add_user_to_group("txpkgupload", base.user())
+ set_flag("service.txpkgupload-configured")
+
+
+@when("service.txpkgupload-configured")
+@when_not_all("service.configured", "upload-queue-processor.available")
+def txpkgupload_deconfigured():
+ hookenv.status_set("blocked", "Txpkgupload not yet configured")
+ clear_flag("service.txpkgupload-configured")
+
+
+@when("service.configured", "service.txpkgupload-configured")
+def check_is_running():
+ hookenv.status_set("active", "Ready")
+
+
+@when("service.configured")
+@when_not_all("launchpad.db.configured", "memcache.available")
+def deconfigure():
+ remove_state("service.configured")
+
+
+@hook("{requires:memcache}-relation-{joined,changed,broken,departed}")
+def memcache_relation_changed(memcache):
+ remove_state("service.configured")
diff --git a/charm/launchpad-ppa-uploader/templates/crontab.j2 b/charm/launchpad-ppa-uploader/templates/crontab.j2
new file mode 100644
index 0000000..79afa82
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/templates/crontab.j2
@@ -0,0 +1,25 @@
+# m h dom mon dow command
+MAILTO={{ cron_mailto }}
+LPCONFIG=launchpad-ppa-uploader
+
+{% if active -%}
+* * * * * nice -n 5 {{ code_dir }}/scripts/process-upload.py -C insecure -q {{ ppa_queue_dir }}/ --log-file=DEBUG:{{ logs_dir }}/process-upload.log
+
+* * * * * nice -n 10 {{ code_dir }}/cronscripts/process-job-source.py IPackageDiffJobSource -q --log-file=DEBUG:{{ logs_dir }}/process-job-source.IPackageDiffJobSource.log
+
+17,47 * * * * nice -n 15 {{ code_dir }}/cronscripts/parse-ppa-apache-access-logs.py -q --log-file=INFO:{{ logs_dir }}/parse-ppa-apache-access-logs.log
+
+# Per bug 62612
+# Files in that directory older than 7 days
+00 00 * * * find {{ ppa_queue_dir }}/accepted -maxdepth 1 -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1
+# Directories older than 7 days (we want to treat the directories as one distinct unit)
+00 00 * * * find {{ ppa_queue_dir }}/accepted -type d -mtime +7 -exec rm -rf {} \; > /dev/null 2>&1
+00 00 * * * find {{ ppa_queue_dir }}/failed -maxdepth 1 -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1
+00 00 * * * find {{ ppa_queue_dir }}/failed -type d -mtime +7 -exec rm -rf {} \; > /dev/null 2>&1
+00 00 * * * find {{ ppa_queue_dir }}/rejected -maxdepth 1 -type f -mtime +7 -exec rm -f {} \; > /dev/null 2>&1
+00 00 * * * find {{ ppa_queue_dir }}/rejected -type d -mtime +7 -exec rm -rf {} \; > /dev/null 2>&1
+
+# OOPS amqp
+*/15 * * * * {{ code_dir }}/bin/datedir2amqp --exchange oopses --host {{ rabbitmq_host }} --username {{ rabbitmq_username }} --password {{ rabbitmq_password }} --vhost {{ rabbitmq_vhost }} --repo {{ oopses_dir }} --key ""
+{% endif %}
+
diff --git a/charm/launchpad-ppa-uploader/templates/launchpad-ppa-auth-lazr.conf.j2 b/charm/launchpad-ppa-uploader/templates/launchpad-ppa-auth-lazr.conf.j2
new file mode 100644
index 0000000..667d033
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/templates/launchpad-ppa-auth-lazr.conf.j2
@@ -0,0 +1,15 @@
+# A config with just enough to run the WSGI authenticator for private PPAs
+# on production. This intentionally doesn't extend ../lpnet-lazr.conf,
+# since Apache can't read the production-secrets overlay.
+
+{% from "macros.j2" import opt -%}
+
+[meta]
+extends: ../../lib/lp/services/config/schema-lazr.conf
+
+[memcache]
+servers: {{ memcache_servers }}
+
+[personalpackagearchive]
+archive_api_endpoint: {{ ppa_api_endpoint }}
+
diff --git a/charm/launchpad-ppa-uploader/templates/launchpad-ppa-uploader-lazr.conf.j2 b/charm/launchpad-ppa-uploader/templates/launchpad-ppa-uploader-lazr.conf.j2
new file mode 100644
index 0000000..5412363
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/templates/launchpad-ppa-uploader-lazr.conf.j2
@@ -0,0 +1,35 @@
+# Public configuration data. The contents of this file may be freely shared
+# with developers if needed for debugging.
+
+# A schema's sections, keys, and values are automatically inherited,
+# except for '.optional' sections. Update this config to override key
+# values. Values are strings, except for numbers that look like ints.
+# The tokens true, false, and none are treated as True, False, and None.
+
+{% from "macros.j2" import opt -%}
+
+
+[meta]
+extends: ../launchpad-db-lazr.conf
+
+[archivepublisher]
+oval_data_rsync_endpoint: {{ oval_data_rsync_endpoint }}
+
+[personalpackagearchive]
+root: {{ ppa_archive_root }}
+private_root: {{ ppa_archive_private_root }}
+signing_keys_root: {{ ppa_keys_root }}
+require_signing_keys: {{ require_signing_keys }}
+
+[ppa_apache_log_parser]
+logs_root: {{ logs_dir }}/apache
+
+[launchpad]
+{{- opt("min_legitimate_account_age", min_legitimate_account_age) }}
+{{- opt("min_legitimate_karma", min_legitimate_karma) }}
+{{- opt("logparser_max_parsed_lines", logparser_max_parsed_lines) }}
+
+[signing]
+signing_endpoint: {{ signing_endpoint }}
+client_public_key: {{ signing_client_public_key }}
+
diff --git a/charm/launchpad-ppa-uploader/templates/logrotate.conf.j2 b/charm/launchpad-ppa-uploader/templates/logrotate.conf.j2
new file mode 100644
index 0000000..494f246
--- /dev/null
+++ b/charm/launchpad-ppa-uploader/templates/logrotate.conf.j2
@@ -0,0 +1,16 @@
+{{ logs_dir }}/launchpad.log
+{
+ rotate 21
+ daily
+ dateext
+ delaycompress
+ compress
+ notifempty
+ missingok
+ create 0644 syslog adm
+ sharedscripts
+ postrotate
+ invoke-rc.d rsyslog rotate >/dev/null
+ endscript
+}
+
diff --git a/cronscripts/publishing/cron.base-ppa.sh b/cronscripts/publishing/cron.base-ppa.sh
index e663fac..a772d39 100644
--- a/cronscripts/publishing/cron.base-ppa.sh
+++ b/cronscripts/publishing/cron.base-ppa.sh
@@ -5,10 +5,17 @@
# Initial setup for PPA cronscripts.
# DO NOT set LPCONFIG here, it should come from the crontab or the shell.
+if [ -d "/srv/launchpad.net" ]
+then
+ BASE_DIR=/srv/launchpad.net
+else
+ BASE_DIR=/srv/launchpad
+fi
+
# Define common variables (also used by cron.daily-ppa).
-PPAROOT=/srv/launchpad.net/ppa-archive
+PPAROOT=$BASE_DIR/ppa-archive
# shellcheck disable=SC2034 # not used here, but used by cron.daily-ppa
-P3AROOT=/srv/launchpad.net/private-ppa-archive
+P3AROOT=$BASE_DIR/private-ppa-archive
LOCKFILE=$PPAROOT/.lock
# Default lockfile options, retry once if it's locked.
if [ "$LOCKFILEOPTIONS" == "" ]; then
Follow ups
