launchpad-reviewers team mailing list archive

Thread
Date

[Merge] ~ilkeremrekoc/launchpad:upgrade-celery into launchpad:master

To: mp+492996@xxxxxxxxxxxxxxxxxx
From: İlker Emre Koç <mp+492996@xxxxxxxxxxxxxxxxxx>
Date: Mon, 22 Sep 2025 04:23:37 -0000
Reply-to: mp+492996@xxxxxxxxxxxxxxxxxx
Sender: noreply@xxxxxxxxxxxxx

İlker Emre Koç has proposed merging ~ilkeremrekoc/launchpad:upgrade-celery into launchpad:master.

Commit message:
Upgrade celery

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~ilkeremrekoc/launchpad/+git/launchpad/+merge/492996

Celery 5 breaks the previous CLI option distribution, we updated them to
fit with the new standard.

Celery 5 also gets rid of celery.task module, which was used in our
tests, we replace that.

We also add both the dependencies and transitive dependencies of celery
5 and upgrade lazr.jobrunner since the celery version range of the
previous version was blocking us.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~ilkeremrekoc/launchpad:upgrade-celery into launchpad:master.

diff --git a/charm/launchpad-native-publisher/templates/celerybeat_native_publisher.service.j2 b/charm/launchpad-native-publisher/templates/celerybeat_native_publisher.service.j2
index 363c33e..f590a8d 100644
--- a/charm/launchpad-native-publisher/templates/celerybeat_native_publisher.service.j2
+++ b/charm/launchpad-native-publisher/templates/celerybeat_native_publisher.service.j2
@@ -6,8 +6,10 @@ After=syslog.target network.target remote-fs.target nss-lookup.target
 Environment=LPCONFIG=launchpad-native-publisher
 User=launchpad
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery beat \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    beat \
     --pidfile={{ var_dir }}/celerybeat_native_publisher.pid \
     --logfile={{ logs_dir }}/celerybeat_native_publisher.log \
     --schedule={{ var_dir }}/celerybeat-schedule.db
diff --git a/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job.service.j2 b/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job.service.j2
index 472cc7a..1e28bcb 100644
--- a/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job.service.j2
+++ b/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-native-publisher
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue native_publisher_job \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue native_publisher_job \
     --hostname=native_publisher_job \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_native_publisher_job.log
diff --git a/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job_slow.service.j2 b/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job_slow.service.j2
index ceb86a4..15430be 100644
--- a/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job_slow.service.j2
+++ b/charm/launchpad-native-publisher/templates/celeryd_native_publisher_job_slow.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-native-publisher
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue native_publisher_job_slow \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue native_publisher_job_slow \
     --hostname=native_publisher_job_slow \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_native_publisher_job_slow.log
diff --git a/charm/launchpad-scripts-bzrsyncd/templates/celerybeat_bzrsyncd.service.j2 b/charm/launchpad-scripts-bzrsyncd/templates/celerybeat_bzrsyncd.service.j2
index 5ef40f8..52cfdf3 100644
--- a/charm/launchpad-scripts-bzrsyncd/templates/celerybeat_bzrsyncd.service.j2
+++ b/charm/launchpad-scripts-bzrsyncd/templates/celerybeat_bzrsyncd.service.j2
@@ -6,8 +6,10 @@ After=syslog.target network.target remote-fs.target nss-lookup.target
 Environment=LPCONFIG=launchpad-scripts-bzrsyncd
 User=launchpad
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery beat \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    beat \
     --pidfile={{ var_dir }}/celerybeat_bzrsyncd.pid \
     --logfile={{ logs_dir }}/celerybeat_bzrsyncd.log \
     --schedule={{ var_dir }}/celerybeat-schedule.db
diff --git a/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job.service.j2 b/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job.service.j2
index 47d6eda..86a7c07 100644
--- a/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job.service.j2
+++ b/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-scripts-bzrsyncd
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue bzrsyncd_job \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue bzrsyncd_job \
     --hostname=launchpad_bzrsyncd_job \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_launchpad_bzrsyncd_job.log
diff --git a/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job_slow.service.j2 b/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job_slow.service.j2
index 34923be..363206c 100644
--- a/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job_slow.service.j2
+++ b/charm/launchpad-scripts-bzrsyncd/templates/celeryd_bzrsyncd_job_slow.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-scripts-bzrsyncd
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue bzrsyncd_job_slow \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue bzrsyncd_job_slow \
     --hostname=launchpad_bzrsyncd_job_slow \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_launchpad_bzrsyncd_job_slow.log
diff --git a/charm/launchpad-scripts/templates/celerybeat_launchpad.service.j2 b/charm/launchpad-scripts/templates/celerybeat_launchpad.service.j2
index e109591..4af0e8e 100644
--- a/charm/launchpad-scripts/templates/celerybeat_launchpad.service.j2
+++ b/charm/launchpad-scripts/templates/celerybeat_launchpad.service.j2
@@ -6,8 +6,10 @@ After=syslog.target network.target remote-fs.target nss-lookup.target
 Environment=LPCONFIG=launchpad-scripts
 User=launchpad
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery beat \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    beat \
     --pidfile={{ var_dir }}/celerybeat_launchpad.pid \
     --logfile={{ logs_dir }}/celerybeat_launchpad.log \
     --schedule={{ var_dir }}/celerybeat-schedule.db
diff --git a/charm/launchpad-scripts/templates/celeryd_launchpad_job.service.j2 b/charm/launchpad-scripts/templates/celeryd_launchpad_job.service.j2
index 67f9f71..29f8b6f 100644
--- a/charm/launchpad-scripts/templates/celeryd_launchpad_job.service.j2
+++ b/charm/launchpad-scripts/templates/celeryd_launchpad_job.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-scripts
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue launchpad_job \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue launchpad_job \
     --hostname=launchpad_job \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_launchpad_job.log
diff --git a/charm/launchpad-scripts/templates/celeryd_launchpad_job_slow.service.j2 b/charm/launchpad-scripts/templates/celeryd_launchpad_job_slow.service.j2
index 9492e60..260ddf1 100644
--- a/charm/launchpad-scripts/templates/celeryd_launchpad_job_slow.service.j2
+++ b/charm/launchpad-scripts/templates/celeryd_launchpad_job_slow.service.j2
@@ -8,9 +8,11 @@ User=launchpad
 Group=launchpad
 Environment=LPCONFIG=launchpad-scripts
 WorkingDirectory={{ code_dir }}
-ExecStart={{ code_dir }}/bin/celery worker \
-    --queue launchpad_job_slow \
+ExecStart={{ code_dir }}/bin/celery \
+    -A lp.services.job.celeryjob \
     --config=lp.services.job.celeryconfig \
+    worker \
+    --queue launchpad_job_slow \
     --hostname=launchpad_job_slow \
     --loglevel=DEBUG \
     --logfile={{ logs_dir }}/celeryd_launchpad_job_slow.log
diff --git a/lib/lp/services/job/celeryconfig.py b/lib/lp/services/job/celeryconfig.py
index 6d7946e..00dac88 100644
--- a/lib/lp/services/job/celeryconfig.py
+++ b/lib/lp/services/job/celeryconfig.py
@@ -52,7 +52,7 @@ def configure(argv):
     # A queue must be specified as a command line parameter for each
     # "celery worker" instance, but this is not required for a Launchpad app
     # server.
-    if "celery" in argv[0] and argv[1] == "worker":
+    if "celery" in argv[0] and "worker" in argv:
         if queues is None or queues == "":
             raise ConfigurationError("A queue must be specified.")
         queues = queues.split(",")
diff --git a/lib/lp/services/job/tests/__init__.py b/lib/lp/services/job/tests/__init__.py
index 179afe3..9a62a25 100644
--- a/lib/lp/services/job/tests/__init__.py
+++ b/lib/lp/services/job/tests/__init__.py
@@ -32,11 +32,13 @@ def celery_worker(queue, cwd=None):
     with celery_app.broker_connection() as connection:
         broker_uri = connection.as_uri(include_password=True)
     cmd_args = (
-        "worker",
+        "-A",
+        "lp.services.job.celeryjob",
         "--config",
         "lp.services.job.celeryconfig",
         "--broker",
         broker_uri,
+        "worker",
         "--concurrency",
         "1",
         "--loglevel",
diff --git a/lib/lp/services/job/tests/celery_helpers.py b/lib/lp/services/job/tests/celery_helpers.py
index d79ae57..48a84cc 100644
--- a/lib/lp/services/job/tests/celery_helpers.py
+++ b/lib/lp/services/job/tests/celery_helpers.py
@@ -3,23 +3,17 @@
 
 __all__ = ["noop", "pop_notifications"]
 
-from celery.task import task
+from lp.services.job.celeryjob import celery_app
 
-# Force the correct celeryconfig to be used.
-import lp.services.job.celeryjob
 
-# Quiet lint unused import warning.
-lp.services.job.celeryjob
-
-
-@task
+@celery_app.task
 def pop_notifications():
     from lp.testing.mail_helpers import pop_notifications
 
     return [message.as_string() for message in pop_notifications()]
 
 
-@task
+@celery_app.task
 def noop():
     """Task that does nothing.
 
diff --git a/requirements/launchpad.txt b/requirements/launchpad.txt
index dfb4119..b9337ba 100644
--- a/requirements/launchpad.txt
+++ b/requirements/launchpad.txt
@@ -6,7 +6,7 @@
 # versions; they will be included automatically.
 
 ampoule==24.10.0
-amqp==2.6.1
+amqp==5.3.1
 annotated-types==0.7.0
 anyjson==0.3.3
 appdirs==1.4.3
@@ -15,11 +15,12 @@ attrs==19.3.0
 Automat==20.2.0
 backcall==0.2.0
 backports.functools-lru-cache==1.5
+backports.zoneinfo==0.2.1
 # ztk-versions.cfg uses 3.2.0 on Python 3, but that drops support for Python
 # 3.5.  Pin to 3.1.7 until we no longer care about xenial.
 bcrypt==3.1.7
 beautifulsoup4==4.12.3
-billiard==3.6.4.0
+billiard==4.2.2
 bleach==6.1.0
 bleach-allowlist==1.0.3
 breezy==3.2.0
@@ -27,8 +28,12 @@ brz-builder==0.7.4
 bson==0.5.9
 boto3==1.35.71
 botocore==1.35.71
-celery==4.4.7
+celery==5.5.3
 Chameleon==3.6.2
+click==8.1.8
+click-didyoumean==0.3.1
+click-plugins==1.1.1.2
+click-repl==0.3.0
 configobj==5.0.6
 contextvars==2.4
 constantly==15.1.0
@@ -74,14 +79,14 @@ jmespath==0.10.0
 jsautobuild==0.2
 keyring==0.6.2
 keystoneauth1==4.1.0
-kombu==4.6.11
+kombu==5.5.4
 launchpad-buildd==206
 launchpadlib==2.1.0
 lazr.batchnavigator==1.3.1
 lazr.config==2.2.3
 lazr.delegates==2.0.4
 lazr.enum==1.2.1
-lazr.jobrunner==0.17
+lazr.jobrunner==1.1
 lazr.lifecycle==1.2.1
 lazr.restful==2.0.2
 lazr.restfulclient==0.14.5
@@ -126,6 +131,7 @@ pexpect==4.8.0
 pgbouncer==0.0.9
 pickleshare==0.7.5
 pkginfo==1.11.2
+poetry-core==1.9.1
 prettytable==0.7.2
 psutil==7.0.0
 psycopg2==2.8.6
@@ -193,9 +199,10 @@ txfixtures==0.4.3
 txfixtures==0.5.2; python_version >= "3.10"
 txpkgupload==0.5
 typing_extensions==4.12.2; python_version >= "3.9"
+tzdata==2025.2
 urllib3==1.26.20
 van.testing==3.0.0
-vine==1.3.0
+vine===5.1.0
 virtualenv-tools3==3.1.1
 wadllib==1.3.6
 waitress==2.1.2

Follow ups

[Merge] ~ilkeremrekoc/launchpad:upgrade-celery into launchpad:master
From: İlker Emre Koç, 2025-09-22