launchpad-reviewers team mailing list archive

[Raphaël Badin <raphael.badin@xxxxxxxxxxxxx>]

Raphaël Badin has proposed merging lp:~rvb/maas/celery-basics-2 into lp:maas with lp:~julian-edwards/maas/celery-basics as a prerequisite.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~rvb/maas/celery-basics-2/+merge/106375

This branch adds celeryd as a service in our arsenal.

= Pre imp =

Spoke about this with Gavin.  This is a first step towards having a proper dev/test Celery environment.

= Notes =

Note that this uses celery and *not* django-celery.  The reason for that is that we want to be as close as possible to our production environment. In production we will be using "pure" celeryd daemons that might be located on a different box than where the MAAS server is (the celery runners will communicate with MAAS exclusively using API calls).

I left django-celery as dependency because we might need it when we will be creating the celery fixture (in particular, we might need the custom testrunner that it provides) but I reckon we should not use it directly.  If we don't need to use the custom testrunner it provides, we might as well completely remove the dependency to django-celery.

I fixed HACKING.txt where we forgot to add python-celery.

The fiddling with PYTHONPATH in services/celeryd/run is required to get the celery workers to find the celery configuration file and the tasks.

= Demo =

If you want proof that this works: here is a tiny diff to create a Celery task and hook it up into the login view: http://paste.ubuntu.com/994162/.
- Apply the diff above
- run the server: make run+webapp
- see the worker process the tasks "tail -f logs/celeryd/current"
-- 
https://code.launchpad.net/~rvb/maas/celery-basics-2/+merge/106375
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~rvb/maas/celery-basics-2 into lp:maas.

=== modified file '.bzrignore'
--- .bzrignore	2012-05-15 13:13:16 +0000
+++ .bzrignore	2012-05-18 13:30:25 +0000
@@ -6,6 +6,7 @@
 ./.noseids
 ./bin
 ./build
+./celerybeat-schedule
 ./db
 ./develop-eggs
 ./dist

=== modified file 'HACKING.txt'
--- HACKING.txt	2012-05-15 13:28:31 +0000
+++ HACKING.txt	2012-05-18 13:30:25 +0000
@@ -44,7 +44,7 @@
         python-formencode python-oauth python-oops python-oops-datedir-repo \
         python-twisted python-oops-wsgi python-oops-twisted \
         python-psycopg2 python-yaml python-convoy python-django-south \
-        python-avahi python-dbus
+        python-avahi python-dbus python-celery
 
 Additionally, you need to install the following python libraries
 for development convenience::

=== modified file 'Makefile'
--- Makefile	2012-05-15 13:32:50 +0000
+++ Makefile	2012-05-18 13:30:25 +0000
@@ -96,6 +96,7 @@
 	$(RM) -r media/demo/* media/development
 	$(RM) $(js_enums)
 	$(RM) *.log
+	$(RM) celerybeat-schedule
 
 distclean: clean stop
 	utilities/maasdb delete-cluster ./db/
@@ -137,7 +138,7 @@
 # Development services.
 #
 
-service_names := pserv reloader txlongpoll web webapp
+service_names := pserv celeryd reloader txlongpoll web webapp
 services := $(patsubst %,services/%/,$(service_names))
 
 run:
@@ -220,6 +221,8 @@
 
 services/webapp/@deps: bin/maas dev-db
 
+services/celeryd/@deps:
+
 #
 # Phony stuff.
 #

=== added directory 'services/celeryd'
=== added file 'services/celeryd/down'
=== added file 'services/celeryd/run'
--- services/celeryd/run	1970-01-01 00:00:00 +0000
+++ services/celeryd/run	2012-05-18 13:30:25 +0000
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+# Exit immediately if a command exits with a non-zero status.
+set -o errexit
+
+# Move to the project root.
+cd "$(dirname "$0")/../.."
+
+# Start logging, if requested. Not using multilog here right now
+# because there are race issues when restarting.
+[ -z "${logdir:-}" ] || exec &>> "${logdir}/current"
+
+# Run celeryd.
+script="$(readlink -f bin/maas)"
+export PYTHONPATH=$PYTHONPATH:`pwd`/src:`pwd`/src/maas
+celeryd --beat -I provisioningserver.tasks -l INFO