launchpad-reviewers team mailing list archive

[Tom Wardill <tom.wardill@xxxxxxxxxxxxx>]

Tom Wardill has proposed merging ~twom/turnip:add-prefetch-multiplier-to-celery-workers into turnip:master.

Commit message:
Add prefetch multiplier to celery workers

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~twom/turnip/+git/turnip/+merge/402396

Some of our tasks are short, some can run for quite some time. We should limit the number of tasks that are prefetched by a worker to enable the queue to be distributed more evenly.

This limits the default to 1 (for a total of 2) for each worker.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~twom/turnip:add-prefetch-multiplier-to-celery-workers into turnip:master.

diff --git a/charm/turnip-celery/config.yaml b/charm/turnip-celery/config.yaml
index 6f8ffb1..3b2cad9 100644
--- a/charm/turnip-celery/config.yaml
+++ b/charm/turnip-celery/config.yaml
@@ -11,3 +11,7 @@ options:
     type: int
     default: 15
     description: Timeout for virtinfo requests, in seconds.
+  prefetch_multiplier:
+    type: int
+    default: 1
+    description: Celery prefetch multiplier for each worker.
diff --git a/charm/turnip-celery/templates/turnip-celery.service.j2 b/charm/turnip-celery/templates/turnip-celery.service.j2
index a075e6b..a631b86 100644
--- a/charm/turnip-celery/templates/turnip-celery.service.j2
+++ b/charm/turnip-celery/templates/turnip-celery.service.j2
@@ -16,7 +16,7 @@ Environment=TURNIP_LOG_DIR={{ logs_dir }}
 Environment=CELERY_BROKER={{ celery_broker }}
 Environment=VIRTINFO_ENDPOINT={{ virtinfo_endpoint }}
 Environment=VIRTINFO_TIMEOUT={{ virtinfo_timeout }}
-ExecStart={{ venv_dir }}/bin/celery -A turnip.tasks worker --logfile={{ logs_dir }}/turnip-celery.log --loglevel=DEBUG --pool=gevent
+ExecStart={{ venv_dir }}/bin/celery -A turnip.tasks worker --logfile={{ logs_dir }}/turnip-celery.log --loglevel=DEBUG --pool=gevent --prefetch-multiplier={{ prefetch_multiplier }}
 ExecReload=/bin/kill -s HUP $MAINPID
 ExecStop=/bin/kill -s TERM $MAINPID
 LimitNOFILE=1048576