← Back to team overview

openstack team mailing list archive

Re: Queue Service


Hi Pete,

Two initial use cases are a comparable service to what other public
cloud providers already have (similar HTTP/REST API, but eventually
with more features) and a flexible, high-performance queue/event system
for backend log and event processing for other OpenStack services (see
the message I just sent on another thread regarding the "firehose").

As for general uses of queues, there are too many to list. One simple
example I usually start with is large file processing, whether it
be images, videos, or some other manipulation. Say you had a web
application that processed images, and rather than consuming the CPU
of the front-end machine and blocking the request, you can instead
push the URL/path of the image into a "processing queue". This allows
the request to return immediately while your queue workers actually
perform the processing (possibly on dedicated machines more appropriate
for the task).

Async queues provide specialization of tasks, help lower response time,
and make horizontal scale-out much easier, among other things.


On Mon, Feb 14, 2011 at 11:20:41AM -0700, Pete Zaitcev wrote:
> On Mon, 14 Feb 2011 09:51:42 -0800
> Eric Day <eday@xxxxxxxxxxxx> wrote:
> > [] A queue service can
> > not only provide a useful public cloud service, but can also provide
> > one of the building blocks for other services.
> > I'm looking forward to your feedback. Thanks!
> Do you have a specific application and/or testbed in mind? Can you give
> us any specifics?
> The question of pub/sub and queues seem to be floating around but I never
> can find a good use case to study. Proponents of queueing architectures
> keep saying how wonderful they are, but I do not see what's so good
> about them.
> -- Pete