launchpad-dev team mailing list archive

[Ian Booth <ian.booth@xxxxxxxxxxxxx>]

> 
> Firstly, lets define some forms of queue use we'd like to have - I'm
> going to define three:
> pubsub: publisher knows nothing about the subscribers, just fires off
> events. Perhaps we might want durable subscriptions (where a
> disconnected subscriber can catch up later). All subscribers get all
> messages. Subscribers know what they are subscribed to (but there
> could be many publishers connected to the same subscribers).
>

A quick side note: pubsub also should be able to support content based
filtering (and routing) of messages. This can be done to some extent by
establishing publication channels but true content based
filtering/routing is also more often than not required.

Queue based solutions also can introduce a failure mode which is more
related to scalability and occurs when the subscribers lack the ability
to consume messages at a rate fast enough to keep up with the
publishers. In these cases, if there is no discard policy or ability to
throttle, the queue sizes ramp up and the system can get into a downward
spiral to failure. Of course horizontal scalability helps mitigate this
but is not a total panacea. In our system, given the volume of messages
likely to be processed, this may not ever be an issue but I thought I'd
mention it for completeness.