launchpad-dev team mailing list archive

[Ian Booth <ian.booth@xxxxxxxxxxxxx>]

> 
> Right.  I was wondering if Launchpad was going to (now, or later)
> change to just sending "this branch tip changed" and then have the
> recipients decide what to do with that.  For instance, we want to
> update precalculated mp preview diffs when the branch changes.
> Ideally, in a message system, the branch scanner wouldn't need to know
> anything about that, only the job that's subscribed to those events.
> If we want to take that direction perhaps it's better not considering
> it only as a task dispatch system?
> 

This of course is the standard publish/subscribe pattern and one I am
personally fond of. It can and does work well in many situations where
there is a need for true decoupling between producer and consumer. Care
is needed however not to overuse the pattern or use it as a panacea for,
or in lieu of, proper workflow modelling and, where needed, direct
point-point handoff or other coupling mechanisms. I've been on projects
where overuse of this pattern has lead to race conditions, data
consistency issues and other such nasty effects. Don't get me wrong -
the pub/sub pattern definitely has its place - but if it hasn't already
been done we should ensure that the various collaborating services and
associated dataflows are well understood. As a perhaps contrived example
based on the quoted email above, there would be issues if a branch
change event triggered a diff update as well as downstream task which
itself expected the diff update to be completed before it runs. There
would obviously need to be some coordination between the two downstream
consumers of the branch change event publication.

I don't think this has been mentioned yet (?) but another reason why a
message queuing implementation is often used for this type of purpose is
to handle things like: load balancing, quaranteed message delivery (ie
when the recipient is down), quality of service requirements etc. These
features of course contribute towards high availability and allowing
SPOF to be designed out of the system.

Just some more 2c from me.