launchpad-dev team mailing list archive

[Jeroen Vermeulen <jtv@xxxxxxxxxxxxx>]

On 2011-05-14 14:19, Robert Collins wrote:

> I believe 2pc is possible with amqp 0.10, but rabbitmq does 0.8 - and
> see the debate around 0mq for

Something went missing here!


> Beyond that, I think we need consider ordering things (where possible)
> to be idempotent and failure tolerate (given the potential overheads
> of 2pc).

Emphatically agreed.  There's the performance cost and increased lock 
contention but also, we'd need to manage the state of persistent 
halfway-committed transactions across database restarts.


> > Another problem happens when things work too well: you create a
> > database-backed object.  You fire off a job related to that object.  You
> > commit.  But the consumer of that job picks it up before your commit has
> > propagated and boom!  The job dies in flames because it accesses objects
> > that aren't decisively in the database yet.
>
> For this sort of thing fire-after-commit should be easy and sufficient.

Yes, it eliminates this particular hazard.


> > I imagine both problems go away if every message carries a database
> > transaction id, and the job runner keeps an eye on the database transaction
> > log: the runner shouldn't consume a job until the producing transaction has
> > committed, and it should drop jobs whose producers have aborted.  Is
> > something along those lines built in?
>
> No, and that sounds terrifying to me - if we tie things up that
> tightly, we may as well just have a queue in the db and poll it every
> 50ms.

That would be much more costly in both computational and development 
terms.  But what makes the check for transaction completion sound so 
scary to you?


Jeroen