launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Mon, May 16, 2011 at 2:59 PM, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:
> On 2011-05-14 14:19, Robert Collins wrote:
>
>> I believe 2pc is possible with amqp 0.10, but rabbitmq does 0.8 - and
>> see the debate around 0mq for
>
> Something went missing here!

Huh. So anyhow - 0mq is an attempt to build an even faster message
substrate than rabbit (which is no slouch) by ditching everything, and
then building redundancy etc on top of the super fast substrate.

>>> I imagine both problems go away if every message carries a database
>>> transaction id, and the job runner keeps an eye on the database
>>> transaction
>>> log: the runner shouldn't consume a job until the producing transaction
>>> has
>>> committed, and it should drop jobs whose producers have aborted.  Is
>>> something along those lines built in?
>>
>> No, and that sounds terrifying to me - if we tie things up that
>> tightly, we may as well just have a queue in the db and poll it every
>> 50ms.
>
> That would be much more costly in both computational and development terms.
>  But what makes the check for transaction completion sound so scary to you?

The check - not scary. The coupling involved - very scary.

There are three angles - firstly I can see /some/ messages and queue
consumers needing such a sync-and-check, but not all. So
s/every/appropriate/ and that would be less scary.

Secondly I'd like to be isolating our systems from the db more, rather
than building in deeper connections - because our existing deep
connections are regularly giving us grief (with issues related to
stale transactions,  resource management, failover logic, performance
(scale up vs scale out)).

Lastly we'd be taking messages from a (optionally) persistent queue
and having to repersist them until they could be handled (and also
deal with the question of whether to do them out-of-order).

I think a better model is to only send a message when we're happy for
it to be dealt with. E.g. using pg_amqp to send during the commit.

-Rob