launchpad-dev team mailing list archive

[Jeroen Vermeulen <jtv@xxxxxxxxxxxxx>]

On 2011-05-16 10:14, Robert Collins wrote:

> The check - not scary. The coupling involved - very scary.
>
> There are three angles - firstly I can see /some/ messages and queue
> consumers needing such a sync-and-check, but not all. So
> s/every/appropriate/ and that would be less scary.

I think you're making mountains out of rabbit droppings here.  Of course 
you don't use it if you don't want it.  The "every message" applies to 
the kinds of messages that we're not willing to risk losing, 
duplicating, or delivering improperly.


> Lastly we'd be taking messages from a (optionally) persistent queue
> and having to repersist them until they could be handled (and also
> deal with the question of whether to do them out-of-order).

I don't see how any of that follows from the notion of checking for 
transaction ids.  As far as the re-persisting goes, if you're thinking 
of putting a message back if it's not ripe yet: that's not needed. 
Destructive gets ("take a message off the queue and hand it to me") are 
optional in AMQP and indeed in Rabbit.  You can receive a message, 
process it, and then acknowledge completion so that Rabbit can take it 
off the queue.  If you die while processing the message, it gets retried.

So a message handler on the consumer side might do something like:

class ProducerStatus:
    PENDING = 1
    COMMITTED = 2
    ABORTED = 3


def handle(*message):
    producer_status = get_producer_status(*message)
    if producer_status == PENDING:
        return
    try:
        # If the message was never committed, or has already
        # been processed, consume it without executing it.
        if producer_status == COMMITTED and is_fresh(*message):
            do_job(*message)
    except:
        abort()
    finally:
        acknowledge_message(*message)
        commit()

Here, get_producer_status looks up the producing transaction's status 
and is_fresh checks against duplicate processing.


> I think a better model is to only send a message when we're happy for
> it to be dealt with. E.g. using pg_amqp to send during the commit.

AIUI that prevents improper delivery, but will still merrily drop 
messages.  It also seems to require custom hand-holding code just to 
report the loss in a meaningful way, after which there's no clear answer 
on what to do about it.

It's hard to say from the code though, which is straight uncommented, 
untested C.  Shudder.


Jeroen