launchpad-dev team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

On Sat, May 14, 2011 at 10:06 AM, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:

> Without having looked into it properly, I think we'll need some kind of
> wrapper to support this distinction.  Traditional transactional messaging
> uses two-phase commit; other products use database queues similar to our
> Job.  Both are probably overweight to the point where our baby would go out
> with the bathwater.  We could fake it by queuing up jobs in memory and
> sending them after commit, but that leaves open a window for message loss.

I think we do want to queue messages and send them after the transaction.

Thankfully, this is trivial now good documentation exists. A utility
that registers itself as a Synchronizer seems fine, and can provide a
high level API that hides the rabbitmq and python library specifics.

See http://readthedocs.org/docs/zodb/en/latest/transactions.html for
the wonderful docs.

I imagine we will want both ITransactionalMessenger and
INontransactionalMessenger available with the same API, as sometimes
we really do want to send a message mid-transaction. Two things I can
think of immediately are log messages (no more rsync delays), and
script activity keep alives (I'm still alive!).

> Another problem happens when things work too well: you create a
> database-backed object.  You fire off a job related to that object.  You
> commit.  But the consumer of that job picks it up before your commit has
> propagated and boom!  The job dies in flames because it accesses objects
> that aren't decisively in the database yet.

We also have to deal with transactions that are retried - we don't
want to repeat messages.

-- 
Stuart Bishop <stuart@xxxxxxxxxxxxxxxx>
http://www.stuartbishop.net/