launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Tuesday 04 January 2011 19:11:47 Robert Collins wrote:
> On Wed, Jan 5, 2011 at 7:24 AM, Clint Byrum <clint.byrum@xxxxxxxxxxxxx> 
wrote:
> > Trying to emulate the queue's robustness seems a noble, but possibly
> > unnecessary effort if queues are coming any time soon.
> 
> Me too, and also I'd like to know about the actual use cases being
> designed for; the idea of a single fixed sequence marker across the
> system worries me from a performance and scaling aspect: it creates a
> bottleneck in the system that we should demand a large win for having.
> (Compare other bottleneck things like the code import machine table
> which have regularly experienced contention issues).

I'm not sure if you misread or if I didn't explain myself properly, but there 
is no single fixed sequence marker.  The design is for a new one every time 
it's needed, typically one per consumer.

The point of this exercise is *not* to replace MQs, I merely want something I 
can use *now*, is re-usable and forward compatible with MQs.  In other words, 
I'm trying to do the right thing rather than cobble something in.

I'd be happy to consider any suggestions you can make to improve mine.

The immediate use case is that for the process we're developing that 
calculates the differences between two distro series.  Scanning the whole 
series each scan is going to take a long time, so we can figure out where we 
left off last time and only consider new publishing records since the last 
time the process ran.  If I was doing this with MQs, I'd feed all the 
publishing records into a queue and let the consumer process just get on with 
it in real time.

Cheers.