openstack team mailing list archive

[Yun Mao <yunmao@xxxxxxxxx>]

Sorry to bring back a rather quiet thread from 3 days ago.

How fast do we need to queueing component to be? My observation from
Amazon EC2 us-east-1 is about 2 VMs provisioned per second on average.
Let's say there are 100 messages exchanged for the workload per second
per VM (which I believe is over estimated), and the peak time workload
is 100x higher. Then we need a queue that can do 20,000 messages per
second at the peak rate. Either Rabbit or 0MQ should handle this very
easily. So I'm assuming performance is not a concern.

Now if we go brokerless completely for all messages, that's an obvious
gain as we get rid of one type source of failure. Can that be
achieved? My impression after quickly skimming through the 0MQ
document is that those direct connection can be brokerless but things
more like broadcast can't be. I might very much be wrong at this, and
I would appreciate a lot if someone could help to explain.

Thanks,

Yun

On Wed, Jan 25, 2012 at 11:56 AM, Alexis Richardson <alexis@xxxxxxxxxxxx> wrote:
> Eric
>
> Understood ;-)
>
> I am all in favour of community experimentation.
>
> 1:1 messaging is a core use case for RabbitMQ.  Unlike regular
> queueing systems which use queues for shared topics, Rabbit is
> designed to support very large numbers of short lived queues as well
> as long lived queues.  These can be private or shared.  In other
> words: queues are buffers.  ZeroMQ goes one step further and
> co-locates the consumer with the buffer, and the routing logic with
> the producer.  The cases for which this is useful are discussed on the
> web site.
>
> alexis
>
>
>
> On Wed, Jan 25, 2012 at 4:49 PM, Eric Windisch <eric@xxxxxxxxxxxxxxxx> wrote:
>> Alexis,
>>
>> It is also obvious that the link I provided is a particularly biased source,
>> so it should be taken with a grain of salt. I only mentioned Qpid because
>> Nova just received a driver for it, I didn't know the differences in such
>> detail.
>>
>> One of the problems Nova has is that it registers N queues for N hosts, with
>> one host pulling from each queue (1:1). This is why ZeroMQ is a good fit,
>> whereby messages can be sent directly to those hosts. There are also a small
>> (but active) number of N to N queues which remain centralized and for which
>> running Rabbit or Qpid is a good fit.
>>
>> It would be interesting exercise to allow the ZeroMQ driver to defer back to
>> the Kombu or Qpid driver for those messages which must remain centralized.
>>
>> --
>> Eric Windisch
>>
>> On Wednesday, January 25, 2012 at 1:18 AM, Alexis Richardson wrote:
>>
>> On Wed, Jan 25, 2012 at 4:46 AM, Eric Windisch <eric@xxxxxxxxxxxxxxxx>
>> wrote:
>>
>> Sorry, I had originally sent only to Yun Mao. Sending to list.
>>
>> ---
>>
>> Rather than attempt to answer this, I defer to the ZeroMQ guide. It should
>> be noted that the designers of AMPQ, iMatix, designed and build ZeroMQ.
>> (RabbitMQ and QUID implement AMQP)
>>
>>
>> Hold on a second there...
>>
>> There has been a LOT of muddle and fud ("fuddle"?) around AMQP. Let
>> me try to clear that up.
>>
>> Qpid's default option is AMQP 0-10. While feature-rich, 0-10 is not
>> widely used and was described by the AMQP chairman as too long and
>> complicated not long after it was published. See also commentary on
>> the web, on the subject of its length. Rabbit does not and will not
>> support this version, and other folks have not implemented it either.
>>
>> WHEREAS --
>>
>> RabbitMQ implements AMQP 0-91, a 40 page spec. It's the one most people use.
>>
>> 0-9-1 is the version of AMQP that is used across a very large number
>> of use cases, that is quite easy to implement. It was created by all
>> the implementers of AMQP that existed at time of writing including
>> Rabbit, Redhat, JPMorgan, and of course iMatix. Pieter @iMatix was
>> the spec editor, and did a fantastic job. 0-9-1 provides
>> interoperable messaging as witnessed by the large number (100s) of
>> clients and add-ons that have been created by the community. There
>> have also been several servers implemented, that all just work with
>> those clients. For example Kaazing, StormMQ, and OpenAMQ. I believe
>> that Qpid also supports it, which might be important for this
>> community (Redhat guys please note).
>>
>> This is what Pieter said: "Read AMQP/0.9.1, it is a beautiful, clean,
>> minimalist work essentially created by cooperation in the working
>> group to improve AMQP/0.8. I edited AMQP/0.9.1, based on a hundred or
>> more fixes made by the best individual brains in that group. Alexis is
>> right - this is not disappearing." (Ref - comments here:
>> http://it.toolbox.com/blogs/open-source-smb/whats-the-future-of-amqp-44450)
>>
>> I agree with Pieter. We also like the way that 0-9-1 can play nicely
>> with 0mq and other protocols. Note that Rabbit supports these via
>> plugins.
>>
>> alexis
>>
>>
>>
>>
>>
>>
>> http://zguide.zeromq.org/page:all#Why-We-Needed-MQ
>>
>>
>> --
>> Eric Windisch
>>
>> On Tuesday, January 24, 2012 at 5:20 PM, Yun Mao wrote:
>>
>> Hi I'm curious and unfamiliar with the subject. What's the benefit of
>> 0MQ vs Kombu? Thanks,
>>
>> Yun
>>
>> On Tue, Jan 24, 2012 at 7:08 PM, Eric Windisch <eric@xxxxxxxxxxxxxxxx>
>> wrote:
>>
>> Per today's meeting, I am proposing the ZeroMQ RPC driver for a
>> feature-freeze exception.
>>
>> I am making good progress on this blueprint, it adds a new optional module
>> and service without modifying any existing code or modules. I have been
>> pushing to complete this work by E3, so I am close to completion, but cannot
>> finish by tonight's deadline.
>>
>> The ZeroMQ driver will provide an alternative to Kombu (RabbitMQ) and QPID
>> for messaging within Nova. Currently, the code passes unit tests but fails
>> on smoketests. I expect to have the code viable for a merge proposal in less
>> than a week, tomorrow if I'm smart, lucky, and the store doesn't sell out of
>> RedBull. A two week grace would give me a nice buffer.
>>
>> Thanks,
>> Eric Windisch
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>>