openstack team mailing list archive

[George Reese <george.reese@xxxxxxxxxxxxx>]

This isn't a message queue, it's a push system.

In other words, consumers don't pull info from a queue, the info is pushed out to any number of subscribers as the message is generated.

Amazon SNS vs. SQS, except this isn't a cloud service but a mechanism for notifying interested party of cloud changes.

-George

On May 10, 2011, at 1:49 PM, Eric Day wrote:

> We may also want to put in some kind version or self-documenting URL
> so it's easier to accommodate message format changes later on.
> 
> As for the issue of things getting backed up in the queues for other
> non-PuSH mechanisms (and fanout), burrow has fanout functionality
> that depends on messages to expire (every message is inserted with
> a TTL). This would allow multiple readers to see the same message
> and for it to disappear after say an hour. This allows deployments,
> third party tools, and clients to write workers to act on events from
> the raw queue.
> 
> With burrow, it will also be possible for clients to pull raw messages
> directly from the queue via a REST API in a secure fashion using
> the same account credentials as other OpenStack service (whatever
> keystone is configured for). So while an email notification will want
> to strip any sensitive information, a direct queue client could see
> more details.
> 
> -Eric
> 
> On Mon, May 09, 2011 at 10:20:04PM +0000, Matt Dietz wrote:
>>   Hey guys,
>>   Monsyne Dragon and myself are proposing an implementation for
>>   notifications going forward. Currently my branch exists
>>   under https://code.launchpad.net/~cerberus/nova/nova_notifications. you'll
>>   see that's it been proposed for merge, but we're currently refactoring it
>>   around changes proposed at the summit during the notifications discussion,
>>   which you can see at http://etherpad.openstack.org/notifications
>>   At the heart of the above branch is the idea that, because nova is about
>>   compute, we get notifications away from Nova as quickly as possible. As
>>   such, we've implemented a simple modular driver system which merely pushes
>>   messages out. The two sample "drivers" are for pushing messages into
>>   Rabbit, or doing nothing at all. There's been talk about adding Burrow as
>>   a third possible driver, which I don't think would be an issue.
>>   One of the proposals is to have priority levels for each notification.
>>   What we're proposing is emulating the standard Python logging module and
>>   providing levels like "WARN' and "CRITICAL" in the notification.
>>   Additionally, the message format we're proposing will be a JSON dictionary
>>   containing the following attributes:
>>   publisher_id - the source worker_type.host of the message.
>>   timestamp - the GMT timestamp the notification was sent at
>>   event_type - the literal type of event (ex. Instance Creation)
>>   priority - patterned after the enumeration of Python logging levels in
>>                  the set (DEBUG, WARN, INFO, ERROR, CRITICAL)
>>   payload - A python dictionary of attributes
>>   Message example:
>>       { 'publisher_id': 'compute.host1',
>>         'timestamp': '2011-05-09 22:00:14.621831',
>>         'priority': 'WARN',
>>         'event_type': 'compute.create_instance',
>>         'payload': {'instance_id': 12, ... }}
>>   There was a lot of concern voiced over messages backing up in any of the
>>   queueing implementations, as well as the intended priority of one message
>>   over another. There are couple of immediately obvious solutions to this.
>>   We think the simplest solution is to implement N queues, where N is equal
>>   the number of priorities. Afterwards, consuming those queues is
>>   implementation specific and dependent on the solution that works best for
>>   the user.
>>   The current plan for the Rackspace specific implementation is to use
>>   PubSubHubBub, with a dedicated worker consuming the notification queues
>>   and providing the glue necessary to work with a standard Hub
>>   implementation. I have a very immature worker implementation
>>   at https://github.com/Cerberus98/yagi if you're interested in checking
>>   that out. 
>>   We'll be going forward with this plan immediately, but we'd love feedback
>>   if you have it. Questions, comments, concerns are very much welcomed!
>>   Matt Dietz
> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

--
George Reese - Chief Technology Officer, enStratus
e: george.reese@xxxxxxxxxxxxx    t: @GeorgeReese    p: +1.207.956.0217    f: +1.612.338.5041
enStratus: Governance for Public, Private, and Hybrid Clouds - @enStratus - http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese

Attachment: smime.p7s
Description: S/MIME cryptographic signature