graphite-dev team mailing list archive

["Brinley, Chris" <Chris.Brinley@xxxxxxxxxx>]

Internally working towards replacing the links between the persistence layer and the routing layer with AMQP. The idea here would be you have certain sets of servers that fill a logical storage role such as "dev metrics" or "bussiness metrics" should be tied to a message queue of the same "topic". Adding and removing capacity in each logical domain then becomes a matter of connecting to the queue. Also be between storage layer and presentation layer I am working on an implementatation along the same lines:

presenation layer requests data about metrics X,Y,Z from the data proivder queue. There are some concerns I have with managing response times here but that's the 20K foot view.

this also starts to make graphite more serivce oriented in that arbitrary app can consume data about metrics.

Chris your familiar with the infrastructure, but i agree sending directly to the storage layer via AMQP would be optimal. thats probably not going to happen in our case so i think adding it in between the routing and storage layers now and side steping carbon entirely in the future may a more practical path for us.

Chris Brinley
________________________________
From: graphite-dev-bounces+chris.brinley=orbitz.com@xxxxxxxxxxxxxxxxxxx [graphite-dev-bounces+chris.brinley=orbitz.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Chris Davis [chrismd@xxxxxxxxx]
Sent: Friday, February 12, 2010 4:00 PM
To: graphite-dev
Subject: [Graphite-dev] request for comments - using amqp and replacing carbon-relay

Hey everyone, recently we have gotten some really great community contributions, one of which adds support for the AMQP messaging protocol to carbon. I have started migrating some of my systems to using this and I think it is a great fit for graphite. If you aren't familiar with it already I highly recommend reading http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for a brief introduction.

One area I think AMQP can be especially useful is with a clustered graphite installation. Most of you are probably not familiar with Graphite's clustering capabilities because I have not documented it at all (sorry, hope to change that soon). But essentially, you just setup N different independent graphite servers and then configure the webapps to know about one another. They then share data in a manner transparent to the user, making each server look like one big graphite installation instead of several independent ones. The tricky part is partitioning your metrics across the servers. Thus far I've solved this problem with a daemon called carbon-relay.py, which basically acts as an application-level load balancer for your metrics. You are supposed to send all of your data to carbon-relay, who then looks up some rules you've given it to decide which carbon-cache(s) to relay each metric on to.

With AMQP, there seems to be a much simpler way to solve this problem. Topic exchanges use a dot-delimited naming structure that supports pattern matching very similar to graphite's. Basically, you could just publish messages containing a value and a timestamp and use the metric name as the routing key. Then each carbon-cache can consume from the exchange with one or more binding patterns. For the simplest case of having only one server the binding pattern would simply be "#" (which is the same as using a fanout exchange). For more complex cases you could control what data goes to what server by means of configuring each carbon-cache's binding patterns. This would effectively replace carbon-relay, and I believe, solve the problem in a more robust way.

This is a bit different than they way it currently works in trunk so I wanted to run it by everyone and see what your thoughts are, especially if you are already using AMQP or carbon-relay. I am currently in the process of testing this configuration and if it works well I will try it in my production system. If that goes well, I would like to include the new behavior in this month's release. So please send me any comments, questions, concerns, etc.

If you don't plan on using AMQP, that's fine too, the old interface is not going away.

-Chris