graphite-dev team mailing list archive

[Chris Davis <chrismd@xxxxxxxxx>]

Hey everyone, recently we have gotten some really great community
contributions, one of which adds support for the AMQP messaging protocol to
carbon. I have started migrating some of my systems to using this and I
think it is a great fit for graphite. If you aren't familiar with it already
I highly recommend reading
http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for a brief
introduction.

One area I think AMQP can be especially useful is with a clustered graphite
installation. Most of you are probably not familiar with Graphite's
clustering capabilities because I have not documented it at all (sorry, hope
to change that soon). But essentially, you just setup N different
independent graphite servers and then configure the webapps to know about
one another. They then share data in a manner transparent to the user,
making each server look like one big graphite installation instead of
several independent ones. The tricky part is partitioning your metrics
across the servers. Thus far I've solved this problem with a daemon called
carbon-relay.py, which basically acts as an application-level load balancer
for your metrics. You are supposed to send all of your data to carbon-relay,
who then looks up some rules you've given it to decide which carbon-cache(s)
to relay each metric on to.

With AMQP, there seems to be a much simpler way to solve this problem. Topic
exchanges use a dot-delimited naming structure that supports pattern
matching very similar to graphite's. Basically, you could just publish
messages containing a value and a timestamp and use the metric name as the
routing key. Then each carbon-cache can consume from the exchange with one
or more binding patterns. For the simplest case of having only one server
the binding pattern would simply be "#" (which is the same as using a fanout
exchange). For more complex cases you could control what data goes to what
server by means of configuring each carbon-cache's binding patterns. This
would effectively replace carbon-relay, and I believe, solve the problem in
a more robust way.

This is a bit different than they way it currently works in trunk so I
wanted to run it by everyone and see what your thoughts are, especially if
you are already using AMQP or carbon-relay. I am currently in the process of
testing this configuration and if it works well I will try it in my
production system. If that goes well, I would like to include the new
behavior in this month's release. So please send me any comments, questions,
concerns, etc.

If you don't plan on using AMQP, that's fine too, the old interface is not
going away.

-Chris