graphite-dev team mailing list archive

[Allan <zirpu@xxxxxxxxx>]

--- On Fri, 2/12/10, Chris Davis <chrismd@xxxxxxxxx> wrote:

> From: Chris Davis <chrismd@xxxxxxxxx>
> Subject: [Graphite-dev] request for comments - using amqp and replacing carbon-relay
> To: "graphite-dev" <graphite-dev@xxxxxxxxxxxxxxxxxxx>
> Date: Friday, February 12, 2010, 2:00 PM
> Hey everyone, recently we have gotten some
> really great community contributions, one of which adds
> support for the AMQP messaging protocol to carbon. I have
> started migrating some of my systems to using this and I
> think it is a great fit for graphite. If you aren't
> familiar with it already I highly recommend reading http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for
> a brief introduction.
> 
> One area I think AMQP can be especially useful
> is with a clustered graphite installation. Most of you are
> probably not familiar with Graphite's clustering
> capabilities because I have not documented it at all (sorry,
> hope to change that soon). But essentially, you just setup N
> different independent graphite servers and then configure
> the webapps to know about one another. They then share data
> in a manner transparent to the user, making each server look
> like one big graphite installation instead of several
> independent ones. The tricky part is partitioning your
> metrics across the servers. Thus far I've solved this
> problem with a daemon called carbon-relay.py, which
> basically acts as an application-level load balancer for
> your metrics. You are supposed to send all of your data to
> carbon-relay, who then looks up some rules you've given
> it to decide which carbon-cache(s) to relay each metric on
> to.
> 
> With AMQP, there seems to be a much simpler way
> to solve this problem. Topic exchanges use a dot-delimited
> naming structure that supports pattern matching very similar
> to graphite's. Basically, you could just publish
> messages containing a value and a timestamp and use the
> metric name as the routing key. Then each carbon-cache can
> consume from the exchange with one or more binding
> patterns. For the simplest case of having only one server
> the binding pattern would simply be "#" (which is
> the same as using a fanout exchange). For more complex cases
> you could control what data goes to what server by means of
> configuring each carbon-cache's binding patterns. This
> would effectively replace carbon-relay, and I believe, solve
> the problem in a more robust way.


We use a simple fanout exchange for our metrics with 1 carbon-cache
server reading from that exchange.   Currently we're processing
75k+/minute of metrics using 1 pair of rabbitmq servers in 1 colo,
receiving for all colos (4).   This works fairly well for us.
The only problem we had was that carbon-cache was writing so hard
to the disk that it slowed things down.  The solution was to 
set MAX_UPDATES_PER_SECOND from 1000 to 50.  This reduced the disk
utilization from 100% to just around 30%.  Works great.

-allan


> This is a bit different than they way it
> currently works in trunk so I wanted to run it by everyone
> and see what your thoughts are, especially if you are
> already using AMQP or carbon-relay. I am currently in the
> process of testing this configuration and if it works well I
> will try it in my production system. If that goes well, I
> would like to include the new behavior in this month's
> release. So please send me any comments, questions,
> concerns, etc.
> 
> If you don't plan on using AMQP, that's
> fine too, the old interface is not going away.
> -Chris
> 
> -----Inline Attachment Follows-----
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help   : https://help.launchpad.net/ListHelp
>