graphite-dev team mailing list archive

[Chris Davis <chrismd@xxxxxxxxx>]

That's excellent. A fanout makes sense when your data doesn't need to be
partitioned across servers. Yes the MAX_UPDATES_PER_SECOND value is a very
important knob to tune to fit your storage scenario. If it's too high you
saturate your disk, if it's too low you'll max eventually max out your CPU
as the cache gets too large (the cache is periodically sorted, cache query
results take longer to serialize, etc...). The default of 1000 is a bit too
high. I have a 6-disk RAID10 array that works well with a value of around
800, handling about 300k metrics per minute.

-Chris

On Sat, Feb 13, 2010 at 10:52 AM, Allan <zirpu@xxxxxxxxx> wrote:

>
>
> --- On Fri, 2/12/10, Chris Davis <chrismd@xxxxxxxxx> wrote:
>
> > From: Chris Davis <chrismd@xxxxxxxxx>
> > Subject: [Graphite-dev] request for comments - using amqp and replacing
> carbon-relay
> > To: "graphite-dev" <graphite-dev@xxxxxxxxxxxxxxxxxxx>
> > Date: Friday, February 12, 2010, 2:00 PM
> > Hey everyone, recently we have gotten some
> > really great community contributions, one of which adds
> > support for the AMQP messaging protocol to carbon. I have
> > started migrating some of my systems to using this and I
> > think it is a great fit for graphite. If you aren't
> > familiar with it already I highly recommend reading
> http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for
> > a brief introduction.
> >
> > One area I think AMQP can be especially useful
> > is with a clustered graphite installation. Most of you are
> > probably not familiar with Graphite's clustering
> > capabilities because I have not documented it at all (sorry,
> > hope to change that soon). But essentially, you just setup N
> > different independent graphite servers and then configure
> > the webapps to know about one another. They then share data
> > in a manner transparent to the user, making each server look
> > like one big graphite installation instead of several
> > independent ones. The tricky part is partitioning your
> > metrics across the servers. Thus far I've solved this
> > problem with a daemon called carbon-relay.py, which
> > basically acts as an application-level load balancer for
> > your metrics. You are supposed to send all of your data to
> > carbon-relay, who then looks up some rules you've given
> > it to decide which carbon-cache(s) to relay each metric on
> > to.
> >
> > With AMQP, there seems to be a much simpler way
> > to solve this problem. Topic exchanges use a dot-delimited
> > naming structure that supports pattern matching very similar
> > to graphite's. Basically, you could just publish
> > messages containing a value and a timestamp and use the
> > metric name as the routing key. Then each carbon-cache can
> > consume from the exchange with one or more binding
> > patterns. For the simplest case of having only one server
> > the binding pattern would simply be "#" (which is
> > the same as using a fanout exchange). For more complex cases
> > you could control what data goes to what server by means of
> > configuring each carbon-cache's binding patterns. This
> > would effectively replace carbon-relay, and I believe, solve
> > the problem in a more robust way.
>
>
> We use a simple fanout exchange for our metrics with 1 carbon-cache
> server reading from that exchange.   Currently we're processing
> 75k+/minute of metrics using 1 pair of rabbitmq servers in 1 colo,
> receiving for all colos (4).   This works fairly well for us.
> The only problem we had was that carbon-cache was writing so hard
> to the disk that it slowed things down.  The solution was to
> set MAX_UPDATES_PER_SECOND from 1000 to 50.  This reduced the disk
> utilization from 100% to just around 30%.  Works great.
>
> -allan
>
>
> > This is a bit different than they way it
> > currently works in trunk so I wanted to run it by everyone
> > and see what your thoughts are, especially if you are
> > already using AMQP or carbon-relay. I am currently in the
> > process of testing this configuration and if it works well I
> > will try it in my production system. If that goes well, I
> > would like to include the new behavior in this month's
> > release. So please send me any comments, questions,
> > concerns, etc.
> >
> > If you don't plan on using AMQP, that's
> > fine too, the old interface is not going away.
> > -Chris
> >
> > -----Inline Attachment Follows-----
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~graphite-dev
> > Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~graphite-dev
> > More help   : https://help.launchpad.net/ListHelp
> >
>
>