graphite-dev team mailing list archive

[Jason Antman <question227319@xxxxxxxxxxxxxxxxxxxxx>]

Question #227319 on Graphite changed:
https://answers.launchpad.net/graphite/+question/227319

Jason Antman posted a new comment:
Cliff,

I'm currently using the F5 in front of Graphite, but I still have not
had a relay failure, so I'm not sure how well it actually works. I was
planning on writing a blog post detailing our setup, but haven't yet, so
here's the gist of it. This is largely based on Jeff Forcier's post at
http://bitprophet.org/blog/2013/03/07/graphite/

We have a cluster of 3 graphite servers. Each one runs two instances of
carbon-cache (what we found as the best starting point for balancing CPU
and IO) and two instances of carbon-relay - a "master" and a "local". I
have all incoming metrics going to a VIP on the F5. I then have 2 pools
(one for port 2003 and one for 2004) configured identically. Each pool
points at the "carbon-relay-master" process on the cluster nodes, but
has them set up in priority groups. This way, we're only sending the
incoming metrics to one "master" carbon-relay process at a time, but the
other hosts have an identically configured process ready to take over.
The "master" carbon-relay process uses consistent hashing to distribute
metrics to the "local" carbon-relay on each node, which then balances
between the local carbon-caches.

A lot of this would change - and be a lot simpler - if/when the
"megacarbon" rewrite comes out. But that's alpha now, and with what I've
seen of the pace of releases, that's probably years away from beta if
we'll ever see it at all (graphite devs feel free to say otherwise...)

So the F5 side of it seems to work fine just using the standard TCP
profile on the F5, a tcp_half_open health check (I had issues with the
regular tcp check), and priority groups to balance between already-
running relays on the nodes.

If I finish writing up the full docs on how I did this, I'll update with
a link.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.