graphite-dev team mailing list archive

[chrismd <question121579@xxxxxxxxxxxxxxxxxxxxx>]

Question #121579 on Graphite changed:
https://answers.launchpad.net/graphite/+question/121579

    Status: Open => Answered

chrismd proposed the following answer:
Hi Bryant, actually I'd recommend using a checkout of trunk instead of
0.9.6. I really need to cut a new release as there have been a lot of
fixes since 0.9.6. As for PypeD, the only purpose that ever served was
offloading the work of multiplexing lots of client connections
(particularly from the ITA farms). It was better suited to that task
than carbon because it was written with the Twisted framework, but a
while ago carbon was rewritten to use Twisted as well so pyped really
isn't necessary anymore.

Your machine specs sound really good, and using a SAN and Linux with
separate mounts is excellent. Have you run into any particular
performance bottlenecks in this new setup? I've got a 2 node graphite
cluster where each machine has a local RAID array instead of a SAN mount
and it handles ~250k datapoints per minute very well so you should be
able to handle your load easily.

One thing thing that is important to know is that having lots of clients
sending data directly to carbon using the plaintext ("metric-name value
timestamp\n") protocol isn't particularly scalable. The most scalable
approach is to have fewer clients send the datapoints in large batches
via the pickle protocol (connect to port 2004 and send length-prefixed
pickled lists of (metric, value, timestamp) tuples). If that doesn't
make sense let me know and I can explain more thoroughly. That really
only matters if carbon is using lots of CPU, as the pickle method can be
much less CPU intensive if done right. Anyways, let me know what if any
bottlenecks you've run into in load testing your system.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.