graphite-dev team mailing list archive

[Mark Seger <question177524@xxxxxxxxxxxxxxxxxxxxx>]

New question #177524 on Graphite:
https://answers.launchpad.net/graphite/+question/177524

As the author of collectl, I'm always interested in finding better ways to plot real-time data and graphite sounds promising.  

One of collectls' hallmarks is its ability to collect a LOT of data at frequencies < 10 seconds, sometimes even at 1 second or less (yes, it CAN run as sub-second frequencies).  Anyhow, collectl is used on many of the top-500 super computers and tools like rrd had to be quickly dismissed for 2 main reasons:
- it simply couldn't keep up with the load: think (hundreds of metrics) X (1-3 thousand systems) X (frequency of 5-10 seconds)
- rrd isn't accurate!  it insists on ' nomalizing' your data for you and as a result when you plot it you lose information which is totally unacceptable when trying to diagnose complex problems

I'm happy to hear graphite has an API, which I haven't yet studied, and collectl does too, making it relatively easy to connect it to other systems.  Feels like it would be pretty easy to connect the two together.

BUT my main concern is how much data can it really handle?  I saw a reference to 160K metrics/minute in the FAQ, which sounds great until you reduce the monitoring frequency, which is about 25K metrics/10 seconds.  If a system is generating 25 metrics (collectl can actually do multiple hundreds, but I could live with a small number) that would mean it can handle about 1K systems at this rate.

I'm curious - have people looked at what it would take to scale up graphite by a factor of 10 or more?  Is it disk limited?  Do people feel that throwing more hardware at it would help?

in any event I am still interested and if I do find some time will look into connecting the two, unless of course someone else wants to raise their hand ;)

-mark

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.