graphite-dev team mailing list archive

[madireddy <question110431@xxxxxxxxxxxxxxxxxxxxx>]

Question #110431 on Graphite changed:
https://answers.launchpad.net/graphite/+question/110431

    Status: Answered => Open

madireddy is still having a problem:
Amazing! That was quite a tour of carbon that you gave me thanks for
that.

Just two quick questions.
1) Is there way to set the limit on the no of datapoints that get cached?
2) I can use any wsgi framework to work along side carbon right?

Sam
"""
Anyday...
Above the ground and vertical
... is a good day!
"""
http://megam.info
:)~



On Fri, May 14, 2010 at 9:52 AM, chrismd <
question110431@xxxxxxxxxxxxxxxxxxxxx> wrote:

> Your question #110431 on Graphite changed:
> https://answers.launchpad.net/graphite/+question/110431
>
>     Status: Open => Answered
>
> chrismd proposed the following answer:
> You're quite right Sam, the docs are very much lacking. I'd be glad to
> answer any specific questions you have though about how carbon works.
> There is some good information in this questions forum too.
>
> Carbon used to be 3 daemons, but it was rewritten as a single daemon
> using the Twisted framework. This unified carbon daemon is carbon-
> cache.py. Its job is to listen for datapoints from clients and to write
> them to disk as fast as possible using the whisper database library.
> When the local disk cannot keep up with the incoming data, carbon
> accrues a cache of data. The larger this cache gets the more efficient
> carbon becomes (more datapoints get compacted into each write() system
> call, each write() call is typically the same cost regardless of size
> since it always fits in a single page). Essentially the cache grows
> until carbon is efficient enough (the rate at which it writes datapoints
> to disk becomes high enough) to keep up with the rate of data you send
> it. This can be quite large, one of my systems sustains a cache size of
> approximately 2 million datapoints, equivalent to about 7 minutes worth
> of data for about 300,000 metrics. Whenever a substantial external I/O
> operation occurs on the system (like doing a backup) the kernel's I/O
> cache gets polluted with data other than carbon's and thus its
> throughput can slow down dramatically. When this happens carbon's cache
> simply grows until it matches the rate necessary to bring the system
> back into equilibrium. Just a few weeks ago my system had such an issue
> and the cache grew to over 10 million datapoints (30 minute backlog of
> data) and then recovered without any noticeable impact.
>
> Another important aspect of carbon's functionality is that when the
> graphite webapp gets a request to render a graph, it fetches datapoints
> off the local disk using the whisper library *and* it queries carbon-
> cache over a TCP socket to retrieve any cached datapoints. It then
> combines the results to give a graph with real-time data even when there
> is a perpetual backlog of several minutes of data in the cache.
>
> The carbon-relay.py program is only useful when you've got a cluster of
> graphite servers, and even then it is optional. It's job is to listen
> for datapoints the same way carbon-cache does but then it simply relays
> the datapoints along to one or more carbon-cache processes in your
> cluster based on a set of rules you give it. It's basically an
> application-level load balancer. It can also be used to duplicate
> datapoints to more than one carbon-cache server for redundant storage of
> the data.
>
> That's the 2 minute tour of carbon, let me know if you have any other
> questions. I'd be glad to help.
>
> -Chris
>
> --
> If this answers your question, please go to the following page to let us
> know that it is solved:
>
> https://answers.launchpad.net/graphite/+question/110431/+confirm?answer_id=0
>
> If you still need help, you can reply to this email or go to the
> following page to enter your feedback:
> https://answers.launchpad.net/graphite/+question/110431
>
> You received this question notification because you are a direct
> subscriber of the question.
>

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.