graphite-dev team mailing list archive

[Nicholas Leskiw <nleskiw@xxxxxxxxx>]

Does everything have the same retention rate? If not, your server may be working on creating larger files with longer/more granular retention rates.  Also, as new files are created, you're now populating those 50k metrics with updates as well as creating new metrics.  

What do you see in the carbon stats in Graphite for those instances?

-Nick

Sent from a mobile device.
Please excuse terse language and spelling mistakes.

Scott Smith <question176084@xxxxxxxxxxxxxxxxxxxxx> wrote:

>New question #176084 on Graphite:
>https://answers.launchpad.net/graphite/+question/176084
>
>I've set up a 12-instance carbon-cache cluster with a single carbon-relay sending data to them. Here is the basic architecture:
>
>host 2x8core, 12GB RAM
>------
>graphite-web
>carbon-cache x4
>200GB SSD x2
>
>I have three of these instances running. with replication factor = 2.The carbon-relay receives metrics from my old graphite server, also running carbon-relay.
>
>When I turned on the relaying of metrics to the new cluster, it created about 50k metrics in a couple hours. That began at ~2am Pacific last night. It's currently at just over 87k metrics and growing, but very slowly.
>
>The old host has 59k metrics, so I expect the final count in the new cluster to be ~120k due to the replication factor setting.
>
>Some data:
>
>- There's nothing else but carbon-cache and Apache/mod_wsgi running on these.
>- Each drive avg. ~400 writes/s over a minute sampling, never bursting over 2k writes/s.
>- Each host has allocated ~10GB RAM for the buffer cache.
>
>http://dl.dropbox.com/u/1613178/Screenshots/graphite-writeops.png
>http://dl.dropbox.com/u/1613178/Screenshots/graphite-average-wait.png
>
>Is there any obvious reason creations have slowed so much? What should I be looking for?
>
>-- 
>You received this question notification because you are a member of
>graphite-dev, which is an answer contact for Graphite.
>
>_______________________________________________
>Mailing list: https://launchpad.net/~graphite-dev
>Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
>Unsubscribe : https://launchpad.net/~graphite-dev
>More help   : https://help.launchpad.net/ListHelp