graphite-dev team mailing list archive

[Scott Smith <question176084@xxxxxxxxxxxxxxxxxxxxx>]

New question #176084 on Graphite:
https://answers.launchpad.net/graphite/+question/176084

I've set up a 12-instance carbon-cache cluster with a single carbon-relay sending data to them. Here is the basic architecture:

host 2x8core, 12GB RAM
------
graphite-web
carbon-cache x4
200GB SSD x2

I have three of these instances running. with replication factor = 2.The carbon-relay receives metrics from my old graphite server, also running carbon-relay.

When I turned on the relaying of metrics to the new cluster, it created about 50k metrics in a couple hours. That began at ~2am Pacific last night. It's currently at just over 87k metrics and growing, but very slowly.

The old host has 59k metrics, so I expect the final count in the new cluster to be ~120k due to the replication factor setting.

Some data:

- There's nothing else but carbon-cache and Apache/mod_wsgi running on these.
- Each drive avg. ~400 writes/s over a minute sampling, never bursting over 2k writes/s.
- Each host has allocated ~10GB RAM for the buffer cache.

http://dl.dropbox.com/u/1613178/Screenshots/graphite-writeops.png
http://dl.dropbox.com/u/1613178/Screenshots/graphite-average-wait.png

Is there any obvious reason creations have slowed so much? What should I be looking for?

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.