graphite-dev team mailing list archive

[Slawomir Skowron <question226895@xxxxxxxxxxxxxxxxxxxxx>]

Question #226895 on Graphite changed:
https://answers.launchpad.net/graphite/+question/226895

Slawomir Skowron posted a new comment:
I'am using graphite-carbon and whisper from 0.9.12. All hosts, and
carbons have same configuration on all 3 machines in AWS.

MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 5000
MAX_CREATES_PER_MINUTE = 1000

io-waits are from 10-12% on this SSD drives on all machines.

But with MAX_CACHE_SIZE = 10000000 (or other big, or low number) all is droping down, and carbon-caches get 100% cpu and now metric not showing in web-app
That's why it is cache_size is inf

With this setup numbers in carbon-cache are:

update_operations ~ 50k
commited_points ~ 200k
received_points ~ 200k

avg update time = 0.002 sec

Strange is that some carbon caches making 25-30k update operations and
some of then only 10k. This carbon who's making 30k updates, flushing
into drives more offten and queues/cache size drops to 0 for a while and
than going up.

This carbon have much less reads, and little more writes. All host makes
in summary 10k iops (5k per carbon-cache)

When i look @ graphs, then some metrics not catching time with late
about 1-2 minutes (4 x 30sec updates)

For carbon daemons minimal interval is 10 seconds, and for diamond 30
sec.

Now whatever change i made in my carbon.conf for carbon-cache it's only
worst.

Other strange thing is that, when i run first time whole graphite, then
when creates happen all carbon-caches working perfect, but when creates
and data was filled on drives and updates catch received number of
metric all goes down to 50k updates from 200k updates and now toping low
values and it's not good :(

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.