graphite-dev team mailing list archive

[bhardy <question202816@xxxxxxxxxxxxxxxxxxxxx>]

Question #202816 on Graphite changed:
https://answers.launchpad.net/graphite/+question/202816

bhardy posted a new comment:
Petri,

I am doing some of the same testing now for local disks.  Everyone does
not have the liberty to move to SSDs.   :-)  Sorry if some of the info
below is Dr. Obvious but I was hoping to be thorough.  Hope it helps.

Like you, my systems have plenty of CPUs and Memory but the crux is disk
I/O.  I believe a good starting point is knowing how many metrics you
expect to receive each minute.  With this in hand you can test and tune
the three values below in carbon.conf.  You can also run multiple
carbon-cache daemons on one server possibly fed by a separate server
solely running carbon-relay.

MAX_CACHE_SIZE = 300000
MAX_UPDATES_PER_SECOND = 300
MAX_CREATES_PER_MINUTE = 500

Thus far in my testing I have found that you may want to set
MAX_UPDATES_PER_SECOND to a third of the metrics you expect to receive
each minute.  By not maximizing the updates setting, you force the
system to use cache.  It then writes multiple updates to the disk in an
optimal fashion (*seen in updates.log).

You can check your disk using the command below.  I believe in a former thread it was stated that ideally this should be between 20%-50% utilized.
iostat -dxk 30

Example of tuning the system.  (YMMV)      
metrics-per-min  /  updates-per-min  =  optimal writes 
60000  /  18000 =  3 optimal wites

updates-per-min is gathered from the MAX_UPDATES_PER_SECOND setting
(300/sec * 60 = 12000)

Additionally, I have the liberty of plenty of memory so I set
MAX_CACHE_SIZE to 5times the volume I receive each min.  (60000 * 5)

*After testing, you may also want to set LOG_UPDATES to false as this is
expensive as it writes a line for each metric.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.