graphite-dev team mailing list archive

[Petri Ojala <question202816@xxxxxxxxxxxxxxxxxxxxx>]

New question #202816 on Graphite:
https://answers.launchpad.net/graphite/+question/202816

I wonder what are the recommended file system optimizations people are using when not running Graphite on SSD disks or high-performance RAID-arrays?   buffers?  noatime?  other options? ext2/3/4?  etc


I'm currently feeding about 60k/min + 450k/5min metrics to our graphite server.  The data is being collected on other server, sent to Redis queues on the graphite server for simple post-processing and fed to graphite locally (localhost) as ~2000 datapoints/pickle bundle.  I was hoping to feed everything /min but that just eventually fills the cache and the disk i/o cannot keep up with it -- and we would like to feed other data to the system as well.

The graphite server itself is 8-core/32G/RAID10 system.  The disk configuration is mostly for reliability, not for IOPS-performance that whisper seems to require.  There is one carbon-relay and three carbon-caches running, MAX_CACHE_SIZE = 8000000 and MAX_UPDATES_PER_SECOND = 500.  I have also set WHISPER_AUTOFLUSH to True.  The server is running Debian and running iostat 1 gives %iowait between 20 and 30% most of the time.  Cache size seems to stabilize at around 500k for each carbon-cache although I still need to run the system long enough to see how the values settle.

Are there any guidelines how the carbon.* metrics should behave when the system performance is good?  Running a smaller setup with just a few thousands metrics seems to be easy but as the numbers grow, things starts to feel "hit or miss".  Switching to SSD is obviously one solution for disk i/o which seems to be the only real bottleneck.

I am also running webapp on another host (NFS mounted the whisper files, CARBONLINK_HOSTS points to the actual graphite server over the network) -- not sure if it works 100% properly yet, though.  The idea is to keep the webapp, data collecting and storage separate.  The need for the webapp is somewhat limited as most of the data will be accessed just as JSON objects used as data for Flot graphs.

All hints and ideas how to improve the setup are also very welcome, especially wrt to keeping the webapp "away" from the carbon/whisper storage.


-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.