graphite-dev team mailing list archive

[Nicholas Leskiw <question135119@xxxxxxxxxxxxxxxxxxxxx>]

Question #135119 on Graphite changed:
https://answers.launchpad.net/graphite/+question/135119

    Status: Open => Answered

Nicholas Leskiw proposed the following answer:
Where do you see the holes?

In the archived files or in your live data?  If in the archived files
are they gone by the next day's backup?

====================
If you see it in the archived files, 
and they're gone by the next day: 
====================
Carbon-cache specifically doesn't write to disk all the time; it waits until it can do a large enough write to make it "worthwhile" on a traditional spinning disk.  Most of the time of a write to a traditional disk is in the seek.  Writing one byte or two sequential bytes to disk takes nearly the same amount of time.  Since whisper metrics are all sequential, carbon queues up many datapoints before dumping to disk.  That's why you'd see 'missing data' if you were looking at copied whisper files; carbon hasn't written them to disk yet. 

Since you're using a RAM disk, you may be able to increase the
MAX_UPDATES_PER_SECOND setting (default 1000) in the carbon.conf file.
I'd run that in a test environment first,  and see how high of a value
your RAM disk can tolerate with 150,000 test metrics being sent to
carbon.  That will reduce the amount of data in the carbon cache by
writing to disk more often (which your RAM disk should be able to do...)

A more simple solution might be to schedule your backup to 10 or 15
minutes later, by then all those data points should be written to disk
and you'll just have some extra data after the official backup time.


====================
If you see it in the live data: 
====================

Slow down your copy.  Your RAM disk is probably going to far outpace
your hard disk drive.  Maybe use rsync --bwlimit=XXXX ? There may be so
many I/O operations that your whole system is slowing down during the
copy, forcing the usage of cache despite the RAM disk...

Please let me know if this helps.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.