graphite-dev team mailing list archive

[D DC <question244564@xxxxxxxxxxxxxxxxxxxxx>]

Question #244564 on Graphite changed:
https://answers.launchpad.net/graphite/+question/244564

D DC gave more information on the question:
Adding more information:

I started carbon-cache in debug mode and it ran for 3 days without any problem. Max cache-size reached 480k (configured as  MAX_CACHE_SIZE = 25000000, so no limit problem during that time). 
Yesterday at 22:30 the cache size started to increase drastically, no data written on storage anymore, and eventually the 25M cache limit was reached in a matter of few hours, as per the log info:
19/07/2014 22:05:58 :: [console] Sorted 28049 cache queues in 0.014046 seconds
19/07/2014 22:17:50 :: [console] Sorted 28122 cache queues in 0.016169 seconds
<- last line in console.log.

Same for the debug messages, the updates were fine until that time, and then there was a huge delay until the next message: Max memory reached, see here:
19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.x.disk-sda.disk_time.read in 0.00916 seconds
19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.y.disk-sdad.disk_ops.write in 0.00929 seconds
19/07/2014 22:28:02 :: [updates] wrote 13 datapoints for server.x.disk-sdp.disk_octets.read in 0.00886 seconds
20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000000
20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000001
20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000002
20/07/2014 12:30:18 :: [console] MetricCache is full: self.size=25000003
.... and so on...


As we can see there is a huge 14 hours gap, while no activity occurred on the storage, and the cache memory increased to the limit of 25M.

In the end the process hang.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.