graphite-dev team mailing list archive

[Christopher Swingler <question271250@xxxxxxxxxxxxxxxxxxxxx>]

New question #271250 on Graphite:
https://answers.launchpad.net/graphite/+question/271250

Hello Graphite Launchpad Community,

I'm having issues with an infinitely-growing carbon cache. 

I have just upgraded our Graphite install from 0.9.12 to 0.9.13. The source of the 0.9.12 package was the one supplied with Ubuntu 14.04 LTS; 0.9.13 was made by rebuilding the packages using the .dsc files from Debian Unstable.  

The upgrade seemed to have gone without any significant hitches, and no change was made to the configuration immediately afterward.

The following day, I'd noticed that our queue size had grown from nominal to more than 12 million.  MAX_CACHE_SIZE had been set to "inf", but in 0.9.12 things the cache stayed more or less within reason. 

http://chrisswingler.com/spr/graphite/render-6.png

Also, we noticed that the number of points received by carbon-cache, and those committed to disk, starting diverging by about 10k points.

http://chrisswingler.com/spr/graphite/graphite.png

It seems that that's what we're getting stuck in the cache. If we overlay the derivative of the cache size + committed points, that pretty much lines up with received points.

I/O load isn't fantastically high, and we just plain cannot seem to convince Carbon to purge that cache short of a restart.

I've tried the following:
* Tuning MAX_UPDATES_PER_SECOND up and down. This does have an effect on the i/o load, but nothing on getting the cache to purge.

* Setting MAX_CACHE_SIZE to something other than "inf". This ends up being catastrophic - everything's fine until we hit that point, and then we start dropping points. 

* Declare QUEUE_LOW_WATERMARK_PCT to 0.8 (previously this was unset, this is new in Carbon 0.9.13).

Nothing we've done seems to have any useful effect. I'm leaning toward this being a bug, but I haven't seen anyone complain about a similar issue.

Our carbon config can be reviewed at 
https://gist.github.com/cswingler/a2b2c2643a10b47f61aa


Thanks!

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.