graphite-dev team mailing list archive

[Brian Hatfield <bmhatfield@xxxxxxxxx>]

Hi all,

I've been fighting Carbon for a few days now, and I've traced through the
code, but I am seeing an odd behavior: Carbon-Cache is not attempting to
write to disk until the cache is full (as defined by MAX_CACHE_SIZE).

See attached image for what it looks like.

Note that if I set MAX_CACHE_SIZE to 'inf', the growth line you see on the
left grows forever. It never drains, and I see little to no IO on the disk
until the system runs out of RAM and starts swapping.

I see in the code that we are supposed to be writing data continuously to
disk... it just doesn't seem to be happening.

def writeForever():
  while reactor.running:
    try:
      writeCachedDataPoints()
    except:
      log.err()

    time.sleep(1) # The writer thread only sleeps when the cache is empty
or an error occurs

This behavior is causing me all sorts of issues, because the queues back up
and then FLOW_CONTROL causes me to lose metrics.

I'm running a recent build of 0.9.10_pre1.

Any thoughts?

Thank you!
Brian Hatfield

Attachment: cache_size.png
Description: PNG image