graphite-dev team mailing list archive

[Michael Leinartas <mleinartas@xxxxxxxxx>]

Yeah, that's quite crappy behavior you're seeing. I cant immediately
think of anything that could be triggering that behavior. I assume
you've been running just fine on a previous version (0.9.9 or
somewhere in between)? There have been a few fixes to cache full
behavior, but the big one there was that metric receiving was not
getting unpaused (with flow control turned on) until the cache was
emptied (https://github.com/graphite-project/carbon/commit/becf65801b178e17412d6e6a21f9db4f1c560c08#lib/carbon/writer.py).
Other than that, this is the other behavioral change I see since 0.9.9
in the writer/cache logic:
https://github.com/graphite-project/carbon/commit/494e5e14426451ce21964ef7d9501bdd77a30281#lib/carbon/cache.py

As I dont have any suggestions to give right now, I'm just going to
ask a few questions to hopefully uncover a hint as to what's going on:

Are you using USE_FLOW_CONTROL? It sounds like no since you say
metrics are being dropped.. Have you tested whether the behavior is
any different with it on?

Have you verified with the pointsPerUpdate, updateOperations, and
committedPoints metrics that there is indeed *no* writing until the
cache is full?

What's your MAX_UPDATES_PER_SECOND set to?

Are you seeing the "Sorted X cache queues in Y seconds" messages
regularly in the logs while the cache is filling up?

On Wed, May 16, 2012 at 11:06 AM, Brian Hatfield <bmhatfield@xxxxxxxxx> wrote:
> Hi all,
>
> I've been fighting Carbon for a few days now, and I've traced through the
> code, but I am seeing an odd behavior: Carbon-Cache is not attempting to
> write to disk until the cache is full (as defined by MAX_CACHE_SIZE).
>
> See attached image for what it looks like.
>
> Note that if I set MAX_CACHE_SIZE to 'inf', the growth line you see on the
> left grows forever. It never drains, and I see little to no IO on the disk
> until the system runs out of RAM and starts swapping.
>
> I see in the code that we are supposed to be writing data continuously to
> disk... it just doesn't seem to be happening.
>
> def writeForever():
>   while reactor.running:
>     try:
>       writeCachedDataPoints()
>     except:
>       log.err()
>
>     time.sleep(1) # The writer thread only sleeps when the cache is empty or
> an error occurs
>
> This behavior is causing me all sorts of issues, because the queues back up
> and then FLOW_CONTROL causes me to lose metrics.
>
> I'm running a recent build of 0.9.10_pre1.
>
> Any thoughts?
>
> Thank you!
> Brian Hatfield
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help   : https://help.launchpad.net/ListHelp
>