graphite-dev team mailing list archive

[Borom Ir <question220210@xxxxxxxxxxxxxxxxxxxxx>]

New question #220210 on Graphite:
https://answers.launchpad.net/graphite/+question/220210

Hi!

I'm trying to figure out Graphite's behaviour when the data points arrive "late".

My storage schema is very simple:
[default]
pattern = .*
retentions = 60s:7d, 5m:90d

And I'm sending data to RabbitMQ every minute (so this matches my storage schema). 
Things work pretty well at the moment, but I'm trying to figure out 1 edge case that is driving me crazy (probably just missing some configuration).

>From time to time, the network fails and Graphite can't connect to RabbitMQ (or I can just force the situation and stop Carbon for a while). This should be ok, as my metrics keep collecting in a queue which Graphite can process when connectivity is restored.

My problem is that once the connectivity is restored (or I start the Carbon agent again) and Graphite starts processing all these "buffered" messages, it doesn't store these metrics in whisper files, it seems like it just ignores these old timestamps and once it catches up with the current time, it will start inserting values in whisper files again, leaving huge gaps (I'm talking about hours here) in my whisper files (using whisper-fetch I can see all have the value "None"). 

I have checked the data in the queue and it is correct and has the right timestamp (well, actually my metrics are sent every 60s but not exactlly at HH::MM::00, some metrics will arrive at HH::MM::13, other will arrive at HH::MM::43, but I guess this is ok for an 60s interval), so:

1) Does Graphite require a continuous flow of data to work? I mean...what happens if It receives data points late? My understanding was that it didn't matter as long as my data point arrived with a Timestamp.
2) How does it behave when it receives data points out of order? That is...what happens if I receive first timestamp T3, then T1 and then T2? My understanding is that this was irrelevant

Does anyone have a clue or could point me out in the right direction so I can keep debugging this issue? I have run out of ideas...

Thanks!




-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.