graphite-dev team mailing list archive

[Styx <question280890@xxxxxxxxxxxxxxxxxxxxx>]

New question #280890 on Graphite:
https://answers.launchpad.net/graphite/+question/280890

Hello,

>From my observation and a quick look at the code, it looks like whisper is storing data in the files relative to the current time.

I.E.: I create a new whisper file (empty) with the following retention : 1s:5m 10s:1d 1m:30d 5m:90d 15m:1y
If the current date is January 1st 2016 and for whatever reason I want to store a datapoint from august 15h 2015, this datapoint (and its timestamp) will be stored directly in the last archive (15m:1y)

I am wondering if it would be easy to modify whisper's behavior to the following:
for a given whisper file, as a new datapoint is added to the file:
 => check its timestamp
 ==> if it's timestamp is newer than any timestamp previously recorded, make this new timestamp the relative "starting date" of the file. (if this is the first datapoint added to the file, then make its timestamp be the "starting date")
And then use this "starting date" instead of the current time.

This way, if I take my previous example again, this time, when I add my datapoint from august 15th 2015, it will be stored in all archives, and august 15th 2015 will be considered as the "starting date" of this whisper file.
Then, if I add a new datapoint to that whisper file long after that date, let say on December 25th 2015, then December 25th 2015 becomes the new "starting date" and this new datapoint get stored in all archives, while my previous datapoint gets removed form all archives but the last.

If this makes sense, the idea behind that is to allow to store in whisper some old data but still at a high precision as long as there has been no new datapoint added.

Let's say I have stored somewhere (a cvs file for example) some datapoint from last year at a precision of 10 seconds and just for a duration of 4 hours.
If I wanted to import them to the whisper database, with the current behavior of whisper and with the retention setup indicated above, it would get stored with a 15 minutes precision (so I would lose my 10s precision).
With the modified behavior, it would actually store it with a 10s precision.

Anybody has an idea if this could be done easily ?

If I am not clear in my explanation, let me know and I will try to explain better.

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.