← Back to team overview

graphite-dev team mailing list archive

[Question #675575]: All of a sudden metrics getting lost

 

New question #675575 on Graphite:
https://answers.launchpad.net/graphite/+question/675575

I'm using graphite to monitor our systems, especially aerospike nodes. Till last week, I was able to see the historical data (3months, 6 months) of the metrics. Two days back when I checked, all the old metrics are gone and only a day of metrics were present. 
Today when I checked, only yesterday's data is present.

I read through forums and understood that metrics are retained as per definitions in /etc/carbon/storage-schemas.conf.
It has the default entries. 
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

I checked the creates.log and see the below entries.

15/10/2018 05:04:18 :: new metric instances.aerospike.dummy-v2-8.dummy.appeals_records_exonerated matched schema default_1min_for_1day
15/10/2018 05:04:18 :: new metric instances.aerospike.dummy-v2-8.dummy.appeals_records_exonerated matched aggregation schema default_average
15/10/2018 05:04:18 :: creating database file /var/lib/graphite/whisper/instances/aerospike/idstore-aerospike-dummy-v2-8/dummy/appeals_records_exonerated.wsp (archive=[(60, 1440)] xff=0.5 agg=average)

So as per my understanding, the metrics are matching the [default_1min_for_1day] tag and so they're dropped from the whisper storage.

My question is how all of a sudden the metrics are matching the [default_1min_for_1day] definition and getting dropped. Why they didn't match till last week?
I confirm that no changes are made in the configurations or the metrics exporters.

Also can you explain the term "The retentions line is saying that each datapoint represents 10 seconds, and we want to keep enough datapoints so that they add up to 14 days of data." referring to 'retentions = 10s:14d'?
Does it mean that all data will be retained for 14days? What will happen if i change the 10s to 1s?

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.