graphite-dev team mailing list archive

[Glen <question285178@xxxxxxxxxxxxxxxxxxxxx>]

New question #285178 on Graphite:
https://answers.launchpad.net/graphite/+question/285178

Hello,

I'm really scratching my head here. We've been running a grafana/graphite-api/carbon/whisper stack for a while now and it's working generally ok. However, I've noticed that if I drill into data in grafana, once I get to a certain level of detail, the chart is blank.

Here is some config. Our storage schema looks like this, store on a 10 sec interval for 7 days, then 1 minute for 2 years.

[Web_Prod]
priority = 90
pattern = ^Production.*.web.*.WebServer.*
retentions = 10s:7d,1m:2y 

I can verify this in the whisper files themselves, like this: -

/usr/local/src/whisper/bin/whisper-dump.py /opt/graphite/storage/whisper/Production/Live/web/web2-vm/WebServer/Customer/HPS.wsp | less

Meta data:RETURN)
  aggregation method: average
  max retention: 63072000
  xFilesFactor: 0

Archive 0 info:
  offset: 40
  seconds per point: 10
  points: 60480
  retention: 604800
  size: 725760

Archive 1 info:
  offset: 725800
  seconds per point: 60
  points: 1051200
  retention: 63072000
  size: 12614400

I've noticed the problem only happens, when querying data older than 7 days i..e after it's been averaged to a 60 second interval. If I pick a time period older than 7 days, across a three minute interval, and look directly inside the whisper file, it all looks good: -

/usr/local/src/whisper/bin/whisper-fetch.py --from 1454230700 --until 1454230880 /opt/graphite/storage/whisper/Production/Live/web/web2-vm/WebServer/Customer/HPS.wsp

1454230740      8.000000
1454230800      8.700000
1454230860      8.233333

However, if I query through graphite-api, it returns a 10 second interval (the wrong retention period, because I'm querying older than 7 days), and all items (even the ones that match the timestamps above) are null.

http://www.dashboard.com/render?target=Production.Live.web.web2-vm.WebServer.Customer.HPS&from=1454230700&until=1454230880&format=json&maxDataPoints=1000

[{"target": "Production.Live.web.571854-web2-vm.WebServer.GAME.HPS", "datapoints": [[null, 1454230710], [null, 1454230720], [null, 1454230730], [null, 1454230740], [null, 1454230750], [null, 1454230760], [null, 1454230770], [null, 1454230780], [null, 1454230790], [null, 1454230800], [null, 1454230810], [null, 1454230820], [null, 1454230830], [null, 1454230840], [null, 1454230850], [null, 1454230860], [null, 1454230870], [null, 1454230880]]}]

If I go for a wider time span, I start to get data back, but some are null and some are populated. What am I doing wrong?!

Thanks,
Glen.

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.