graphite-dev team mailing list archive

[Zdeněk Pižl <question246639@xxxxxxxxxxxxxxxxxxxxx>]

Question #246639 on Graphite changed:
https://answers.launchpad.net/graphite/+question/246639

Description changed to:
Greetings,

 I am facing strange behaviour  of Graphite infrastructure - there are
latest datapoints missing from graphs.

My infrastructure contains:
a) 1x dedicated graphite-web for unique access point for any dashboard tool
b) 2x relay hosts running carbon-relay only
c) 2x storage hosts running carbon-cache (and graphite-web) only

The intention of this deployment design is as follows:

- any node sends datapoints metrics to one from defined relay hosts,
- this relay hosts know about all (currently 2) storage nodes and using consistent-hashing distributes datapoints among them,
- storage node with 'MAX_UPDATES_PER_SECOND = 50' and 'MAX_CACHE_SIZE = inf' saves datapoints to whisper database

The issue I am facing is that mostly in each graph there are missing
latest datapoints from some metrics. But after some time and multiple
reloads of graph (probably after saving those missing datapoint to
whisper finally) those datapoints  have appeared, but again the new
latest ones are missing, and so on ...

My dashboarding tool (Grafana) is asking the graphite-web node (a) which
is aware of all carbon-cache (storage) nodes defined in CLUSTER_SERVERS
directive of local-setings.py.

What could be the reason of missing data? I assume there should be at
least datapoints from all metrics up to some time but not in spread for
i.e. 5 minutes.

Thank you for your ideas and help, regards .zp.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.