graphite-dev team mailing list archive

[Daniel Roschka <question242885@xxxxxxxxxxxxxxxxxxxxx>]

New question #242885 on Graphite:
https://answers.launchpad.net/graphite/+question/242885

We're currently in progress of adding some redunancy to our setup, to still have all metrics available when one host fails. In this process we figured out some issues with synchronization of metrics between multiple carbon-cache instances.

Let's assume the following setup: Four hosts with the following services running:
host A: carbon-relay, configured with RELAY_METHOD = consistent-hashing, REPLICATION_FACTOR = 2 and host B and host C as DESTINATIONS
host B: carbon-cache
host C: carbon-cache
host D: graphite-web with host B and host C configured as CLUSTER_SERVERS

When both carbon-cache hosts (B and C) are running, graphite-web will fetch metrics from host B. When host B is not available it will get the metrics from host C. But host B won't get metrics while it's been not available, so after it's been available again graphite-web will happily ask host B for the metrics again, but won't get the metrics stored on host C during the outage of host B.

So how to ensure metrics on different hosts are in sync?
A solution that comes to mind would be to use rsync to synchronize them regularly, but that doesn't work if each carbon-cache has a different set of metrics stored (REPLICATION_FACTOR != number of carbon-cache instances).

Any suggestions how to solve that issue?

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.