graphite-dev team mailing list archive

[Michael Leinartas <question188509@xxxxxxxxxxxxxxxxxxxxx>]

Question #188509 on Graphite changed:
https://answers.launchpad.net/graphite/+question/188509

Michael Leinartas proposed the following answer:
Unfortunately, I know of no good way to do a rebalance while using
Whisper. As you note, the files need to be moved to match the location
of their new writer or they'll make graphs inconsistent because of the
way the webapp behaves.

Any method I can think of is a pain in the ass and requires some scripting:
1. Start up the new configuration and allow metrics to send, marking the time the new configuration was started. Go through all of the files and fetch their data up until the changeover time, sending back into carbon (basically backloading all of your data). You can restrict it only to files that havent been updated since the change. The data can be fetched with whisper-fetch.py or by coding against the whisper module in python.
2. Write something to rearrange files based on the actual consistent hashing code (taking a downtime while the rsync or copy takes place). This requires some python coding as well.

The promising part of this is that the new database format, Ceres,
handles this case  - files for the same metric can be located on
multiple servers and be combined by the webapp. Unfortunately it'll
likely be at least a month until this code is in trunk (not until after
0.9.10 can be released).

Because of these problems, it seems a lot of people requiring multiple
carbon-caches spread on different machines are instead using the relay-
rules method. It's more work to configure and get balanced, but is
explicit and leaves the controls in your hands.

If you do want to go forward and prepare something, let me know if I can
help. Perhaps we can put something together that can solve this case in
a general fashion for others to use.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.