graphite-dev team mailing list archive

[Jordan Bernstein <question225623@xxxxxxxxxxxxxxxxxxxxx>]

New question #225623 on Graphite:
https://answers.launchpad.net/graphite/+question/225623

We've recently added a lot more metrics to our graphite instance and it seems to be having some issues with performance. Mostly, some of our diamond collectors occasionally error out sending to graphite. We've tweaked a few settings here and there and there's no obvious disk IO load or anything like that. I checked the console log but didn't find anything indicating it ran out of file descriptors. The next thing we thought was to add more carbon caches. Right now, it's setup in an HA-style relay setup with 2 graphite notes.

We wanted to add another carbon-cache to each of these nodes in order to boost performance. I guess the first question is, is this even a valid idea? Should we explore other avenues? Everything is setup to relay, so I'm not sure if we need to switch to consistent hashing, or what other changes to the config might be necessary. I've got our configs below. Any advice on how to proceed would be appreciated! 

[cache]
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
USER = www-data
PID_DIR = /var/run/graphite
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 7500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7012
LOG_UPDATES = False

[relay]
# Relay setup
RELAY_METHOD = rules
DESTINATIONS = graphite1:2014, graphite2:2014
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
MAX_QUEUE_SIZE = 100000


relay-rules.conf:
[default]
default = true
destinations = graphite1:2014, graphite2:2014



I came up with an experimental config, adding in a [cache:b] section, and added it into the relay hosts/destinations accordingly, but without consistent hashing it seemed like this wouldn't be doing anything useful, what I tried was something like below:

[cache]
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
USER = www-data
PID_DIR = /var/run/graphite
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 7500
MAX_CREATES_PER_MINUTE = 50
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7012
LOG_UPDATES = False
WHISPER_LOCK_WRITES = True

[cache:b]
LINE_RECEIVER_PORT = 2113
PICKLE_RECEIVER_PORT = 2114
CACHE_QUERY_PORT = 7112

[relay]
# Relay setup
RELAY_METHOD = rules
DESTINATIONS = graphite1:2014, graphite2:2014, graphite1:2114, graphite2:2114
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2004
MAX_QUEUE_SIZE = 100000

relay-rules.conf:
[default]
default = true
destinations = graphite1:2014, graphite2:2014, graphite1:2114, graphite2:2114

However, I'm really not sure how useful that is, it seems like it would just be further duplicating data but not actually giving any sort of performance improvement anywhere. Any ideas of how to proceed?

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.