graphite-dev team mailing list archive

[harabk <question272229@xxxxxxxxxxxxxxxxxxxxx>]

New question #272229 on Graphite:
https://answers.launchpad.net/graphite/+question/272229

We are running one server with carbon on it - it is a 2core 2GB linux vm - storage is on an SSD SAN. With the following configuration:

CACHE_WRITE_STRATEGY = max
MAX_CACHE_SIZE = inf
USE_FLOW_CONTROL = True
WHISPER_FALLOCATE_CREATE = True
MAX_CREATES_PER_MINUTE = 100
MAX_UPDATES_PER_SECOND = 1000
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
LOG_LISTENER_CONNECTIONS = False
LOG_UPDATES = False
ENABLE_LOGROTATION = True
WHISPER_AUTOFLUSH = False

We are seeing carbon crash multiple times a day because it runs out of ram. Additionally, we are seeing 100% of 1 core utilized (with the other core sitting idle). Other relevant stats - iostat utilization is below 5%, avgUpdateTime increases, metricsReceived is between 60k-150k.

The latest change we have tried is changing CACHE_WRITE_STRATEGY = naive which has helped CPU utilization (dropped to 15%) and increased iostat utilization to between 5-30%. Haven't seen a crash thus far, but the change was made not too long ago.

We have been reading through what we think are relevant forum posts in trying to tune this configuration and would really appreciate any insight/tips on how to move forward. We would also like to avoid dropping data points, if possible.

-- 
You received this question notification because your team graphite-dev
is an answer contact for Graphite.