graphite-dev team mailing list archive

[chrismd <question158830@xxxxxxxxxxxxxxxxxxxxx>]

Question #158830 on Graphite changed:
https://answers.launchpad.net/graphite/+question/158830

    Status: Open => Answered

chrismd proposed the following answer:
The original code block you posted is the reason why, when carbon
receives a datapoint for a metric it checks to see if the wsp file
already exists on the filesystem or not. If it doesn't, the wsp file
will be created but only up to MAX_CREATES_PER_MINUTE new wsp files each
minute, after which any datapoints for metrics that don't have wsp files
allocated yet get dropped until the next minute. So if all of your
datapoints are for new metric names, the committedPoints will be very
low as many datapoints will be getting dropped, but over time as the wsp
files get allocated for each metric it will rise and level off and no
datapoints will be dropped.

The reason creates are treated specially like this is because they have
a large impact on system performance. In particular, allocating new wsp
files at a rapid rate essentially fills up the system's I/O buffers
causing update operations to become synchronous and hence very slow,
which causes carbon's cache to grow, which eats up more memory, causing
less I/O buffer space, etc... it's a downward spiral that eventually
causes carbon to crash as it runs out of memory. However, if the creates
are rate limited this does not happen. Since creating a new metric is a
very rare event (with respect to the lifetime of the metric) this
generally isn't a big deal, it just means you have to wait a little
while for a big batch of new metrics to become visible in the tree. Once
the wsp file's have been allocated though, MAX_CREATES will have no
affect on throughput.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.