graphite-dev team mailing list archive

[Petri Ojala <question215366@xxxxxxxxxxxxxxxxxxxxx>]

New question #215366 on Graphite:
https://answers.launchpad.net/graphite/+question/215366

I'm not sure if this is related to the other "send queue is full" cases but anyway..

I have a mid-sized graphite installation that receives about 600k metrics every 5 minutes (not in a single batch).  I have a mechanism to tell the data feeders that graphite is under maintenance and they will queue the incoming data in Redis.  The graphite is running with carbon-relay and four carbon-cache instances.

I saw the "send queue is full" message on the listener.log, mostly with "send queue has space available" on the same second.  Increasing the queue from 10k to 30k didn't seem to have much effect.  The send queue is full seems to happen for all the carbon-caches at the same time (I've assumed it would have a queue for each cache in case there's a lot of data coming in).

If I take the graphite down for maintenance, data starts to queue up, I seem to hip the "send queue is full" pretty much instantly when the queues are being flushed from Redis and data seems to get lost.

I tried to increase the max queue size from 30k to 3M but that seems to be a mistake as well.  Now there's a 1.7G carbon-relay process running and it has been running at full queue for the last 15 minutes.

Any ideas what would be the efficient way to be able to have Graphite down e.g. for an hour?  Or to be able to feed the queued ~7M datapoints to the graphite without hitting the send queue ceiling?

In the worst case I could limit the rate how fast the queued data is being fed as it's all under control.  Is there a way to check the queue sizes somehow from a living graphite?

I had some trouble with the data types on the pickle format using Perl and the data is currently fed through the line interface.  Does that make a difference in this case?


-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.