graphite-dev team mailing list archive

[Tomas Neme <question242459@xxxxxxxxxxxxxxxxxxxxx>]

New question #242459 on Graphite:
https://answers.launchpad.net/graphite/+question/242459

I'm having a problem sending pickled metrics

I'm running a stress test on my graphite setup, using wishbone and metricfactory https://github.com/smetj/metricfactory

I added a pickle formatting option to wishbone's graphite formatter, but it looks like the pickle listener kills the socket after each transmission and also I can't send more than 2500 metrics per send. If I try to send more than this, python's socket.sendall just blocks forever (or times out, if I'm smarter about it)

that effectively makes plaintext faster than pickle, I'm crunching about 36k metrics per minute in plaintext and about 2k with pickle.

There's a number of things that could be optimized on the wishbone side, but I tried making sure the socket is created before sending every pickle, and even with that I cap at 12k. am I barking up the wrong tree here? or are there more things I might be looking at in wishbone's main loop that might be keeping me slow?

And since plaintext works fine, is it OK for benchmarking, in case I can't fix this issue? My goal is to benchmark the processing side, rather than the network

Thanks
Tomas


-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.