graphite-dev team mailing list archive

[Steve Brunton <question83274@xxxxxxxxxxxxxxxxxxxxx>]

Question #83274 on Graphite changed:
https://answers.launchpad.net/graphite/+question/83274

Steve Brunton posted a new comment:
A little update in case anyone is interested. After looking through the
render function a little I realized it's a sequential request through
whisper as you ask for more than one datapoint per request. I ended up
using the python 2.6 multiprocessing module to abuse me some cpu time. A
calculator process fires up per range and finds all the items per that
range it needs to calculate data for. Each caculator then breaks up
those items into groups of five and creates up to five processes to make
requests against graphite render to get the data back and continues
creating them until it has retrieved all the data sets that it needs.
Then they each do all their calculations and store the data off. On the
production hardware 4,581 data items is taking about 10 miuntes from
start to finish. Each calculator range also has a flock() so they don't
stomp on each other if they run for more than fifteen minutes. The final
result is the popularity calculation that is used to generate the
results on the new NewsPulse product for CNN.com which launches this
weekend.

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.