graphite-dev team mailing list archive

[Kevin Blackham <question148357@xxxxxxxxxxxxxxxxxxxxx>]

New question #148357 on Graphite:
https://answers.launchpad.net/graphite/+question/148357

I recently sliced up my graphite installation into two systems.  The federation-fu is working.  Nice work!  However, many of my common graphs are quite complex, using wildcards and sumSeries() for example.  They are now very, very slow when the data resides on remote storage.  Here's one plot target from a "suck-o-meter" (ratio of connections active vs those storing data):

movingAverage(divideSeries(sumSeries(app.tds.ut4.[0-9]*.https),sumSeries(app.tds.ut4.[0-9]*.storeData)),6)

This one line expands to 270 wsp files.  In total, the whole graph needs 1,540 files.  Each one of these matches results in a single request for pickle data: "GET /render/?target=app.tds.ut4.20-1.storeData&pickle=true&from=1299484963&until=1299614563 HTTP/1.1" 200 532 "-" "-".  In the 19 hours it's been working, I've logged 5 million hits for pickle requests (5 minute dashboard redraws).  Once it's rendered, memcache saves me, but the initial load is at least 10x slower than it is when the data is on local disk.  This causes great pain if I'm working in the composer, trying to mine out some performance data, whatnot.

This seems to be expected behavior, probably because the django app will do a find for who has the needed .wsp without regard for the relay-rules.conf.  It is possible someone, even myself, may start slicing things up in even more complicated ways, duplicating updates to multiple systems, etc., so I think I get why it's doing this.

So, my question is:  Am I doing something wrong here? (other than not sending my requests to the host that has them local)  If this is expected behavior, perhaps we can find some optimization?  For example, keep the target= in the /render/ pickle request as a wildcard.  It should return all the same results as the /metrics/find/ request did.  Possible issues here when there is duplication of data?

[cue joke about one big pickle in the pickle jar]

Kevin Blackham
Mozy


-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.