graphite-dev team mailing list archive

[Michael Leinartas <mleinartas@xxxxxxxxx>]

Yeah, you can create your own finder and matching reader and just add
them to the list of finders in storage.py to start.

The Finder should expose the method find_nodes(query). This method
should take a single parameter, a storage.FindQuery object and yield
node.BranchNode objects and node.LeafNode objects. The creation of the
LeafNode objects are where your Reader will be instantiated and should
be whatever makes sense for your storage - for Whisper this is the
files themselves. For RRD files, it's a single datasource within the
file. If appropriate, it should take into account the requested
interval. Multiple yields of LeafNodes will be merged based on the
time intervals they contain (get_intervals() on the reader).

Your Reader should expose get_intervals() and fetch(startTime,
endTime). get_intervals should return an intervals.IntervalSet
composed of the Intervals (just start and end timestamps) contained in
the LeafNodes you'll be reading (whatever those may actually be).
fetch will return a tuple like this: ( (startTime, endTime, step),
[<list of data values>] ) - details about the data (actual timestamp
of the data, not necessarily the requested time (due to retentions,
rounding, etc) and the data itself.

Adding the settings.py config seems reasonable. I'd try to add it to
settings.py with logic to configure it as it already is from
local_settings.py. Those who need the advanced functionality of adding
their own readers can then add their own later on:
STORAGE_FINDERS.append(...).

Keep us updated on how it works out

michael

On Fri, Jun 15, 2012 at 5:44 PM, Jon Dugan <jdugan@xxxxxxxxx> wrote:
> Hello,
>
> I discussed this with Chris quite some time ago and he suggested that
> I wait until the Ceres patch landed.  Since it has landed I'd like to
> reopen the question.
>
> What is the best way to link in a storage back end other than the
> standard ones: Whisper, RRD and Ceres?
>
> It looks like there is a nod to this in webapp/graphite/storage.py
> starting at line 149:
>
> 149 # Exposed Storage API
> 150 finders = [
> 151   CeresFinder(settings.CERES_DIR),
> 152   StandardFinder(settings.STANDARD_DIRS),
> 153 ]
> 154 STORE = Store(finders, hosts=settings.CLUSTER_SERVERS)
>
> How do I plugin my own finder? This seems like a place holder for a
> storage API.   Obviously I can modify the code but then I have to
> track my own patches to Graphite. Which I have done for some time now
> and it's a bit tiring.
>
> My suggestion would be to enumerate the finders in the settings file.
> Something like this:
>
> STORAGE_FINDERS = [('graphite.finders.CeresFinder', CERES_DIR),
> ('graphite.finders.StandardFinder', STANDARD_DIRS)]
>
> And then creating an instance of each finder called with the second
> item of the tuple as an argument in storage.py.  And passing that list
> of finders to the Store.
>
> (IIRC there were some circular import issues if you just try to create
> a Store instance in settings.py so this sidesteps that.)
>
> I'd be happy to create a patch that does something like this, but I
> figured I'd get some input before I do it.
>
> Does this seem like a reasonable approach?  Does someone else have
> some other way of doing this already?
>
> Thanks,
>
> Jon
>
> _______________________________________________
> Mailing list: https://launchpad.net/~graphite-dev
> Post to     : graphite-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~graphite-dev
> More help   : https://help.launchpad.net/ListHelp