graphite-dev team mailing list archive

[Chris Larsen <question223956@xxxxxxxxxxxxxxxxxxxxx>]

New question #223956 on Graphite:
https://answers.launchpad.net/graphite/+question/223956

Hello! I pinged Chris Davis a while back about modifying the awesome Graphite web app with OpenTSDB as an optional backend and he liked the idea. We’re working on the 2.0 release that will provide a full JSON API and meta-data with organization in a tree-like structure that can be plopped directly into Graphite. We have a bunch of users who love Graphite (including me) because of the gorgeous graphs, dashboard saving and loads of functions. But we also need massive write scalability and with a 4 node, standard hardware, non-SSD Hadoop/HBase cluster you can easily exceed 100,000 writes per second. Need more throughput? Just add more nodes. Hadoop handles all of the sharding/redundancy for you so if a node croaks, you’re still up and running. Plus we’re working on Cassandra as an optional backend (since Hadoop/Hbase had a fair learning curve) and making it extensible so users can add different data stores.

I’ve already written a first draft of the TSDB code and created a super ugly hack in the 0.9x graphite-web branch (on accident, I should have worked on 0.10) that you can peek at here https://github.com/manolama/graphite-web/tree/0.9.x. My fork can pull a list of timeseries from OpenTSDB to display in the tree or navigate through the dashboard and then it will kinda graph the data. I say kinda because there are issues to work out with timing normalization and scaling values.  My first attempts at Python were hacking graphite-web so I need tons of help to do it right and here’s what I’m thinking for the 0.10 branch:

1) We’d need an abstraction layer between the GUI/Javascript/renderers and the data store. Right now it’s very intertwined and all of the front-end code expects the backend to be path based with “.” delimiters
2) The Readers and Finders code is going the right direction but I think more of the dependent code could move into these. E.g. maybe there would be something like this http://imgur.com/FkxEwCE
3) Nodes/Leaves should ID a timeseries with a generic ID field and provide a display_name field for GUI use
4) The ID would map to the path for Ceres, Whisper and RRDs but would map to the TSUID in OpenTSDB or any other system
5) Search would have to be implemented for each main abstraction layer

Some secondary ideas:
A) We’re implementing meta-data surrounding the timeseries data in OpenTSDB. Since Django supports multiple DBs, I think it would be pretty cool to add similar support to Graphite. The front-end could pass meta-data requests to the local DB for standard Graphite types or pass it through to OpenTSDB. Some docs: http://www.euphoriaaudio.com/opentsdb/http-api-meta.html
B) We’re also adding annotation support to track/mark events. Same thing, Graphite could store notes in the DB or get the info from OpenTSDB

So please let me know what ya’ll think and if you want to chime in you can see our roadmap at https://groups.google.com/forum/?fromgroups=#!topic/opentsdb/7MBVTMbvVZQ. Thank you!

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.