graphite-dev team mailing list archive

[Chris Larsen <question223956@xxxxxxxxxxxxxxxxxxxxx>]

Question #223956 on Graphite changed:
https://answers.launchpad.net/graphite/+question/223956

Chris Larsen posted a new comment:
Wow, thanks a ton for all of your comments Dieter:

> re: "Nodes/Leaves should ID a timeseries with a generic ID field and provide a display_name field for GUI use"
> 1) In graphite only leaves in the tree point to timeseries. (i.e. if you have a metric name "foo.bar.baz" then "foo.bar" is nothing). I 
> think this is sensible.  do you want to change this? (if so what would "foo.bar" be?)

No change, a leaf is the only thing that could point to a timeseries,
but each node (or branch) can have metadata (like # of child
branches/leaves) or a different name for other types of storage systems.

> 2) what are reasons why the display name would be different from the metric name? (assuming the metric name would still be all 
> nodes in the tree "as.we.are.used.to" and which would translate into the TSUID) i don't see a need for this, especially in graphite 
> where you interact so closely with the metric names (when building graphs and dashboards) that hiding their names seems to be 
> disadvantageous (probably read the tagging section below first)

For us data geeks, "as.we.are.used.to" is perfectly fine, but if you
want to share the information with less technical users, having an
optional name override is very useful. Or if you abbreviate a metric,
e.g. "if.tx.eth0", it would be nice to display on the graph "Interface
Transmits: Ethernet 0". I'm thinking of the broader audience. For
situations where no display name is provided, everything just defaults
to the node name.

> === tagging ===
> I was actually going to start a separate discussion, but since you bring it up here...
> First, you should know about:
> * https://github.com/Dieterbe/graph-explorer/tree/master/structured_metrics : a library that converts the graphite metric list into 
> a tag space of metrics
> * https://github.com/Dieterbe/graph-explorer: a graphite dashboard that takes this tag space and provides a query language so you 
> can filter metrics and group them into graphs by tag(s) I will refer to these as 'G-E'

That’s some neat work and really helps with the naming restrictions
inherit in Graphite.

> to take the last example from http://www.euphoriaaudio.com/opentsdb/http-api-meta.html
> that metric can be written as:
> {
>         "name": "tsd.http.latency_50pct",
>         "display_name": "HTTP Latency 50pct",
>         "tags": {'host': 'hobbes-64bit', 'type': 'all'} }
> 1) as you can see, I brought down the markup for tags substantially. I find the syntax demonstrated in your opentsdb RFC quite 
> overengineered, which is also evident because so many fields are just empty.

I'm sorry if this bit was a little confusing. An actual timeseries data
point can be written to OpenTSDB (with the upcoming JSON format) via:

{ 
   "metric":"tsd.http.latency_50pct",
   "timestamp":1341144000,
  "value":42,
   "tags":{"host":"hobbes-64bit","type":"all"}
}

So the only data needed to submit a data point is the metric name and
one or more sets of tags. The metadata is user supplied information
associated with a single timeseries, metric, tag name or a tag value.
When a brand new timeseries data point is written, a metadata entry is
generated with default values, which is why the fields are blank. Then
users can go in and add their description, a display name override,
notes, or custom key/values. It's very useful for documenting things
like: Where the data came from, who's in charge of the data source, what
the data means, etc. When you have millions of unique timeseries and
many users working with the system, this kind of data can be invaluable
and I think Graphite users could benefit from having it as well. But no
one is forced to use it.

> 2) one thing that I learned with G-E is that the more information you can capture in tags, the better (because it's structured data, 
> clearly defined, so more usable.   "name" has no clear meaning for metrics and so can only be used for text filtering ). Luckily, there's 
> no need for a "name" attribute, if you add additional tags such as protocol=http, what=seconds, type=latency_50pct.  this gives more 
> power for filtering, aggregating and grouping metrics when composing graphs.  As a rule of thumb, I would say never have a 'name' 
> attribute, always aim to structure data in more specific tags.
> (the canonical opentsdb example metric "mysql.bytes_sent schema=foo host=db2" becomes "service=mysql what=bytes type=sent schema=foo foo=db2")

The "name" field is only used in the general UID meta object for
OpenTSDB and reflects either a METRIC, a TAGK or a TAGV name so it's
valid in this situation but you're absolutely right that it's a terrible
tag name, e.g. "name=something".

OpenTSDB's schema is more efficient when you have fewer tags, hence the
combination of a Graphite style dotted syntax for the metric and tags
for differentiation and aggregation. Metadata can also be used to break
these up into tags like you're suggesting and make it easier to filter
and search (I'm pushing everything into ElasticSearch and it works
great)

> Furthermore:
> 1) I would argue that the display of metric names should not be configured at the metric level as in your examples, but can easily be 
> generated.
> * in a composer interface you can just list all the tags in a predefined order: In G-E it's just "%what %type %target_type <other tags 
> alphabetically sorted> %server %plugin")
> * this becomes more apparent when viewing a graph: say you are plotting these two metrics on one graph:
> {'what': 'bytes', 'service': 'mysql', 'server':'host1', 'type': 'sent'}
> {'what': 'bytes', 'service': 'mysql', 'server':'host1', 'type': 'received'}
> for this graph, the tags 'what', 'service' and 'server' are constant, and the 'type' tag is variable.  So the graph title can be computed to 
> be "host1 mysql bytes" and the entries on the legend would be 'sent' and 'received'.  This is what G-E does, it's trivial to implement 
> and creates a non-reduntant display of information independent of how you group metrics into graphs.

That's a neat idea and I think a combo of the two would work really
well.

> 2) nowadays, many metrics in graphite have names that are just to unclear. often they don't specify the unit of measurement 
> (seconds? ms?, bits? bytes? elements in a queue? an amount of errors?), prefixes used (M,G, etc) and how it should be interpreted 
> (is this a number per second? per flushinterval (like statsd counts), etc).  G-E solves this by making the 'what' and 'target_type' tags 
> mandatory and clearly defined.

We don't want to mandate any tags in OpenTSDB as per the tag size issue,
so that is another example of what the metadata is meant to solve,
particularly with the "units" field. It's something that could be bolted
on to Graphite directly so that folks could upgrade and fill out the
meta without having to rename their existing data files.

> 3) the current tree based organisational paradigm is a bit too simple.  There's basically no way to organise your tree to support all 
> ways of later querying it (so that you can later do "give me all metrics related to service mysql, or all metrics that are an amount of 
> errors", which causes people spending too much time trying to.
> (see also the amounts of statsd issues/PR's related to suffixes, prefixes and namespacing). a tag based system makes this moot.

You're absolutely correct. That was a pain when I was trying to get
OpenTSDB info into Graphite for the default composer view. I wound up
writing a tree creation rule system that lets users define a number of
trees with information they want to see. E.g. Netops could build a tree
with only network related metrics, Sysops could write one for their
system metrics, etc. But we also have full-text searching so that would
need to be integrated with Graphite as well.

=== events ===
> quote: "B) We’re also adding annotation support to track/mark events. Same thing, Graphite could store notes in the DB or get the 
> info from OpenTSDB".
> I think there's no benefit of deep integration between an event/change management system with a metrics database/management 
> system, because events/changes are inherently very different things than metrics.   They have different requirements wrt ingestion, 
> storage, management, GUI's, etc.  (I believe deep integration leads to feature creep, scope dilution, and harder integration with 
> other software i.e. monolithic software)

Right, we don't intend to use Graphite or OpenTSDB for tracking all
events or being used for change management. But we do want the ability
to let users place a simple note associated with a timeseries in the
storage system. The note could be something as simple as "I started up
process X on this box, it pegged the CPU until I stopped it, watch out".
Or folks could write scripts to comb their change/event systems and put
notes in the appropriate spots, e.g. "<a href='link'>Change # Upgrade
Web Servers</a>". The goal is to have these pop-up when a graph is
rendered, then a user can hover or click to get to the details.

> They do go alongside on graphs, which is AFAICT the only place where metrics and events meet.  That's why I think it's sensible to 
> have a separate change/event management system, and have a timeseries graphing widget where they can be rendered together 
> (as directed by dashboard software)
> From this conviction, I've written:
> * https://github.com/Dieterbe/anthracite change/event management system (inspired by graphite's philosophy)
> * https://github.com/Dieterbe/timeserieswidget to render graphs and events/changes in a "rich" way (with annotation text etc), as 
> you would expect it supports and targets graphite and anthracite.

That's really cool and I like your timeseries widge display, that's
exactly what we're going for. For us to achieve maximum performance, it
made sense to stash timeseries related annotations inline with the data
points so that queries don't have to make another call each time data is
fetched.

> Btw, are you going to monitorama? I will, as will a bunch of other
graphite devs.

Shoot, I would like to but I was just called out to my firm's
headquarters so I won't be around on those dates. I'm Connecticut based
so if ya'll have any other meetups I'd really like to pow-wow with
Graphite devs. Thanks!

-- 
You received this question notification because you are a member of
graphite-dev, which is an answer contact for Graphite.