openstack team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services)
<Whit.Turner@xxxxxx> wrote:
> Hi - I think a flexible aggregation scheme is needed; the levels of
> aggregation available should be definable in the meter independent of the
> sources of usage data themselves. If invoices need to be very granular down
> to the lowest possible level, then this drives higher data requirements all
> through the processing chain, including the rating engine. Traditional
> systems tend to pass less granular (more highly aggregated) data into the
> rating engine so that bill runs and invoices can be generated efficiently.
> At cloud-scale, this can be problematic. Given some “big data” approaches,
> though, this could be handled in a more granular and real-time fashion.

Has anyone looked at what statsd does? It has very similar
requirements (simple to use, no hard a-priori definition of things to
count, a few base types to track), and needs to be horizontally
scalable.

We could, as a riff on my prior email, define the statsd (or a similar
thing) as a common substrate, and then let different implementations
discard detail, or preserve it as needed. The key difference I see vs
defining a Python API is that if someone is writing a different
language implementation of an Openstack component, they would have a
common thing to target.

OTOH it should be trivial to write a network component that thunks
*into* the stock Python API, and from there to the configured backend,
so there is no need to pick any specific network protocol up front -
though bearing in mind that we want network handoffs is probably a
good thing when looking at the nitty gritty.

-Rob