openstack team mailing list archive

[Nick Barcet <nick.barcet@xxxxxxxxxxxxx>]

On 05/04/2012 02:50 AM, Thierry Carrez wrote:
> Robert Collins wrote:
>> On Fri, May 4, 2012 at 5:27 AM, Turner, Whit (Cloud Services)
>> <Whit.Turner@xxxxxx> wrote:
>>> Hi - I think a flexible aggregation scheme is needed; the levels of
>>> aggregation available should be definable in the meter independent of the
>>> sources of usage data themselves. If invoices need to be very granular down
>>> to the lowest possible level, then this drives higher data requirements all
>>> through the processing chain, including the rating engine. Traditional
>>> systems tend to pass less granular (more highly aggregated) data into the
>>> rating engine so that bill runs and invoices can be generated efficiently.
>>> At cloud-scale, this can be problematic. Given some “big data” approaches,
>>> though, this could be handled in a more granular and real-time fashion.
>>
>> Has anyone looked at what statsd does? It has very similar
>> requirements (simple to use, no hard a-priori definition of things to
>> count, a few base types to track), and needs to be horizontally
>> scalable.
> 
> Also Swift has plans to use statsd for instrumentation/monitoring, so
> it's definitely worth a look to see if it could be used here as well.
> 
> http://folsomdesignsummit2012.sched.org/event/d9135eabdd775432c74c3f1d32a325d3
> http://etherpad.openstack.org/FolsomSwiftStatsd

I am no Stastd expert, but a quick look at the project shows that it is
aimed add data collection for the requirements of monitoring, and uses
UDP as a way to aggregate vast quantity of data at short interval.  The
use of UDP implies that delivery is not guaranteed, which is fines for
the objectives of monitoring, but is conflicting with the requirements
of metering (as a sub component of a billing system).

Stastd does not seem either to allow for message signature and
authentication of collectors.

Here are the requirements I think we have:
 * The data is sent from agents to the storage daemon via a trusted
messaging system
 * The messages in queue are signed and non repudiable
 * The agents collecting data are authenticated to avoid pollution of
the metering service

Nick

Attachment: signature.asc
Description: OpenPGP digital signature