openstack team mailing list archive

["John Purrier" <john@xxxxxxxxxxxxx>]

There are a lot of benefits to having an external system be responsible for
handling usage data from Nova, Swift, or other OpenStack services. I would
call out:

 

1.       Simplification of code within each service. The collection and
publication of usage data is "dumb". store the service usage data in a
service defined schema (could just be logfiles) until it is fetched/pushed
and then delete. No additional service API's required beyond that to
efficiently move the data out of the service.

 

2.       Simplification of hardware required for each service. If the
service is required to do analytics/processing on the usage data this will
drive more powerful hardware solutions.

 

3.       Single target for organizational usage data. This allows the best
selection of hardware and software for collecting and maintaining large
datasets. This also allows the analytic/billing/auditing/warehousing service
to scale independently of any particular service.

 

4.       Data warehousing for compliance and billing repeatability. It is a
likely scenario that billing and usage disputes will occur, sometimes months
after the fact. It is imperative that the billing and compliance reports be
able to be recreated and analyzed. Holding all of this data inside the
individual services for long retention periods doesn't make sense.

 

5.       Data warehousing for cold storage. The ability to tier storage for
long term retention, at the cheapest possible cost.

 

6.       Being able to do joins effectively across different service usages
to create consolidated billing.

 

7.       In addition to billing and auditing use cases, being able to make
the data available to scheduled and ad hoc reporting. This involved analytic
processing, data reduction, and other processing that is not appropriate on
the provisioning and real-time service controllers.

 

8.       Specialized data processing to allow real-time updates to system
behaviors. This is getting into the future, but creating a feed forward
analytic process may allow some really smart auto-scaling and dynamic
provisioning scenarios.

 

John

 

-----Original Message-----
From: openstack-bounces+john=openstack.org@xxxxxxxxxxxxxxxxxxx
[mailto:openstack-bounces+john=openstack.org@xxxxxxxxxxxxxxxxxxx] On Behalf
Of Monsyne Dragon
Sent: Monday, February 07, 2011 2:58 PM
To: openstack@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Openstack] Pondering multi-tenant needs in nova.

 

On 2/7/11 2:49 PM, Eric Day wrote:

> Thanks for explaining things further, Jay.

> 

> I agree if we want external systems poking into Nova for audit/billing

> queries, then yes, this gets inefficient. My assumption is that Nova

> specific DBs only contain operational data required for production and

> it would push billing/audit events to some external system that can

> collect, aggregate, and answer those queries efficiently. Trying to

> design a common data store that fits both use cases of provisioning

> instances/networks/volumes along with handling queries for

> billing/audit would be difficult (as we are seeing). Pushing

> billing/audit data to another system gives us the flexibility to

> choose the most suitable data store and querying abilities for each

> use case without making sacrifices for the other.

> 

 

Yes this is the model proposed with the system-usage blueprint. Nova 

publishes usage data, via a pub/sub interface, and a separate 

billing/audit system subscribes to those events, and builds it's 

datastore as it sees fit.