openstack team mailing list archive

[Aimon Bustardo <abustardo@xxxxxxxx>]

John, Don't forget two huge additional benefits to it being external:
Scalability and distribution. The service handling this can be clustered
and distributed if it is external.


Aimon


On 2/7/11 3:12 PM, John Purrier wrote:
>
> There are a lot of benefits to having an external system be
> responsible for handling usage data from Nova, Swift, or other
> OpenStack services. I would call out:
>
>  
>
> 1.       Simplification of code within each service. The collection
> and publication of usage data is "dumb"... store the service usage
> data in a service defined schema (could just be logfiles) until it is
> fetched/pushed and then delete. No additional service API's required
> beyond that to efficiently move the data out of the service.
>
>  
>
> 2.       Simplification of hardware required for each service. If the
> service is required to do analytics/processing on the usage data this
> will drive more powerful hardware solutions.
>
>  
>
> 3.       Single target for organizational usage data. This allows the
> best selection of hardware and software for collecting and maintaining
> large datasets. This also allows the
> analytic/billing/auditing/warehousing service to scale independently
> of any particular service.
>
>  
>
> 4.       Data warehousing for compliance and billing repeatability. It
> is a likely scenario that billing and usage disputes will occur,
> sometimes months after the fact. It is imperative that the billing and
> compliance reports be able to be recreated and analyzed. Holding all
> of this data inside the individual services for long retention periods
> doesn't make sense.
>
>  
>
> 5.       Data warehousing for cold storage. The ability to tier
> storage for long term retention, at the cheapest possible cost.
>
>  
>
> 6.       Being able to do joins effectively across different service
> usages to create consolidated billing.
>
>  
>
> 7.       In addition to billing and auditing use cases, being able to
> make the data available to scheduled and ad hoc reporting. This
> involved analytic processing, data reduction, and other processing
> that is not appropriate on the provisioning and real-time service
> controllers.
>
>  
>
> 8.       Specialized data processing to allow real-time updates to
> system behaviors. This is getting into the future, but creating a feed
> forward analytic process may allow some really smart auto-scaling and
> dynamic provisioning scenarios.
>
>  
>
> John
>
>  
>
> -----Original Message-----
> From: openstack-bounces+john=openstack.org@xxxxxxxxxxxxxxxxxxx
> [mailto:openstack-bounces+john=openstack.org@xxxxxxxxxxxxxxxxxxx] On
> Behalf Of Monsyne Dragon
> Sent: Monday, February 07, 2011 2:58 PM
> To: openstack@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Openstack] Pondering multi-tenant needs in nova.
>
>  
>
> On 2/7/11 2:49 PM, Eric Day wrote:
>
> > Thanks for explaining things further, Jay.
>
> > 
>
> > I agree if we want external systems poking into Nova for audit/billing
>
> > queries, then yes, this gets inefficient. My assumption is that Nova
>
> > specific DBs only contain operational data required for production and
>
> > it would push billing/audit events to some external system that can
>
> > collect, aggregate, and answer those queries efficiently. Trying to
>
> > design a common data store that fits both use cases of provisioning
>
> > instances/networks/volumes along with handling queries for
>
> > billing/audit would be difficult (as we are seeing). Pushing
>
> > billing/audit data to another system gives us the flexibility to
>
> > choose the most suitable data store and querying abilities for each
>
> > use case without making sacrifices for the other.
>
> > 
>
>  
>
> Yes this is the model proposed with the system-usage blueprint. Nova
>
> publishes usage data, via a pub/sub interface, and a separate
>
> billing/audit system subscribes to those events, and builds it's
>
> datastore as it sees fit.
>
>  
>
>  
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

-- 
Aimon Bustardo | Morph Labs | +1.310.625.0608 (mobile) | +1.310.437.4898 (office) | www.morphlabs.com | aimon@xxxxxx