openstack team mailing list archive

[Doug Hellmann <doug.hellmann@xxxxxxxxxxxxx>]

On Thu, May 17, 2012 at 5:47 AM, Nick Barcet <nick.barcet@xxxxxxxxxxxxx>wrote:

> On 05/17/2012 11:13 AM, Loic Dachary wrote:
> > On 05/16/2012 11:00 PM, Francis J. Lacoste wrote:
> >>
> >> I'm now of the opinion that we exclude storage and API from the
> >> metering project scope. Let's just focus on defining a metering
> >> message format, bus, and maybe a client-library to make it easy to
> >> write metering consumers.
>

The plan, as I understand it, is to ensure that all metering messages
appear on a common bus using a documented format. Deployers who do not want
the storage system and REST API will not need to use it, and can set up
their own clients to listen on that bus. I'm not sure how much of a client
library is needed, since the bus is AMQP and the messages are JSON, both of
which have standard libraries in most common languages.


> >>
> >> That way we avoid building a lot of things that we only _think will
> >> be useful_ for potential billing integration. Only
> >> writing/delivering such an integration component would prove that
> >> we built at least something that is useful.
> >>
> >>
> >
> > Hi,
> >
> > I'm a little reluctant to question the whole approach because I'm
> > engaged in it :-) It's scary to contemplate the idea of throwing away
> > some of the work done but I welcome the challenge. Better lose a few
> > days work than keep going in the wrong direction.
> >
> > Are you proposing that such a library would then be integrated in
> > whatever billing system someone has in mind ? For instance
> >
> > Dough https://github.com/lzyeval/dough trystack.org billing
> > https://github.com/trystack/dash_billing nova-billing
> > https://github.com/griddynamics/nova-billing
> >
> > Would depend on this library and rely on its API to collect what they
> > need. The same would be done by proprietary billing systems ?
> >
> > Regarding the relevance of the metrics exposed by the API, I made
> > sure they match the need of the eNovance sales. I'm quite sure
> > Nicolas checked for Canonical and Doug did the same regarding the
> > needs of Dreamhost. I'm confident that the information is actualy
> > useful, at least in these practical cases.
>

Yes, all of the feedback I have provided has been directly related to needs
we have here at DreamHost. I expect us to use the API service to pull data
into our legacy billing system, which does not have an API of its own.


> >
> > Getting rid of the storage imposes a constraint on the billing system
> > : it must make 100% sure that once a message is consumed it will be
> > reliably archived. It also adds a constraint on the chosen bus : it
> > must be able to retain all messages for as long as a consumer needs,
> > which may be days or weeks. Or it adds a constraint on the billing
> > system which must make 100% sure it will consume all relevant
> > messages the bus at all times before they expire.
>

That's exactly right. It will be easier for me to bridge between our two
systems by pulling a day's worth of details from the ceilometer API and
storing them in the billing system using a batch job, rather than trying to
ensure that the billing database performs well enough to record the
information in real time. My goal is to not have to change the billing
system at all.


> >
> > I have no strong feelings about exposing a bus for everyone to use
> > instead of a REST API. I tend to prefer the REST API because it is an
> > established standard for OpenStack. Could you expand on why a bus
> > would be significantly better than a REST API in this specific case
> > ?
>
> Thanks for challenging our thought process, Francis, it is a great
> sanity check :)
>
> I do have a few use cases where a REST API is better than a bus:
>
> * Private clouds:  Users are unlikely to want to activate a complete
> billing system but still want to be able top provide to their users some
> metrics of their uses.  The REST API plus some scripts would allow them
> to do this without too much pain.
>
> * Metric of usage on GUI: It might be usefull to provide a quick way to
> assess usage in almost real time to users as part of an extension of
> Horizon for example.  The REST API would allow for such data to be
> extracted dynamically without having to run a full billing tool in real
> time.
>

I don't know about real time, but I do know we eventually plan to provide
usage information like this to our users. So, change "might be useful" to
"will be useful." :-)


>
> * In house billing tools: about half of the ISPs I surveyed are running
> some form of a customized ERP system to handle the billing for their
> customers.  They need to be able to produce CSVs on a weekly basis to
> feed their custom billing solutions.  Integrating a bus would be much
> more complex than the script that would issue request to the rest API.
>

Right. A lot of accounting systems run in a batch model like this.


>
> * Auditability/Non repudiability: if the messages go from the queue to
> some unknown DB, how do you solve, in a generic way, coherent audit
> check and ensure non-repudiability?
>
> I do not mean to totally discard Francis' bus idea though, and think
> that we should allow for (a not necessarily db-less integration but)
> direct queue attachment model for billing systems. I do think that for
> all the above reasons, plus the simplification of testing of the overall
> system, the REST API must remain part of this project.
>

As I mentioned above, we will have the bus available. I expect to have
several routing keys, much as is done with notifications. For example, we
should have an easy way to receive all messages ("metering.all") as well as
a way to select only messages for counters the consumer is interested in
("metering.c1", "metering.n2", etc.). That should allow someone to replace
the ceilometer collector daemon with their own client and storage backend,
or to simply watch the messages for monitoring purposes.

Doug