openstack team mailing list archive

[Loic Dachary <loic@xxxxxxxxxxxx>]

On 05/11/2012 05:55 PM, Doug Hellmann wrote:
> During yesterday's meeting we discussed the API proposal at
> http://wiki.openstack.org/EfficientMetering#API and came up with a few
> missing items and other points for discussion. We should try to work out
> those details on the list before the meeting next week.
>
> The original proposal included these API calls:
>
> GET list components
> GET list components meters (argument : name of the component)
> GET list [user_id|project_id|source]
> GET list of meter_type
> GET list of events per [user_id|project_id|source] ( allow to specify user_id or project_id or both )
> GET sum of (meter_volume, meter_duration) for meter_type and [user_id|project_id|source]
>
> They can be broken down into three groups:
>
> - Discover the sorts of things the server can provide:
>
> - list the components providing metering data
> - list the meters for a given component
> - list the known users
> - list the known projects
> - list the known sources
> - list the types of meters known
>
> - Fetch raw event data, without aggregation:
>
> - per user
> - per project
> - per source
> - per user and project
>
> - Produce aggregate views of the data:
>
> - sum "volume" field for meter type over a time period
> - per user
> - per project
> - per source
> - sum "duration" field for meter type over a period of time
> - per user
> - per project
> - per source
I updated the wiki with these three groups, it is a good starting point for discussion.
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=75&rev1=74
>
> We agreed that all of the queries related to metering data would need
> to have parameters to set the start and end times with the start time
> inclusive and the end time exclusive (i.e., start <= timestamp < end).
>
> Callers also are likely to want to filter the raw data queries based
> on the meter type, so we need to add that as an optional argument
> there. The aggregate values have the meter type as a required argument
> (because it does not make sense to aggregate data from separate meters
> together).
>
> There are a few other queries missing from that list. The items below
> are based on the meeting notes and my own thoughts from yesterday,
> please add to the list if I have left anything out (or suggest why we
> should not implement something suggested here, of course):
>
> - list discrete events that may not have a duration (instance
> creation, IP allocation, etc.)
> - list raw event data for a resource ("what happened with a specific
> instance?")
> - aggregate event data per meter type for a resource over a period of
> time ("what costs are related to this instance?")
> - sum volume for meter type over a time period for a specific resource
> ("how much total bandwidth was used by a VIF?")
> - sum duration for meter type over a time period for a specific
> resource ("how long did an instance run?")
> - metadata for resources (such as location of instances)
> - aggregating averages in addition to sums
>
I've added this list as a note in the API chapter.
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=76&rev1=75
> Some of these items may be better handled in the consumer of this API
> (the system that actually bills the user). I am curious to know how
> others are handling auditing or other billing detail reporting for
> users.
I will be in a meeting with the participants of the http://www.opencloudware.org/ project early next week and I'll ask them. Maybe Daniel Dyer could share a practical experience with us.

> In order to support the discrete events, we need to capture
> them. Perhaps those could be implemented as additional counter
> types. Thoughts?
Are there resources that should be billed (henced metered) if their lifespan is shorter than the polling / publishing event period ? I mean that if someone creates a VM and immediately destroys it, does it really matter. While writing this statement, I though of a way to abuse a system that does not account for discrete events like "created a VM" but only account for "this VM has run for at least X minutes" :-) One could write a software to launch short lived VM at specific times and get computing power for free.

Do discrete event need to be separate ? Back to the example above, someone creates a VM : that does not create a record in the ceilometer storage. The VM is destroyed a few seconds later : it creates an event, as if it was polled / published to measure the uptime of the VM. The c1 meter would show the VM has been up for 15 seconds since it was created. If the c1 meter is generated from an exist event captured from http://wiki.openstack.org/SystemUsageData that occurs every 5 minutes, the uptime will be 5 minutes more than the previous one.

While thinking about this use case I also had a simple idea to optimize the storage of events and added it to the wiki
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=77&rev1=76

Cheers

-- 
Loïc Dachary         Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ loic@xxxxxxxxxxxx  ☎ +33 1 49 70 99 82