openstack team mailing list archive

[Loic Dachary <loic@xxxxxxxxxxxx>]

On 05/06/2012 10:36 PM, Tomasz Paszkowski wrote:
> Hi,
>
> I'd like to share my thoughts on metering openstack resources usage.
> Except those data available from SystemUsageData and those mentioned
> in blueprints some of the cloud providers charge for I/O on disk
> drives (to prevent users from dd if=/dev/zero nosense and to teach
> them properly implementing cache strategies). 
Hi Tomasz,

I could not agree more and this is the reason why I/O shows in the list of meters shown in http://wiki.openstack.org/EfficientMetering (c5) "disk IO in megabyte per second has a high impact on the service availability and could be billed separately ".


> Those data along with
> network card usage can be gathered from libvirt using domblkstat and
> domifstat. 
Thanks for the hint
http://wiki.openstack.org/EfficientMetering?action=diff&rev2=60&rev1=59
> My idea is to gather data using agent (or modified
> nova-compute)  and send them to messaging queue using following jsoned
> data schema:
>
> {'instance': 'instance-0000003b',
> 'host':'tytan-1','zone':'r4cz1','counters': {'interface': {'vnet0':
> (80796L, 1212L, 0L, 0L, 53403L, 621L, 0L, 0L)}, 'disk': {'vda': (629L,
> 11699200L, 58L, 219136L, 0L)}}}
>
> interface is a result of: interfaceStats(), disk is a result of: blockStats()
>
>
> Those messages are consumed from queue and stored in mysql tables. I
> assume that instance is a parent resource for each disk (ephemeral or
> volume) and for each network interface.
>
> So for message mentioned earlier we have three resources:
>
> 1) instances resource: instance-0000003b
> 2) child resource: vnet0 (network)
> 3) child resource: vda 'ephemeral
>
>
> Mysql table for resources is like below:
>
> CREATE TABLE resources (
>     id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
>     parent  BIGINT UNSIGNED,
>     type VARCHAR(255) NOT NULL,
>     value VARCHAR(255) NOT NULL,
>     zone VARCHAR(255) NOT NULL,
>     added TIMESTAMP,
> ) ENGINE=INNODB;
>
> Counters are stored also in Mysql using following table:
>
>
> CREATE TABLE counters (
>     resource BIGINT UNSIGNED NOT NULL,
>     type VARCHAR(255) NOT NULL,
>     value BIGINT UNSIGNED,
>     delta BIGINT UNISGNED,
>     added TIMESTAMP NOT NULL,
>     prev TIMESTAMP NOT NULL,
> ) ENGINE=INNODB;
>
> Where prev is a reference to previous counter value. Process which is
> reading data from queue is puting raw counter value into the table and
> if possible (reference to previous entry present) evaluates delta
> value.
>
>
> By using this model of stroing usage counter's it very easy for
> billing system to evaluate charges. We just run SUM(delta) on each
> counter for given time range.
>
> This model could be very easy adopted to other counters (IP Traffic
> external/internal counters from iptables).
>

It looks like you already have a codebase that could be useful for the metering implementation. Would you be willing to share it ?

Cheers
>
>
>
> On Mon, Apr 30, 2012 at 12:15 PM, Loic Dachary <loic@xxxxxxxxxxxx> wrote:
>> Hi,
>>
>> To prepare for the next meeting ( thursday 3rd, may 2012 http://wiki.openstack.org/Meetings/MeteringAgenda ) I cleaned up and reorganized the Metering blueprint so that it ( hopefully ) incorporates all the information temporarily stored in the etherpad ( http://etherpad.openstack.org/EfficientMetering revision 67 in case it is vandalized ).
>>
>> We could start a discussion from the content of the following sections:
>>
>> http://wiki.openstack.org/EfficientMetering#Counters
>> http://wiki.openstack.org/EfficientMetering#Storage
>>
>> and come up with a list of the counters that should exist by default and how they should be stored.
>>
>> This morning we had a discussion with Zhongyue Luo on irc.freenode.net#openstack-metering about how Dough could use the metering service. Since it already knows about instance creations, counter c1 that records how long a given instance was up is of no interest. However, other counters such as the external bandwidth used would be useful. I advocated that one of the advantages for Dough to rely on metering to collect counters is that it does not need to know about each OpenStack component and can rely on metering to figure out how to extract such counters from nova-compute, nova-network soon to be quantum, nova-volume soon to be cinder, swift, glance and free it from the burden of tracking structural changes.
>>
>> Cheers
>>
>> --
>> Loïc Dachary         Chief Research Officer
>> // eNovance labs   http://labs.enovance.com
>> // ✉ loic@xxxxxxxxxxxx  ☎ +33 1 49 70 99 82
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Loïc Dachary         Chief Research Officer
// eNovance labs   http://labs.enovance.com
// ✉ loic@xxxxxxxxxxxx  ☎ +33 1 49 70 99 82