openstack team mailing list archive

[Luis Gervaso <luis@xxxxxxxxx>]

Forgot to copy the list

---------- Forwarded message ----------
From: Luis Gervaso <luis@xxxxxxxxx>
Date: Wed, May 9, 2012 at 7:14 PM
Subject: Re: [Openstack] ceilometer (java implementation)
To: Doug Hellmann <doug.hellmann@xxxxxxxxxxxxx>


I asked for these fields in my previous email, they are not clear enough.

After analyze the problem, i ended that metrics can not be calculated per
event. We need at least 2 events to calculate the duration (start/end) in
some metrics. We can afford this on the counter processes, but it requires
an update of the previous pesisted start event.

So as we can calculate this info everywhere, and i don't like updates on
the counters logic i'm calculating the duration as part of
agreggation/mediation process (this will be only a view of the raw data).

This way we maintain a very simple logic on the counters:

* counters listen for specific events
* the interpretation of raw data can be outside




On Wed, May 9, 2012 at 7:01 PM, Doug Hellmann
<doug.hellmann@xxxxxxxxxxxxx>wrote:

>
>
> On Wed, May 9, 2012 at 12:46 PM, Luis Gervaso <luis@xxxxxxxxx> wrote:
>
>> That's fantastic!
>>
>> What I mind is that the counters only should gather event info.
>>
>> Maybe multiple counters listening, collecting info about the system. But
>> only one persist the event.
>>
>
> That makes sense. It seems like we will need to split the event-based
> processing from the polling. The agent that runs on the compute node should
> poll libvirt (and any other services on that node) and generate counter
> messages using a "metering" exchange. A separate set of daemons running in
> a more central location will watch messages on the metering exchange and
> save them to the database. Those same processes can watch for notification
> messages and convert them directly to metering data to be written to the
> database. That saves the extra traffic from notification messages resulting
> in several extra metering messages.
>
>
>>
>> Other process agreggates the data and creates different
>> views/reports/billing, this should be outside of openstack (since it
>> depends on the business rules, not depends on the infrastructure)
>>
>
> If I understand Nick's proposal for the API, we will have some minimal
> aggregation there but you are right that any other interpretation will
> probably need to be handled by a custom app.
>
>
>>
>> The agreggator, also should offer an api for polling or can be configured
>> throught drivers to push the the 3rd
>> systems
>>
>> Now, in my implementation i'm putting all the stuff on a directory but it
>> can be as well:
>>
>> - AMQP
>> - SQL / NoSQL
>>
>> (the persistent layer shold be interchangeable)
>>
>> So as response to your duration question, this think is out scope for the
>> counter tasks.
>
>
> I think you misunderstood my question. The "duration" to which I referred
> was the field in the counter schema ("counter_duration"). Only the "exists"
> notification for instances includes enough information to calculate the
> duration. The "create" notification should result in a "0" duration, so
> that can be ignored. The "delete" notification should record the data since
> the end of the last cycle, but does not today (at least it does not seem
> to).
>
>
>>
>>
>>
>>
>> On Wed, May 9, 2012 at 5:30 PM, Doug Hellmann <
>> doug.hellmann@xxxxxxxxxxxxx> wrote:
>>
>>>
>>>
>>> On Tue, May 8, 2012 at 7:19 PM, Luis Gervaso <luis@xxxxxxxxx> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have uploaded a toy version of ceilometer (java implementation).
>>>>
>>>> It does implement the first two counters (instance : rabbitmq listener
>>>> and cpu : polling from libvirt)
>>>>
>>>> i need more clarification on the meaining:
>>>>
>>>> counter_volume
>>>> counter_duration
>>>> counter_datetaime
>>>>
>>>> I hope this helps to figure out how to agreggate these data.
>>>>
>>>> http://github.com/woorea/ceilometer-java
>>>
>>>
>>> Nice!
>>>
>>> I have also been experimenting. I have some Python code in
>>> https://github.com/dhellmann/metering-prototype that listens for
>>> notifications related to instances (create, delete, exists) and converts
>>> them to counter output (see spy.py). There is also a pair of scripts for
>>> recording an event stream and playing it back (useful for testing, see
>>> recorder.py and player.py). I don't have any libvirt polling, yet, though.
>>>
>>> In the course of looking at the notifications being generated for
>>> different scenarios, I discovered that the instance delete messages do not
>>> have any duration information right now. I don't know if that is a bug, or
>>> if the idea is to figure out the durations from looking at the most recent
>>> "exists" event. What do other people think?
>>>
>>> Doug
>>>
>>
>>
>>
>> --
>> -------------------------------------------
>> Luis Alberto Gervaso Martin
>> Woorea Solutions, S.L
>> CEO & CTO
>> mobile: (+34) 627983344
>> luis@ <luis.gervaso@xxxxxxxxx>woorea.es
>>
>>
>


-- 
-------------------------------------------
Luis Alberto Gervaso Martin
Woorea Solutions, S.L
CEO & CTO
mobile: (+34) 627983344
luis@ <luis.gervaso@xxxxxxxxx>woorea.es




-- 
-------------------------------------------
Luis Alberto Gervaso Martin
Woorea Solutions, S.L
CEO & CTO
mobile: (+34) 627983344
luis@ <luis.gervaso@xxxxxxxxx>woorea.es