openstack team mailing list archive

[Julien Danjou <julien.danjou@xxxxxxxxxxxx>]

On Mon, May 14 2012, Loic Dachary wrote:

>> Each set of metering data will need to be associated with the appropriate
>> metadata from the resource at the time the metering information was
>> collected. The rate of change of metadata and metering events are
>> different, though, so the timestamps of the metadata records are unlikely
>> to match exactly with the values in the metering records. Depending on the
>> clock resolution, it would be possible to have metadata changes and meter
>> data with the same timestamp, resulting in an incorrect association.
> Indeed, good point.
>>
>> We can work around that by maintaining proper foreign key references using
>> the metadata version field as you describe in the schema above (so the
>> resource id and metadata version value point to the correct metadata
>> record). It will make recording the metering data less efficient because
>> we will need to determine the current version for the resource metadata,
>> but we can optimize that eventually through indexes and caching.
>>
>> Aggregation will also need to take the metadata version into account, so everywhere in the list of queries we say "by resource_id" we need to change that to "by resource_id and version".
> I added the idea of a format version for when the payload format changes and tried to write down a description of the metadata storage matching this thread in the wiki.
>
> http://wiki.openstack.org/EfficientMetering?action=diff&rev2=80&rev1=78
>
> What do you think ?

I'm jumping in a bit late in the discussion, but there may be a point I
miss in the current definition because, I think it's getting too
complicated.

We now have 2 "payload" fields: one for meter and one for metadata.

For example, if you look at the c1 counter (instance) you need to store
the "type" as payload of the meter. This is a metadata of the instance,
but it's not currently defined as being stored in metadata, but in the
"payload" field of the meter.
Moreover, I'm rather sure there will soon be a counter with the need of
2 different "payload" information, and we'll have a problem since we can
only store one in the current meter schema, so we'll store the second
one as a metadata or something. So clearly the initial "payload"
solution is not enough.

OTOH I find the metadata proposal in another table too much
complicated. Why not storing what metadata in the meter.payload field
in the same table (e.g. as a JSON string)?

I miss the point of the introduction of a dedicated metadata table with
version string. It sounds to me like early optimization, which is the
root of all evil. :) But I might miss something.

-- 
Julien Danjou
// eNovance                      http://enovance.com
// ✉ julien.danjou@xxxxxxxxxxxx  ☎ +33 1 49 70 99 81