openstack team mailing list archive

[Doug Hellmann <doug.hellmann@xxxxxxxxxxxxx>]

On Thu, Nov 1, 2012 at 1:31 PM, Eoghan Glynn <eglynn@xxxxxxxxxx> wrote:

>
>
> > > if you have:
> > >
> > > Time | Value
> > > 0 | 10
> > > 1 | 30
> > > 2 | 50
> > > 3 | 80
> > > 4 | 100
> > >
> > > If your delta-pollster is down at 1 and 2, you restart at 3,
> > > therefore at 4 you'll send "20" as usage (100 minus 80).
> > > So you miss the delta between 10 (time 0) and 80 (time 3)
> > > (therefore 70 for free!).  If you send right away 80 at
> > > time 3 when restarting, the API will be able to guess that
> > > between 0 and 3 the value went from 10 to 80.  With delta
> > > approach, the API cannot guess that.
> >
> >
> > Sure it can, you just need to move where the caching is done. Using a
> > local cache to maintain the previous time a value was published you
> > would know at time 3 that the last published value was 10, and so
> > send 70. So the total will be correct.
>
> Good point, previously IIUC there was an implicit assumption that
> any prev time caching would be done in-memory, hence lost across
> process restarts.
>
> But as you point out, these data could be persisted locally by the
> compute agent.
>
> What would be the best way to achieve this? A small sqlite DB
> per-agent, or even simpler just a pickled dict? The latter would
> avoid the complexity of DB versioning and migration.
>

I discussed this issue at the summit with James Penick of Yahoo, and he
showed me some code in their agent that is using a sqllite db. We will want
to build a nice API so pollsters can use the cache without having to worry
about how it is implemented, which would let us deal with any versioning
issues in a central spot.

Doug