dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

On 9 December 2010 11:40, Jo Størset <storset@xxxxxxxxx> wrote:
>
> Den 7. des. 2010 kl. 18.21 skrev Bob Jolliffe:
>
>> To me versioning is not different to stable.  ie if a metadata item
>> has an id of 42, an owner of "Kenya National MOH" and a version of
>> 2.4, then I (ideally) expect that id to be constant in perpetuity for
>> that version.
>
> I agree (of course), but I'm not sure I understand the implications.
>
> As far as I can tell versioning wasn't in your discussion on identifiers,
Wasn't a complete discussion so much as a response

>do you think you want it there?

It has to be there in one form or another.  The fact is that a client
will make use of a set of identifiers.  And they will change from time
to time.  I don't want to argue that they can never change.  Of course
they will.  But if and when they do, we will be talking about
different versions.  The question is whether there is sufficient use
case to acknowledge that explicitly - to take the bull by the horns as
it were - or whether it is sufficient to manage the process of getting
everyting back in synch in a manual, flexible ad-hoc manner.

I think the problem with this latter approach is that on the surface
it appears least complex.

But beneath the surface users will be (and are) managing versions
anyway without any system support to do so.  Something which is easier
or harder depending on (i) how closed the system is and (ii) how big
the system is and (iii) how far forward in time metadata governance
proceeds.  And (hopefully) increasingly on the amount of 3rd party
collaboration/interoperability going on.  I suspect that we find it is
convenient to go into new environments and set up pilot projects and
the like if we can temporarily put aside complex concerns like
metadata versioning.  A sort of suspension of disbelief which allows
things to get done.  But in time these systems must get harder to
maintain.  Probably some battle stories from the field would shed some
light on this - particularly on dhis deployments which have survived
over some time (India might be interesting here) as well as
deployments which have fallen apart or collapsed into disuse.

I suspect the sweet spot is to have a rigorously correct system
capacity to maintain complex metadatasets over time (with all the
implications of potentially complex metadata governance user roles) as
well as the ability to ignore or override them, particularly in
situations where there is rapid and frequent change like you wouild
likely find in the early life of projects.

On sdmx side we sidestepped the issues slightly by using an external
dxf metadata dump as a canonical reference to be shared with 3rd party
clients (transformed to sdmx of course, but that is incidental).  This
temporarily solves the issue of the lack of stability of integer ids
in the database but doesn't resolve the version problem.  A solution I
have toyed with is to stamp a version number on the entire dxf
metadata dump.  That gives at least coarse grained versioning.  But is
also draconian in its way - for example you cannot change a single
orgunit or dataelement or category in the entire set without
triggering an overall version update.

One way to think about it might be to consider different kinds of
change differently.  eg. some additive changes (new dataelement),
changes in textual identifiers (spelling correction of facility name)
etc need not completely invalidate existing metadata that a client
might have.  The kinds of change which are most cataclysmic would be
the ones where for example existing facilities acquire new integer
ids.  And this is in fact the most common kind of change that one
might see given the way we assign those ids currently - effectively
arbitrarily using whatever particular db sequence generators give us.
It might well be that as we move to stabilize those integer
identifiers that we actually end up reducing these kinds of breaking
changes significantly and simplify the versioning requirements in the
process.  Not sure .. i'm really trying not to think too much about
this until January.

I think what you are doing (and not to dissimilar to what I have done)
is to temporarily suspend disbelief and work with the integer
identifiers in the hope (and reasonable expectation in some
circumstances) that they won't get shuffled on you :-)  At least not
without warning.  I think that will work for a short while and will
drive the effort to stabilize those in 2.0.7.  And I think we will
need to build in some concept of versioning to this, but hopefully not
as draconian or difficult to work with as you are fearing.

Regards
Bob

>If versioning is going to be part of the id regime, like in the above example, it would also have to be part of the discussion, somehow.. If it's not going to be there, I think it would be good to at least have some discussions on what to do for the most immediate use cases that would need some way to coordinate metadata changes. A third option would of course be that it is too ambitious for us to try to coordinate on versioning issues for the time being, but then I think it would be good if we could managed to at least explicitly agree on that.
>
> The main reason for my interest is that on the mobile side, we're going to have to come up with some solution for versioning. Just as I guess you already have had to make some sort of versioning for import-export use cases. And, as Olas original mail pointed to, there is a potential use case for better support for syncing dhis instances.
>
> Should we just think of versioning as a specific use case issue for the time being, or would we be able to take some small steps to at least avoid unnecessary fragmentation? I guess both fragmentation and coordination have their built-in challenges, I gather that the silence on the topic maybe should be taken as a signal that we should go with the first option for the time being? :)
>
> Jo