dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

On 9 December 2010 18:25, Lars Helge Øverland <larshelge@xxxxxxxxx> wrote:
> Just a quick battlefield story from what we experience in Sierra Leone where
> we are facing insufficient internet connectivity and must resort to offline
> deployments. The districts are quite rural and we experience an "offset"
> when distributing new databases (metadatasets), which means that for each
> update there will be districts that still exports data to the national
> server using the previous metadataset for some time until the new set has
> percolated completely.
> Like Bob says we deal with different types of changes, the ones I can think
> of are
> 1) Adding of elements (dataelements, orgunits) (in the national database)
> 2) Removal of elements (in the national database)
> 3) Updates of element names/properties (in the national database)
> The point of departure is that the current solution where an export message
> contains both data and metadata and we match on the display name of elements
> is not appropriate, and that we will move to using dedicated, agreed and
> stable metadata identifiers of some sort. As Jo says we are not planning for
> versioning in 2.0.7.
> For the current solution the implication of the mentioned changes
> when importing data from the out-of-date districts into the national
> database for the change types respectively are as follows:
> 1) No data will come in. No problem.
> 2) Old metadata elements will continue to come in. Minor problem.
> 3) Metadata elements and its data which represents the same element will
> come in but under a different name. Big problem and source for complete
> chaos.
> With the new proposed solution with stable metadata identifiers the
> situations is like this when receiving data from the out-of-date districts:
> 1) No data will come in. No problem. As before.
> 2) Data will be ignored as the metadata identifier will not match anything
> in the national database. No problem.
> 3) Data will match on the stable identifier even if the name/properties are
> changed. No problem.
> So in the offline deployment of the "regular" DHIS scenario this move will
> be a huge improvement.

Very much agree.  But if we are to have stable identifiers then they
will have to be distributed somehow as a metadata set.  At the very
least that might have a timestamp, eg
<dxf><metadata published="2011-01-01" publisher="SL MOHS"><de's,
mcdonalds, orgunits etc ..></metadata></dxf>
And Jo this could be a fine grained transaction (even jason encoded!)
rather than a full metadata set.  eg metadata for a particular dataset
or orgunit.  Either way its not harmful to have  version attribute or
publication date nor does it necessarily have to be too complicated.

On reading data into say national db, it might look like
<dxf><data metadataversion="2010-11-01" metadatapublisher="SL MOHS"><.
..datavalues .../></data></dxf>

Note that the metadata is outdated.  The simplest case on the national
side is that it simply ignores the metadata attributes.  That would
essentially be the improved situation that Lars describes above so
nothing is lost.  A slightly more complex response would be to trigger
a metadata preview which shows a reasonably useful diff of the the two
versions which might indicate if anything "bad" can happen or whether
any data will be lost.  Following which the user can decide how to
continue: to import or not or perform some kind of translation.  These
different kinds of responses can be controlled with a system setting.
At the very least we will have useful information that the sending
system still needs to update its metadata.

Implementing could be fairly simple.  I'm thinking, for example, that
every time national MOH "publishes" (ie agrees on a set of metadata to
be sent out to districts, provinces and all) it retains a copy of that
published metadata (export-2010-01-01.xml).  That way the system has
some memory of what is out there in the darkness.  That's the most
basic level of versioning I can foresee.  Though it could start to
become wild if metadata is published every other day.  This I imagine
is most typical of startup scenarios and needs to be catered for.

Got to go.  Catch up on this later.

Cheers
Bob

>It might not be perfect and there might be situations
> I have not thought about but it will help to alleviate the bigger problems.
> I am not sure how relevant this is in Jo's mobile scenario.
> Lars
> PS. That said I am not sure how appropriate the versioning paradigm is in
> this scenario. What if you want to change the name or add an element in the
> national database, should we then deny all out-of-date districts to report
> any data at all? Can we live well with what is described above?
>
>
>