dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Calle

Here's another take on the problem.

Imagine you had a separate "metadata" instance (could variously be
called facility registry, data dictionary, indicator registry etc).
Basically a dhis2 instance with no data.  And a staging instance which
contains just a copy of the metadata instance.  Then you would always
apply your metadata imports to the staging instance.

Now what we don't have, but it seems like a very feasable app, is
something that can compare the contents of 2 instances.  Making very
clear to the user that the staging instance has more orgunits, less
orgunits or some changed orgunits for example.  And maybe some
rudimentary actions (import this, don't import that ..).

This approach might allow (i) easy user visualization of diffs, and
(ii) rollbacks, all without the need for maintaining dryRun code
within dhis2.  Keeping the dryRun logic out of dhis2 core means
keeping the metadata model relatively simple.  I can understand why
Morten would like to see it dropped. I suspect this could be managed
by implementing the differencing logic outside of the core logic as
well.  Basically doing the dryRun into a separate instance.  In fact,
for something like facility registry functionality, this might be the
only way to sensibly do it.

Just my 2c

Bob

On 2 February 2016 at 11:08, Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:
> Morten,
>
> I was talking about meta-data, and I'm not suggesting a roll-back feature -
> I'm suggesting a more step-wise import process where the user can see and
> control what is being imported/updated:
> - what meta-data appear as new (it might not actually be new meta-data, but
> rather just appear as new due to minor discrepancies in UIDs, names or
> codes)
> - what meta-data appear as updates (some might be wanted, some not).
>
> I'm frequently addressing these challenges by comparing external meta-data
> to DHIS2 meta-data using either PostgreSQL views or ODBC connections via MS
> Access, but that's really cumbersome. Still better than importing what turns
> out to be duplicates etc, and which later has to be cleaned up manually.
>
> A more step-wise process will also have major advantages over the current
> "import summary" in the sense that you can see exactly WHAT fields are
> updated (and presumably also filter out updates where e.g. the LastUpdated
> value is older).
>
> Regards
> Calle
>
> On 2 February 2016 at 10:29, Morten Olav Hansen <mortenoh@xxxxxxxxx> wrote:
>>
>> Hi Calle
>>
>> Yeah, I think its obvious from the quick emails we are getting that we
>> should not remove it. I would be great to have a proper rollback feature,
>> but right now this won't be that easy to add.
>>
>> (I am talking about metadata only though, you seem to also mention
>> aggregate/event data? those importers are not scheduled to be rewritten, and
>> will behave as today)
>>
>> --
>> Morten
>>
>> On Tue, Feb 2, 2016 at 3:26 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>> wrote:
>>>
>>> Morten,
>>>
>>> Dropping the dry run is OK if, and I stress if, the import process itself
>>> becomes more transparent and step-wise. Basically this means
>>> 1.
>>> Display new records coming in, with the ability to remove all or some
>>> from import
>>> 2.
>>> Display updated records coming in and the equivalent existing records,
>>> with the ability to remove all or some from import/updating.
>>>
>>> The fundamental problem with DHIS2 imports are that they are "black
>>> boxes", you cannot see what's happening until after it's done. Without even
>>> the dry run, your only "Safety net" is to back up the whole database just
>>> before importing - which is tricky to do in a production environment when
>>> manager A imports data while user B,C,D, and E are capturing data directly.
>>>
>>> Regards
>>> Calle
>>>
>>> On 2 February 2016 at 10:15, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>>> wrote:
>>>>
>>>> Hi everyone
>>>>
>>>> I'm currently in the process of re-writing our DXF2 metadata importer,
>>>> and was wondering if anyone is actually using the dryRun option? its adds
>>>> quite a bit of complexity to the code, so it will be added only to the new
>>>> importer if people are actually using it for imports
>>>>
>>>> --
>>>> Morten
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> *******************************************
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472
>>>
>>> Cell: +27-82-853-5352
>>>
>>> Iridium SatPhone: +8816-315-19119
>>>
>>> Email: calle.hedberg@xxxxxxxxx
>>>
>>> Skype: calle_hedberg
>>>
>>> *******************************************
>>>
>>>
>>
>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19119
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>