dhis2-users team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Olav,
I have not worked with the DHS API per se, but have imported lots of data
using the same approach which they outline here (
http://api.dhsprogram.com/#/samples-r.cfm)

I have written up a walkthrough of getting data out of one DHIS instance
and into another one, and I think the basic principles would be the same (
http://rpubs.com/jason_p_pickering/139589)

Metadata needs to be mapped (or created), the data needs to be reshaped,
and correctly formatted.

It should not be too difficult. I used R, but there are other examples with
Python and JavaScript on their examples page.

Regards,
Jason


On Tue, Feb 2, 2016 at 3:31 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
wrote:

> Dear Uwe,
>
> Have you tried to send data via the endpoint api/dataValueSets, it may be
> faster. Just stage your data and push it once.
>
> http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s13.html#d5e1372
>
> Also to note, is how you send it, I have seen curl taking ages to submit
> individual values via the api. You need to send it as once file via once
> request or implement concurrency.
>
> Alex
>
> On Tue, Feb 2, 2016 at 5:13 PM, Olav Poppe <olav.poppe@xxxxxx> wrote:
>
>> Hi Randy and Uwe,
>> thanks, interesting to hear you experiences. Uwe, what you are working on
>> sounds quite a bit more complicated, and not least with far more data. I
>> image that with household surveys, it would be a matter of < 100 indicators
>> for < 200 orgunits for 2-3 periods, i.e. a fraction of what you are dealing
>> with!
>>
>> Olav
>>
>>
>>
>>
>>
>>
>> 31. jan. 2016 kl. 09.29 skrev uwe wahser <uwe@xxxxxxxxx>:
>>
>> Hi Olav & Randy,
>>
>> I am currently banging on kettle (aka Pentaho DI) to extract data from a
>> source-system (SQL-ERP in our case) into DHIS2 dataSets in json format. In
>> our current test-scenario (2 dataElements in a dataSet with a
>> categoryCombination of 5 categories) we are currently updating ca. 4 mio
>> dataValues every night in a pseudo-delta mode (reading all data from
>> source, comparing to what is there in DHIS2 already, then only pushing
>> records for creating, updating or deleting dataValues into the api: ca.
>> 150k per night in 1 hour, initial load was 7hrs). We still have to prove,
>> that this is feasible when setting up the first real life dataSet where
>> there will be more categories and more dataElements, thus exploding the
>> number of dataValues.
>>
>> Getting there was a bit painful, but now it seems to work. I chose kettle
>> instead of Talend ETL (both open source) as it seemed to be easier to get
>> used to. However, from a data warehouse perspective I'd prefer to have
>> DHIS2 offering some sort of an integrated ETL landscape on the long run,
>> which would also allow to aggregate data from tracker into dataSets,
>> tracker to tracker, dataSets to dataSets etc.
>>
>> Our current version of the kettle transformations and jobs were designed
>> to be generic (not for a specific dataSet, but you have to design your own
>> extractor which could be a simple csv-reader or maybe a DHS api-call). If
>> you are interested, I will share them. Just be aware that they are
>> currently in a very early and rough state and not documented. You'd have to
>> bring along the willingness to dig yourself into kettle and be pain
>> resistant to a certain degree :-)
>>
>> I'd be interested to hear from other experiences ...
>>
>> Have a nice sunday,
>>
>> Uwe
>>
>> ---
>>
>> Am 29.01.2016 um 17:31 schrieb Wilson, Randy:
>>
>> Not here unfortunately...just doing csv imports from DHS Excel files.
>> Would be useful for our data warehouse.
>> Randy
>> On Jan 29, 2016 2:59 PM, "Olav Poppe" <olav.poppe@xxxxxx> wrote:
>>
>>> Hi all,
>>> I wanted to hear if anyone has any experience with the DHS API (
>>> http://api.dhsprogram.com/#/index.html), and using it to import survey
>>> results into DHIS?
>>>
>>> Olav
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-users
>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>> *This message and its attachments are confidential and solely for the
>> intended recipients. If received in error, please delete them and notify
>> the sender via reply e-mail immediately.*
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Alex Tumwesigye
>
> Technical Advisor - DHIS2 (Consultant),
> Ministry of Health/AFENET
> Kampala
> Uganda
>
> IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya
>
> IT Specialist (Servers, Networks and Security, Health Information Systems
> - DHIS2 ) & Solar Consultant
>
> +256 774149 775, + 256 759 800161
>
> "I don't want to be anything other than what I have been - one tree hill "
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+46764147049