dhis2-users team mailing list archive

[Alex Tumwesigye <atumwesigye@xxxxxxxxx>]

Dear Uwe,

Have you tried to send data via the endpoint api/dataValueSets, it may be
faster. Just stage your data and push it once.
http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s13.html#d5e1372

Also to note, is how you send it, I have seen curl taking ages to submit
individual values via the api. You need to send it as once file via once
request or implement concurrency.

Alex

On Tue, Feb 2, 2016 at 5:13 PM, Olav Poppe <olav.poppe@xxxxxx> wrote:

> Hi Randy and Uwe,
> thanks, interesting to hear you experiences. Uwe, what you are working on
> sounds quite a bit more complicated, and not least with far more data. I
> image that with household surveys, it would be a matter of < 100 indicators
> for < 200 orgunits for 2-3 periods, i.e. a fraction of what you are dealing
> with!
>
> Olav
>
>
>
>
>
>
> 31. jan. 2016 kl. 09.29 skrev uwe wahser <uwe@xxxxxxxxx>:
>
> Hi Olav & Randy,
>
> I am currently banging on kettle (aka Pentaho DI) to extract data from a
> source-system (SQL-ERP in our case) into DHIS2 dataSets in json format. In
> our current test-scenario (2 dataElements in a dataSet with a
> categoryCombination of 5 categories) we are currently updating ca. 4 mio
> dataValues every night in a pseudo-delta mode (reading all data from
> source, comparing to what is there in DHIS2 already, then only pushing
> records for creating, updating or deleting dataValues into the api: ca.
> 150k per night in 1 hour, initial load was 7hrs). We still have to prove,
> that this is feasible when setting up the first real life dataSet where
> there will be more categories and more dataElements, thus exploding the
> number of dataValues.
>
> Getting there was a bit painful, but now it seems to work. I chose kettle
> instead of Talend ETL (both open source) as it seemed to be easier to get
> used to. However, from a data warehouse perspective I'd prefer to have
> DHIS2 offering some sort of an integrated ETL landscape on the long run,
> which would also allow to aggregate data from tracker into dataSets,
> tracker to tracker, dataSets to dataSets etc.
>
> Our current version of the kettle transformations and jobs were designed
> to be generic (not for a specific dataSet, but you have to design your own
> extractor which could be a simple csv-reader or maybe a DHS api-call). If
> you are interested, I will share them. Just be aware that they are
> currently in a very early and rough state and not documented. You'd have to
> bring along the willingness to dig yourself into kettle and be pain
> resistant to a certain degree :-)
>
> I'd be interested to hear from other experiences ...
>
> Have a nice sunday,
>
> Uwe
>
> ---
>
> Am 29.01.2016 um 17:31 schrieb Wilson, Randy:
>
> Not here unfortunately...just doing csv imports from DHS Excel files.
> Would be useful for our data warehouse.
> Randy
> On Jan 29, 2016 2:59 PM, "Olav Poppe" <olav.poppe@xxxxxx> wrote:
>
>> Hi all,
>> I wanted to hear if anyone has any experience with the DHS API (
>> http://api.dhsprogram.com/#/index.html), and using it to import survey
>> results into DHIS?
>>
>> Olav
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> *This message and its attachments are confidential and solely for the
> intended recipients. If received in error, please delete them and notify
> the sender via reply e-mail immediately.*
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET
Kampala
Uganda

IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya

IT Specialist (Servers, Networks and Security, Health Information Systems -
DHIS2 ) & Solar Consultant

+256 774149 775, + 256 759 800161

"I don't want to be anything other than what I have been - one tree hill "