dhis2-users team mailing list archive

[Uwe Wahser <uwe@xxxxxxxxx>]

Hi Jason,

thanks for sharing the links. As I can see on a quick glance, you are also
experimenting with the ADX-api - did you observe any significant performance
differences between ADX and dataValueSets apis?

Regards, 

Uwe

> Jason Pickering <jason.p.pickering@xxxxxxxxx> hat am 2. Februar 2016 um 18:21
> geschrieben:
> 
> 
> Hi Olav,
> I have not worked with the DHS API per se, but have imported lots of data
> using the same approach which they outline here (
> http://api.dhsprogram.com/#/samples-r.cfm)
> 
> I have written up a walkthrough of getting data out of one DHIS instance
> and into another one, and I think the basic principles would be the same (
> http://rpubs.com/jason_p_pickering/139589)
> 
> Metadata needs to be mapped (or created), the data needs to be reshaped,
> and correctly formatted.
> 
> It should not be too difficult. I used R, but there are other examples with
> Python and JavaScript on their examples page.
> 
> Regards,
> Jason
> 
> 
> On Tue, Feb 2, 2016 at 3:31 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
> wrote:
> 
> > Dear Uwe,
> >
> > Have you tried to send data via the endpoint api/dataValueSets, it may be
> > faster. Just stage your data and push it once.
> >
> > http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s13.html#d5e1372
> >
> > Also to note, is how you send it, I have seen curl taking ages to submit
> > individual values via the api. You need to send it as once file via once
> > request or implement concurrency.
> >
> > Alex
> >
> > On Tue, Feb 2, 2016 at 5:13 PM, Olav Poppe <olav.poppe@xxxxxx> wrote:
> >
> >> Hi Randy and Uwe,
> >> thanks, interesting to hear you experiences. Uwe, what you are working on
> >> sounds quite a bit more complicated, and not least with far more data. I
> >> image that with household surveys, it would be a matter of < 100 indicators
> >> for < 200 orgunits for 2-3 periods, i.e. a fraction of what you are dealing
> >> with!
> >>
> >> Olav
> >>
> >>
> >>
> >>
> >>
> >>
> >> 31. jan. 2016 kl. 09.29 skrev uwe wahser <uwe@xxxxxxxxx>:
> >>
> >> Hi Olav & Randy,
> >>
> >> I am currently banging on kettle (aka Pentaho DI) to extract data from a
> >> source-system (SQL-ERP in our case) into DHIS2 dataSets in json format. In
> >> our current test-scenario (2 dataElements in a dataSet with a
> >> categoryCombination of 5 categories) we are currently updating ca. 4 mio
> >> dataValues every night in a pseudo-delta mode (reading all data from
> >> source, comparing to what is there in DHIS2 already, then only pushing
> >> records for creating, updating or deleting dataValues into the api: ca.
> >> 150k per night in 1 hour, initial load was 7hrs). We still have to prove,
> >> that this is feasible when setting up the first real life dataSet where
> >> there will be more categories and more dataElements, thus exploding the
> >> number of dataValues.
> >>
> >> Getting there was a bit painful, but now it seems to work. I chose kettle
> >> instead of Talend ETL (both open source) as it seemed to be easier to get
> >> used to. However, from a data warehouse perspective I'd prefer to have
> >> DHIS2 offering some sort of an integrated ETL landscape on the long run,
> >> which would also allow to aggregate data from tracker into dataSets,
> >> tracker to tracker, dataSets to dataSets etc.
> >>
> >> Our current version of the kettle transformations and jobs were designed
> >> to be generic (not for a specific dataSet, but you have to design your own
> >> extractor which could be a simple csv-reader or maybe a DHS api-call). If
> >> you are interested, I will share them. Just be aware that they are
> >> currently in a very early and rough state and not documented. You'd have to
> >> bring along the willingness to dig yourself into kettle and be pain
> >> resistant to a certain degree :-)
> >>
> >> I'd be interested to hear from other experiences ...
> >>
> >> Have a nice sunday,
> >>
> >> Uwe
> >>
> >> ---
> >>
> >> Am 29.01.2016 um 17:31 schrieb Wilson, Randy:
> >>
> >> Not here unfortunately...just doing csv imports from DHS Excel files.
> >> Would be useful for our data warehouse.
> >> Randy
> >> On Jan 29, 2016 2:59 PM, "Olav Poppe" <olav.poppe@xxxxxx> wrote:
> >>
> >>> Hi all,
> >>> I wanted to hear if anyone has any experience with the DHS API (
> >>> http://api.dhsprogram.com/#/index.html), and using it to import survey
> >>> results into DHIS?
> >>>
> >>> Olav
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~dhis2-users
> >>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~dhis2-users
> >>> More help   : https://help.launchpad.net/ListHelp
> >>>
> >>>
> >> *This message and its attachments are confidential and solely for the
> >> intended recipients. If received in error, please delete them and notify
> >> the sender via reply e-mail immediately.*
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~dhis2-users
> >> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~dhis2-users
> >> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~dhis2-users
> >> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~dhis2-users
> >> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >
> >
> > --
> > Alex Tumwesigye
> >
> > Technical Advisor - DHIS2 (Consultant),
> > Ministry of Health/AFENET
> > Kampala
> > Uganda
> >
> > IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya
> >
> > IT Specialist (Servers, Networks and Security, Health Information Systems
> > - DHIS2 ) & Solar Consultant
> >
> > +256 774149 775, + 256 759 800161
> >
> > "I don't want to be anything other than what I have been - one tree hill "
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help   : https://help.launchpad.net/ListHelp
> >
> >
> 
> 
> -- 
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+46764147049
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp