dhis2-users team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

This was a very trivial lab test,so not really conclusive at all.  I would
just give it a try and see. If you see differences, please let the devs
know.

Given the scale of what you are attempting, have you considered using
direct SQL injection? Not that I am recommending that route as there are
many pitfalls, but it might be an option if implemented properly,
especially considering your reported architecture.

Regards
Jason

On Tue, Feb 2, 2016, 17:04 Uwe Wahser <uwe@xxxxxxxxx> wrote:

> Hi Jason,
>
> thanks for sharing the links. As I can see on a quick glance, you are also
> experimenting with the ADX-api - did you observe any significant
> performance
> differences between ADX and dataValueSets apis?
>
> Regards,
>
> Uwe
>
> > Jason Pickering <jason.p.pickering@xxxxxxxxx> hat am 2. Februar 2016 um
> 18:21
> > geschrieben:
> >
> >
> > Hi Olav,
> > I have not worked with the DHS API per se, but have imported lots of data
> > using the same approach which they outline here (
> > http://api.dhsprogram.com/#/samples-r.cfm)
> >
> > I have written up a walkthrough of getting data out of one DHIS instance
> > and into another one, and I think the basic principles would be the same
> (
> > http://rpubs.com/jason_p_pickering/139589)
> >
> > Metadata needs to be mapped (or created), the data needs to be reshaped,
> > and correctly formatted.
> >
> > It should not be too difficult. I used R, but there are other examples
> with
> > Python and JavaScript on their examples page.
> >
> > Regards,
> > Jason
> >
> >
> > On Tue, Feb 2, 2016 at 3:31 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
> > wrote:
> >
> > > Dear Uwe,
> > >
> > > Have you tried to send data via the endpoint api/dataValueSets, it may
> be
> > > faster. Just stage your data and push it once.
> > >
> > >
> http://dhis2.github.io/dhis2-docs/master/en/developer/html/ch01s13.html#d5e1372
> > >
> > > Also to note, is how you send it, I have seen curl taking ages to
> submit
> > > individual values via the api. You need to send it as once file via
> once
> > > request or implement concurrency.
> > >
> > > Alex
> > >
> > > On Tue, Feb 2, 2016 at 5:13 PM, Olav Poppe <olav.poppe@xxxxxx> wrote:
> > >
> > >> Hi Randy and Uwe,
> > >> thanks, interesting to hear you experiences. Uwe, what you are
> working on
> > >> sounds quite a bit more complicated, and not least with far more
> data. I
> > >> image that with household surveys, it would be a matter of < 100
> indicators
> > >> for < 200 orgunits for 2-3 periods, i.e. a fraction of what you are
> dealing
> > >> with!
> > >>
> > >> Olav
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> 31. jan. 2016 kl. 09.29 skrev uwe wahser <uwe@xxxxxxxxx>:
> > >>
> > >> Hi Olav & Randy,
> > >>
> > >> I am currently banging on kettle (aka Pentaho DI) to extract data
> from a
> > >> source-system (SQL-ERP in our case) into DHIS2 dataSets in json
> format. In
> > >> our current test-scenario (2 dataElements in a dataSet with a
> > >> categoryCombination of 5 categories) we are currently updating ca. 4
> mio
> > >> dataValues every night in a pseudo-delta mode (reading all data from
> > >> source, comparing to what is there in DHIS2 already, then only pushing
> > >> records for creating, updating or deleting dataValues into the api:
> ca.
> > >> 150k per night in 1 hour, initial load was 7hrs). We still have to
> prove,
> > >> that this is feasible when setting up the first real life dataSet
> where
> > >> there will be more categories and more dataElements, thus exploding
> the
> > >> number of dataValues.
> > >>
> > >> Getting there was a bit painful, but now it seems to work. I chose
> kettle
> > >> instead of Talend ETL (both open source) as it seemed to be easier to
> get
> > >> used to. However, from a data warehouse perspective I'd prefer to have
> > >> DHIS2 offering some sort of an integrated ETL landscape on the long
> run,
> > >> which would also allow to aggregate data from tracker into dataSets,
> > >> tracker to tracker, dataSets to dataSets etc.
> > >>
> > >> Our current version of the kettle transformations and jobs were
> designed
> > >> to be generic (not for a specific dataSet, but you have to design
> your own
> > >> extractor which could be a simple csv-reader or maybe a DHS
> api-call). If
> > >> you are interested, I will share them. Just be aware that they are
> > >> currently in a very early and rough state and not documented. You'd
> have to
> > >> bring along the willingness to dig yourself into kettle and be pain
> > >> resistant to a certain degree :-)
> > >>
> > >> I'd be interested to hear from other experiences ...
> > >>
> > >> Have a nice sunday,
> > >>
> > >> Uwe
> > >>
> > >> ---
> > >>
> > >> Am 29.01.2016 um 17:31 schrieb Wilson, Randy:
> > >>
> > >> Not here unfortunately...just doing csv imports from DHS Excel files.
> > >> Would be useful for our data warehouse.
> > >> Randy
> > >> On Jan 29, 2016 2:59 PM, "Olav Poppe" <olav.poppe@xxxxxx> wrote:
> > >>
> > >>> Hi all,
> > >>> I wanted to hear if anyone has any experience with the DHS API (
> > >>> http://api.dhsprogram.com/#/index.html), and using it to import
> survey
> > >>> results into DHIS?
> > >>>
> > >>> Olav
> > >>>
> > >>> _______________________________________________
> > >>> Mailing list: https://launchpad.net/~dhis2-users
> > >>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > >>> Unsubscribe : https://launchpad.net/~dhis2-users
> > >>> More help   : https://help.launchpad.net/ListHelp
> > >>>
> > >>>
> > >> *This message and its attachments are confidential and solely for the
> > >> intended recipients. If received in error, please delete them and
> notify
> > >> the sender via reply e-mail immediately.*
> > >>
> > >> _______________________________________________
> > >> Mailing list: https://launchpad.net/~dhis2-users
> > >> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > >> Unsubscribe : https://launchpad.net/~dhis2-users
> > >> More help   : https://help.launchpad.net/ListHelp
> > >>
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> Mailing list: https://launchpad.net/~dhis2-users
> > >> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > >> Unsubscribe : https://launchpad.net/~dhis2-users
> > >> More help   : https://help.launchpad.net/ListHelp
> > >>
> > >>
> > >
> > >
> > > --
> > > Alex Tumwesigye
> > >
> > > Technical Advisor - DHIS2 (Consultant),
> > > Ministry of Health/AFENET
> > > Kampala
> > > Uganda
> > >
> > > IT Consultant - BarefootPower Uganda Ltd, SmartSolar, Kenya
> > >
> > > IT Specialist (Servers, Networks and Security, Health Information
> Systems
> > > - DHIS2 ) & Solar Consultant
> > >
> > > +256 774149 775, + 256 759 800161
> > >
> > > "I don't want to be anything other than what I have been - one tree
> hill "
> > >
> > > _______________________________________________
> > > Mailing list: https://launchpad.net/~dhis2-users
> > > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > > Unsubscribe : https://launchpad.net/~dhis2-users
> > > More help   : https://help.launchpad.net/ListHelp
> > >
> > >
> >
> >
> > --
> > Jason P. Pickering
> > email: jason.p.pickering@xxxxxxxxx
> > tel:+46764147049
> > _______________________________________________
> > Mailing list: https://launchpad.net/~dhis2-users
> > Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~dhis2-users
> > More help   : https://help.launchpad.net/ListHelp
>