dhis2-users team mailing list archive

[Uwe Wahser <uwe@xxxxxxxxx>]

HI Henri,

sorry for for chipping in late, I am on vaccation. We are using kettle for
extracting data into DHIS2 as a prototype for a datawarehouse in a corporate
setting. A few of Aarons points on Talend apply for kettle as well. Things can
get tricky, but that is partly because of the nature of data import into DHIS2.
Support, community etc are not as vivid as the DHIS2 community, but you still
find some material on the net.

It runs and does the job. Currently we are batteling with memory issues as I am
trying to find some suitable method of packaging data partitions. Again, that is
not only kettles fault.

I just found out that the MoH is also using kettle in their HMIS setting, so I
am quite confident to have a few ressource persons in the country for
sustainable hand-over.

Compared to R etc I like the graphical data-flow, which helps when discussing
transformations with users. You can actually see what is going into and is
coming out of a process with a mouse-click.

Our current transformations are quite generique for extracting from a custom
data-source, hoping to put them on the net after some clean ups.

Generally speaking I'd prefere to have more extensive ETL capabilites within
DHIS2, though, e.g. to better synchronise and monitor load-jobs, not having to
allocate memory to two java engines, consistent management of data-elements
etc... 

Regards,

Uwe
> henri claude moungui <henrimoungui@xxxxxxxx> hat am 23. März 2016 um 11:01
> geschrieben:
> 
> 
> Hi.Thank you Ant, Knut, Jason and Aaron for the tips. I'm sure the discussions
> we are having will make us continue to improve the DHIS2 platform,
> particularly for datawarehouse implementations.I was also looking for Kettle
> and Talend for this purpose. Finally, I found DHIS2 very comfortable for
> Export and Import as it is possible to do this step by step (you choose which
> metadata you want to export/import). Now I see the way is how to TRANSFORM the
> Excel databases to fit with the exported file. For now, I intend to do this
> manually...
> Thank you.
>  
>   
> 
>     Le Mardi 22 mars 2016 7h58, Aaron Gladders <agladders@xxxxxxxxxx> a écrit
> :
>  
> 
>  Hi - we did some work using Talend based on client desires.  We found it to
> be an incredible pain in the butt.  Although our group are typically senior
> software developers or technical architects we had to pull in vendor
> consultants to get training on how to use it and ultimately dumped it.  Beware
> the marketing - they all promote self-service but I suspect you'll either be
> bringing in their trainers, hiring their consultants or finding your own
> software developers to actually get the job done.
> Aaron
> 
> On Tue, Mar 22, 2016 at 12:04 AM, Knut Staring <knutst@xxxxxxxxx> wrote:
> 
> Good point, Ant.
> Here is a link to that conversation, with many interesting
> points:https://lists.launchpad.net/dhis2-users/msg09234.html
> 
> Knut
> On Tue, Mar 22, 2016 at 6:23 AM, Ant Snyman <ant@xxxxxxxx> wrote:
> 
> Hi Henri,I read on this forum that someone was attempting to use Kettle
> (Pentaho Data Integration). I don't know whether they succeeded.RegardsAnt
> Snyman
> On 21 March 2016 at 18:21, henri claude moungui <henrimoungui@xxxxxxxx> wrote:
> 
> Thank you Knut.
> Henri 
>   
> 
>     Le Lundi 21 mars 2016 15h25, Knut Staring <knutst@xxxxxxxxx> a écrit :
>  
> 
>  Hi Henri,
> There is no specific 3rd party software that you have to use, people use many
> different tools. Perhaps most commonly, people use scripts in Python or R or
> similar to load data. You can import both metadata and data in CSV format, for
> example.
> http://dhis2.github.io/dhis2-docs/2.21/en/developer/html/ch01s10.html
> http://dhis2.github.io/dhis2-docs/2.21/en/user/html/ch21.html
> 
> Knut
> On Mon, Mar 21, 2016 at 3:15 PM, henri claude moungui <henrimoungui@xxxxxxxx>
> wrote:
> 
> Hi to all.I want to test the ETL process with DHIS2. Is there an automated way
> to do this? is there a recommended tiers soft to use for this? my operational
> DB to use are excel files.Thank you in advance.
> Henri Moungui 
>  
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> 
> -- 
> Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
> Skype: knutstarhttp://dhis2.org
> 
>    
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> 
> -- 
> 
> Ant SnymanCell: 0824910449Landline: 028 2713242
> Health Information Systems Program - SA
> This message and any attachments are subject to a disclaimer publishedat
> http://www.hisp.org/policies.html#comms_disclaimer. Please read the disclaimer
> before opening any attachment or taking any otheraction in terms of this
> electronic transmission.  If you cannot access thedisclaimer, kindly send an
> email to disclaimer@xxxxxxxx and a copy will be provided to you. Byreplying to
> this e-mail or opening any attachment you agree to be bound by theprovisions
> of the disclaimer.
> 
> 
> 
> -- 
> Knut StaringDept. of Informatics, University of OsloNorway: +4791880522
> Skype: knutstarhttp://dhis2.org
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> 
> -- 
> Aaron Gladders, 2Paths
> http://www.2paths.com/
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to    : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help  : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp