dhis2-users team mailing list archive

["Shinichi Suzuki" <shin461@xxxxxxxxx>]

Hello Jason,

Thanks a lot. I will try to learn on ETL tools.
Also, look into dhis2 documents you pointed.

Best regards,
   Shinichi Suzuki

-----Original Message-----
From: Jason Pickering [mailto:jason.p.pickering@xxxxxxxxx]
Sent: Wednesday, November 10, 2010 6:34 PM
To: Shinichi Suzuki
Cc: Knut Staring; dhis2-users@xxxxxxxxxxxxxxxxxxx
Subject: Re: [Dhis2-users] Data Import tools for DHIS2

Hi Shinchi,
The format I was referring to is actually DXF or DHIS Exchange Format
(I think?). Anyway, it is the internal XML format used by DHIS to
transmit data between different DHIS systems, as well as for import of
data to DHIS from external sources, such as your legacy database. An
XSD of the schema is available here

http://bazaar.launchpad.net/~dhis2-documenters/dhis2/dhis2-docbook-docs/file
s/head%3A/src/schemas/dxf_v1_schema/

ETL tools are simply useful for the transformation of data from data
bases of one format, into different formats, such as CSV or XML (or
other database structures). I personally feel that the learning curve
that Roger mentioned is not so steep, and find Kettle to be extremely
intuitive, but this is very much a personal preference. There are many
different ways to migrate the data, conversion to XML being one of
them. It is really just a matter of personal preference using tools
that you are comfortable with.

Regards,
Jason


On Wed, Nov 10, 2010 at 3:18 PM, Shinichi Suzuki <shin461@xxxxxxxxx> wrote:
> Hello Json,
>
> Thanks for your answer.
> I tried to export the some data and looked it by text editor to understand
> XDF format.
> From my web search, XDF is a AutoCAD or other 3D CAD standard format.
> I could not create from Excel or CSV file to that '.action' file like.
> ETL tools are looks useful for data exporting for private analysis and not
> for import into DHIS2.
> Please give me your recommended tool(s) name to do this.
> Hopefully, could you send me your document draft when you write?
> I am afraid I am confusion right now.
>
> Many thanks again,
>  Shinichi Suzuki
>
> -----Original Message-----
> From: Jason Pickering [mailto:jason.p.pickering@xxxxxxxxx]
> Sent: Tuesday, November 09, 2010 5:25 PM
> To: Shinichi Suzuki
> Cc: Knut Staring; dhis2-users@xxxxxxxxxxxxxxxxxxx
> Subject: Re: [Dhis2-users] Data Import tools for DHIS2
>
> Hi Shinchi,
>
> In fact as Knut highlighted, the appropriate method would be the
> production of XML data (DXF) which DHIS2 already has robust import
> mechanisms for. We have used direct insertion of data in the past, but
> as Knut points out, and I will again, you must be very careful when
> doing this. Much better to try and get the data in the correct format,
> and import it in the recommended way via DXF if you can.
>
> I am right in the middle of migrating data from a legacy system into
> DHIS. I will try and document as much of the process as I can, which
> may help you in your import.
>
> Best regards,
> Jason
>
>
> On Tue, Nov 9, 2010 at 3:35 PM, Shinichi Suzuki <shin461@xxxxxxxxx> wrote:
>> Hi Knut,
>>
>> Thanks a lot. I am encouraged to read your mail.
>> I will try to read two home pages. But I feel familiar with your Access
> based
>> tools.
>> I know your concerning to use these direct modification.
>> I think DHIS2 should have a API to do this and open them. Then your
> concerning
>> will be disappeared.
>>
>> Best regards,
>>    Shinichi Suzuki
>>
>> -----Original Message-----
>> From: Knut Staring [mailto:knutst@xxxxxxxxx]
>> Sent: Monday, November 08, 2010 11:28 AM
>> To: Shinichi Suzuki
>> Cc: larshelge@xxxxxxxxx; dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Subject: Re: Data Import tools for DHIS2
>>
>> Hi Shinichi,
>>
>> You are completely right that it is important to be able to easily
>> import the orgunit hierarchy and historical data into DHIS2. The
>> general term for such operations is Extract, Transform and Load (ETL),
>> for which there are a number of powerful general tools available. Here
>> are two:
>> http://kettle.pentaho.com/
>> http://www.talend.com/index.php
>> I have personally also used ODBC connections in Access to transform
>> and load Excel data into Postgres directly.
>>
>> A disadvantage of going directly into the database is that one looses
>> the validations that the DHIS2 import mechanism can perform (in
>> addition to the database constraints), and of course it is burdensome
>> to learn to use the above tools. DHIS2 has the capability to import
>> XML files, and thus also modern Excel files (.xlsx), and this should
>> probably be the common way.However, given the many different types of
>> data and vast range of potential data sources, we will probably never
>> have a simple wizard that does it all (and there is no sense in trying
>> to replicate Kettle or Talend).
>>
>> Still, we have been thinking about defining a suitable standard format
>> for such import, as well as providing the user with some assistance in
>> transforming the data into shape for loading. We have also done some
>> work on extraction of data automatically from a large number of Excel
>> files (e.g. one or more per district) using Python. Work remains
>> before this work will reach a stage where it is robust and generic
>> enough - but I think working with the Kenyan data could help move this
>> process forward. I will refresh my memory on the status of the Python
>> work and get back to you.
>>
>> Knut
>>
>>
>> On Mon, Nov 8, 2010 at 8:42 AM, Shinichi Suzuki <shin461@xxxxxxxxx>
wrote:
>>> Hello, Knut and dhis2_devs, users
>>>
>>> Do you have a data import tool written by Python and available to use?
>>> I need the tools for the following area.
>>> 1. Existing HIS EXCEL data import to DHIS2. We want to load about
several
>>> years historical data which has been corrected into EXCEL files.
>>> 2. Import from Master Facility List (Web based Facility Master
> maintenance
>>> system) to DHIS2 for facility data synchronization. This MFL has a
>>> capability to export the data into EXCEL file. But MFL does not have a
>>> "Short name" and need to add the data manually. Then want to import it.
>>>
>>> If you do not have it or not available to use it, please advise me if
>>> possible.
>>> I believe this import capability is important for the new user of DHIS
> and
>>> also could use inter system connection more widely.
>>>
>>> Best regards,
>>>  Shinichi Suzuki
>>> -----------------------------------------------------
>>> Shinichi Suzuki
>>> MIS Division MOPHS: LG37 AFYA House, Nairobi, KENYA
>>> E-Mail: shin461@xxxxxxxxx
>>> Phone: 0712-754-963
>>> (JICA Senior Volunteer 21-4)
>>> -----------------------------------------------------
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Cheers,
>> Knut Staring
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+260968395190
>
>
>



--
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+260968395190