dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Hi Pascal

I think posting dxf to the dataValueSets resource is the simplest option -
as described at http://www.dhis2.org/doc/snapshot/en/user/html/ch25s06.html.


While SDMX-HD is still lying unmaintained there is little point in wasting
excessive effort on it, unless it is of course to revamp the process, but
that is not today's proposition.

In most cases I prefer to use codes rather than uids as the resulting
datavaluesets are a bit easier to read and debug and it is reasonably
possible that you might have the same codes (particularly for facilities)
in both systems.  In which case you use the dataElementIdScheme="code"
and orgUnitIdScheme="code"
attributes as described at the bottom of
http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html.

Regards
Bob



On 6 August 2013 15:12, Pascal Brandt <pascal@xxxxxxxxx> wrote:

> Hi,
>
> We have at least one use case where we'll be pushing data, so it sounds
> like posting SDMX-HD or DXF to the dataValueSets resource is the right
> option for us.
>
> Thanks!
>
> Pascal
>
>
>
> On 6 August 2013 16:02, Jason Pickering <jason.p.pickering@xxxxxxxxx>wrote:
>
>> Hi Pascal,
>>
>> I suppose this really depends on your scenario. The integration engine is
>> useful for the case you describe, in that it can be scheduled to run on a
>> periodic (but regular) basis from within DHIS2. If you need to run things
>> on an ad-hoc basis, it may also be possible to manually trigger the engine,
>> but I am not sure about this. Bob Joliffe would know best.
>>
>>  You could also push data to DHIS2 on a periodic basis as described in
>> (2) of your mail. This is using the API of DHIS2.  DXF is the internal
>> format of DHIS2, while SDMX-HD is a more neutral "standard". So, if your
>> application can already "talk" SDMX-HD, then there is no need to develop an
>> exporter to DXF.
>>
>> (1) describes a cross transformation, which will transform SDMX-HD XML to
>> something which DHIS2 can understand. DXF can of course be imported
>> natively by DHIS2. This is essentially what is described here<http://www.dhis2.org/doc/snapshot/en/user/html/ch26s04.html>,
>> where a custom XSL is used to transform SDMX-HD.
>>
>> So, there are many ways, but it really depends on how you plan to do it.
>>
>> As for the documentation on DXF, there is not really anything at this
>> point, other than the stuff in the manual as far as I know (but
>> contributions are always welcome) :)
>>
>> So, it really comes down to whether you are going to pull or push. If you
>> are pulling data from your legacy app, you will need to use the integration
>> engine if you are doing it on a periodic basis. If you want to import your
>> applications XML, you would need to develop an XSL to transform it, similar
>> to the way it is done with SDMX-HD. Otherwise, you alter your legacy
>> application to export to a format which DHIS2 understands (like DXF or CSV)
>> and you push  this to DHIS2.
>>
>> Thats my understanding anyway, but others may have better info.
>>
>> Regards,
>> Jason
>>
>>
>> On Tue, Aug 6, 2013 at 3:53 PM, Pascal Brandt <pascal@xxxxxxxxx> wrote:
>>
>>> Hi,
>>>
>>> Thanks for the response Paulo. I also like the SDMX-HD and dataValueSets
>>> (DXF) options, but am still interested to hear what the DHIS2 devs think.
>>>
>>> Ciao,
>>> Pascal
>>>
>>>
>>>
>>> On 6 August 2013 15:45, Paulo Grácio <pgracio@xxxxxxxxxxxxxxxxxxxx>wrote:
>>>
>>>> Hi all,****
>>>>
>>>> ** **
>>>>
>>>> Please find my comments below. ****
>>>>
>>>> ** **
>>>>
>>>> Regards,****
>>>>
>>>> Paulo Grácio****
>>>>
>>>> ** **
>>>>
>>>> *From:* Pascal Brandt [mailto:pascal@xxxxxxxxx]
>>>> *Sent:* terça-feira, 6 de Agosto de 2013 14:56
>>>> *To:* Paulo Grácio
>>>> *Cc:* Jason Pickering; DHIS 2 developers; Chris Seebregts; Carl
>>>> Fourie; Ryan Crichton
>>>>
>>>> *Subject:* Re: [Dhis2-devs] Interoperability question****
>>>>
>>>> ** **
>>>>
>>>> Hi all,****
>>>>
>>>> ** **
>>>>
>>>> This thread is important for at least two separate projects that Jembi
>>>> is working on.****
>>>>
>>>> ** **
>>>>
>>>> What we're looking for is a way to automatically import data into the
>>>> DHIS. It's not a once-off import, but it's also not transactional (DHIS2
>>>> forms aren't being completed on a regular basis). For a once-off, we could
>>>> use CSV or DXF or one of the other options via the web interface and for
>>>> transactional we could use the Web API.
>>>>
>>>> What would be the recommended way to do this periodic bulk data load? I
>>>> see the following options:****
>>>>
>>>>    1. Sending data values using SDMX-HD<http://www.dhis2.org/doc/snapshot/en/user/html/ch25s08.html>
>>>>    ****
>>>>    2. Sending large bulks of data values<http://www.dhis2.org/doc/snapshot/en/user/html/ch25s09.html>
>>>>    ****
>>>>    3. Integration Engine<http://www.dhis2.org/doc/snapshot/en/user/html/ch26.html>
>>>>    ****
>>>>
>>>> *[Paulo Grácio] IMHO option 1 is definitely the most appropriated. To
>>>> have it automatically loaded into DHIS2 the best solution might be to
>>>> implement an adapter that translates data to SDMX-HD before send it to
>>>> DHIS2. Using this approach you also isolate your application specific data
>>>> format from other systems, communication is always done using standard
>>>> format. * *   *
>>>>
>>>> The main point is that needs to be automatic (no user interaction with
>>>> the UI) and we need to be able to initiate the process on an ad-hoc bases. Is
>>>> there a way to load CSV or DXF using the Web API? ****
>>>>
>>>> *[Paulo Grácio] Please have a look at section 25.6. Sending data
>>>> values of DHIS2 User Manual.*
>>>>
>>>> http://apps.dhis2.org/demo/api/dataValueSets****
>>>>
>>>> * *
>>>>
>>>> Also, I'm having some trouble finding the documentation for the DXF
>>>> format, is it available somewhere?****
>>>>
>>>> *[Paulo Grácio] By Morten Olav Hansen “This one is difficult, but we
>>>> will try and provide something. Our api is very much a moving target
>>>> though, ane because of some design choices we cannot just generate this.”
>>>> *
>>>>
>>>> https://bugs.launchpad.net/dhis2/+bug/990783**
>>>>
>>>> ** **
>>>>
>>>> Thanks for your help.****
>>>>
>>>> ** **
>>>>
>>>> Regards,****
>>>>
>>>> Pascal****
>>>>
>>>> ** **
>>>>
>>>> ** **
>>>>
>>>> On 1 August 2013 18:50, Paulo Grácio <pgracio@xxxxxxxxxxxxxxxxxxxx>
>>>> wrote:****
>>>>
>>>> Hi Jason,****
>>>>
>>>>  ****
>>>>
>>>> Basically what we are trying to achieved is a way to simple export data
>>>> from a legacy system into dhis2.  Once this system only generates
>>>> excel files as output we would like to import the generated file
>>>> directly(without modifications) to DHIS2 and have an import summary. **
>>>> **
>>>>
>>>>  ****
>>>>
>>>> We know that we can do it using several different approaches and we are
>>>> considering them J.****
>>>>
>>>>  ****
>>>>
>>>> Regards,****
>>>>
>>>> Paulo Grácio****
>>>>
>>>>  ****
>>>>
>>>> *From:* Jason Pickering [mailto:jason.p.pickering@xxxxxxxxx]
>>>> *Sent:* quinta-feira, 1 de Agosto de 2013 13:26
>>>> *To:* Paulo Grácio****
>>>>
>>>>
>>>> *Cc:* DHIS 2 developers
>>>> *Subject:* Re: [Dhis2-devs] Interoperability question****
>>>>
>>>>  ****
>>>>
>>>> Hi Paulo,****
>>>>
>>>> I am not sure it would be worth it, but looks like it might work. Why
>>>> can't you just simply export CSV instead? Seems that given the structure of
>>>> your data, you could just simply use the CSV importer instead of setting up
>>>> an integration route?****
>>>>
>>>>
>>>> Regards,****
>>>>
>>>> Jason****
>>>>
>>>>  ****
>>>>
>>>>  ****
>>>>
>>>> On Wed, Jul 31, 2013 at 7:03 PM, Paulo Grácio <
>>>> pgracio@xxxxxxxxxxxxxxxxxxxx> wrote:****
>>>>
>>>> Hi all,****
>>>>
>>>>  ****
>>>>
>>>> We are wondering about the best form to import in DHIS2 data from
>>>> legacy applications.****
>>>>
>>>>  ****
>>>>
>>>> In chapter “26.4. Transforming data - a Java route” of the User Manual
>>>> we see that there is a way to do this integration using Java routes, but we
>>>> are having some difficulties in implementing a proof of concept.****
>>>>
>>>>  ****
>>>>
>>>> In our PoC scenario we have a XLS file sent by an external application
>>>> (DataValues.xls in attach). Our idea is to create a Java route in order to
>>>> import all data values that exist in that file.****
>>>>
>>>>  ****
>>>>
>>>> Following the tutorial “Customer 3: Excel via e-mail” presented in the
>>>> Apache Camel page -
>>>> http://camel.apache.org/tutorial-business-partners.html - we have been
>>>> able to transform the XLS file in a list of DataValue objects. ****
>>>>
>>>>  ****
>>>>
>>>> Basically, we only had to create a XLSDataIn route and a
>>>> ExcelConverterBean (files in attach). The XLSDataIn takes an InputStream of
>>>> an XLS file and calls the ExcelConverterBean. The ExcelConverterBean
>>>> iterates through the file, and for each data value row, creates a new
>>>> DataValue object and adds that object to a list of DataValue objects.**
>>>> **
>>>>
>>>>  ****
>>>>
>>>> Now we are having some issues because we don’t know the best way to
>>>> insert these data values in DHIS2.****
>>>>
>>>>  ****
>>>>
>>>> The questions we are having are:****
>>>>
>>>> 1.       The approach that we have used to implement our PoC is the
>>>> correct one? I mean, XLS -> List<DataValue> -> import is correct or we
>>>> should use XLS -> CSV -> import?****
>>>>
>>>> 2.       How should we proceed, after the XLS file has been processed
>>>> by the ExcelConverterBean?****
>>>>
>>>>  ****
>>>>
>>>>  ****
>>>>
>>>> Thanks for your support,****
>>>>
>>>> *Paulo Grácio <*%20your-email%20*>*
>>>> *Technical Manager*
>>>> Skype: paulojrgracio****
>>>>
>>>> *Critical Software Mozambique* <http://www.criticalsoftware.co.uk/>
>>>> *Dependable Technologies for Critical Systems*****
>>>>
>>>> Critical Software Mozambique is a subsidiary of Critical Software, a
>>>> CMMI® Level 5 rated Company <http://cmmiinstitute.com/>
>>>> CMMI® is registered in the USPTO by CMU <http://www.cmu.edu/>****
>>>>
>>>> Rua Pereira Marinho, 179
>>>> Bairro de Sommerchield
>>>> Maputo
>>>> Moçambique
>>>> Phone: (+258) 214 951 45
>>>> Fax: (+258) 214 951 46****
>>>>
>>>> DISCLAIMER: This message is confidential and may contain privileged
>>>> information. It is for use only by the people or entities to whom it is
>>>> addressed. If you are not an intended recipient, you should not disclose,
>>>> distribute, copy, print, rely on or otherwise make use of this message. If
>>>> an addressing or transmission error has misdirected it to you we would be
>>>> grateful if you would please notify the sender by return, before deleting
>>>> it from your system.****
>>>>
>>>>  ****
>>>>
>>>>  ****
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp****
>>>>
>>>>  ****
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp****
>>>>
>>>>
>>>>
>>>> ****
>>>>
>>>> ** **
>>>>
>>>> --
>>>> *Pascal Brandt
>>>> Senio**r Software Developer, Jembi Health Systems |  SOUTH AFRICA
>>>> Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
>>>> E-mail: pascal@xxxxxxxxx* ****
>>>>
>>>
>>>
>>>
>>> --
>>> *Pascal Brandt
>>> Senio**r Software Developer, Jembi Health Systems |  SOUTH AFRICA
>>> Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
>>> E-mail: pascal@xxxxxxxxx*
>>>
>>
>>
>
>
> --
> *Pascal Brandt
> Senio**r Software Developer, Jembi Health Systems |  SOUTH AFRICA
> Mobile: +27 84 827 9342 | Office: +27 21 701 0939 | Skype: psbrandt
> E-mail: pascal@xxxxxxxxx*
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>