dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi there.

One of the problems with ETL is one must be a bit careful about
ignoring certain business rules that may be built into the procedural
code, but not the DB. It seems that most of DHIS2s logic is built into
the procedural layer (Java / JS) and not the DB itself. Of course
there is some there...foreign key relations and such. I think it could
be a bit risky, but it depends on which tables you are touching. I
like the idea of transformation to DXF, followed by importation, as we
can be sure that all the business logic will be enforced through the
GUI. Direct injection of routine/semipermanent data has gone quite
smoothly for me. I imported a big hunk of population data with Kettle,
and it worked OK. I am not sure there will be a single transformation
that we can offer people, but rather some well documented examples and
tips. Every single data source that is a candidate for transformation
will probably be very different and require some level of
customization.

It might be good to start a branch somewhere "contrib" so we can begin
to collaborate on these issues. it has been mentioned in the past.
Perhaps a "contrib/etl" and "contrib/reports" would be good. I have
developed a couple of generic BIRT reports that others may find
useful. I would think ETL transforms would be useful as well to start
to assemble. I can do it if there is consensus with the group.

Regards,
Jason


On Mon, Nov 2, 2009 at 5:36 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
> Hello,
> As DHIS 2 becomes increasingly popular, it becomes more and more urgent to
> be able to import metadata and data from other systems.
> We have had discussions of using a Extract, transfer, load tools (e.g.
> Kettle/Pentaho Data Integration) - in which case they probably should be
> embedded in some way?
> Ola and I just came up with a simple format now which we think could
> represent many types of data "out there". That is, people should be guided
> in transforming their data into a simple tabular format, which could then be
> read by DHIS.
> We have updated an existing blueprint:
> https://blueprints.launchpad.net/dhis2/+spec/import-data
> As an alternative approach could be to require people to convert their data
> to SDMX or DXF, it would be good to have everyone's thoughts on this.
> Knut
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>