dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Well, of course, as was the case with my previous mail, I saw my
mistake immediately after I sent the mail, which was an incorrect
attribute in the dataValue block  (periodid instead of period). After
several more tedious hours of looking at XML, I have something that
works, and plan to share my experience with the group once I am done.
This conversion of legacy data is always something that needs to be
done, but up until now, it has been a process that we have not spent
so much time documenting, especially via the route from a legacy
database to DXF.

A few more impressions  and comments as well.

1) Periods...DHIS2 will happily import periods which are actually not
quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
1998. It would be a good to put validation on this.  Of course, the
dates should be valid to begin with, but perhaps we cannot always
trust legacy data.

I made another mistake as well, and had an end date of "1997-09-31"
which is not a valid date. I kept getting an NPE, but could not figure
out what the heck was causing it. The date itself was actually
invalid, so the enddate field ended up being transformed to NULL,
which violates the NOT NULL constraint on the enddate in periods, and
thus the NPE.

 A more informative error message would have been helpful. In general,
trying to untangle the error messages that were provided during the
workup to a functional XML file resulted in a small migraine.

2) Users...When considering how to import a big block of data into an
empty database, I could not really figure out how to specify a user,
which is used in the data value attribute "storedBy"

<dataValue dataElement="2" period="37" source="6795" value="17"
storedBy="14" timeStamp="" comment="PATH Harvard Dataset"
categoryOptionCombo="11"></dataValue>

Looking at the DXF XSD it is pretty clear...

 <xs:attribute name="storedBy" type="xs:integer"/>
 that the ID of the user has to be provided, but there does not appear
to be a way to specify a user actually anywhere else in the XML. I
know in DHIS 1.4, users can be transmitted from one DB to the other. I
was not really sure how to handle this, so I had DHIS2 create the DB
first, and then I had a peek in the users table to see what the
"admin" user's ID actually was. Again..a bit troublesome if you want
to do a bulk import of data from XML, without having to mess around
with the DB.


I plan on adding a bit of a case study to the documentation once I am
done, and will inform the devs once I have had time to do this.

Regards,
Jason




On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering
<jason.p.pickering@xxxxxxxxx> wrote:
> Hi again,
> I had sent a message to the group earlier regarding importation of
> data from a legacy source by DXF. Previously, I have used direct
> inserts into the DHIS2 database, but because of the nature of this
> data source, I want to try and use DXF. I am having problems however,
> and could use some pointers.
>
> The DXF file is here (382kB)
>
> http://dl.dropbox.com/u/11802917/test_dxf_export8.7z
>
>
> Basically, everything runs smoothly, up until the last stage when the
> data values are to be imported.. When I get the error message posted
> at the end of the mail.
>
> If I import the same DXF without the <dataValues> block, it imports
> just fine. I really cannot decipher the log message however to
> determine what the exact problem is. Any help appreciated.
>
> Thanks,
> Jason
>
>
>
>
> * INFO  08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
> Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
> _00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
> * INFO  08:08:50,782 Internal process started (AbstractStatementInternalProcess.
> java [Thread-32])
> * INFO  08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
> * INFO  08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
> ead-32])
> * INFO  08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
> * WARN  08:08:50,836 Resource Leaking:  Could not remove uploaded file 'C:\Users
> \jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
> d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
> * INFO  08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
> ead-32])
> * INFO  08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
> 2])
> * INFO  08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
> ad-32])
> * INFO  08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
>  [Thread-32])
> * INFO  08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
> java [Thread-32])
> * INFO  08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
> ava [Thread-32])
> * INFO  08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
> * INFO  08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
> * INFO  08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
> [Thread-32])
> * INFO  08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
> 32])
> * INFO  08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
> Thread-32])
> * INFO  08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
> ad-32])
> * INFO  08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
> a [Thread-32])
> * INFO  08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
> 32])
> * ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
> ava [Thread-32])
> java.lang.NumberFormatException: null
>        at java.lang.Integer.parseInt(Integer.java:417)
>        at java.lang.Integer.parseInt(Integer.java:499)
>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
> ValueConverter.java:195)
>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
> (DefaultConverterInvoker.java:45)
>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
> on(AopUtils.java:309)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
> inpoint(ReflectiveMethodInvocation.java:183)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
> voke(TransactionInterceptor.java:110)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
> cAopProxy.java:202)
>        at $Proxy49.invokeRead(Unknown Source)
>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
> er.java:968)
>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
> ortService.java:185)
>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
> portInternalProcess.java:89)
>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
> (AbstractStatementInternalProcess.java:91)
>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
> (AbstractStatementInternalProcess.java:40)
>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
> 25)
>        at java.lang.Thread.run(Thread.java:662)
> java.lang.NumberFormatException: null
>        at java.lang.Integer.parseInt(Integer.java:417)
>        at java.lang.Integer.parseInt(Integer.java:499)
>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
> ValueConverter.java:195)
>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
> (DefaultConverterInvoker.java:45)
>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
> on(AopUtils.java:309)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
> inpoint(ReflectiveMethodInvocation.java:183)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:150)
>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
> voke(TransactionInterceptor.java:110)
>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
> ReflectiveMethodInvocation.java:172)
>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
> cAopProxy.java:202)
>        at $Proxy49.invokeRead(Unknown Source)
>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
> er.java:968)
>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
> ortService.java:185)
>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
> portInternalProcess.java:89)
>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
> (AbstractStatementInternalProcess.java:91)
>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
> (AbstractStatementInternalProcess.java:40)
>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
> 25)
>        at java.lang.Thread.run(Thread.java:662)
> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
> ewForm.vm [line 92, column 157]
> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
> ewForm.vm [line 93, column 161]
>
>
>
> Current user:
>    admin
> Version:
>    2.0.5
> Build revision:
>    2095
> Build date:
>    2010-11-25 10:47
> User agent:
>    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
> Gecko/20101203 Firefox/3.6.13
> External configuration directory:
>    C:\Program Files\DHIS2-2.0.5\conf
> Environment variable:
>    DHIS2_HOME
> Database type:
>    PostgreSQL
> Database name:
>    harvard
> Database user:
>    postgres
> Java Opts:
> Java version:
>    1.6.0_22
> Java vendor:
>    Sun Microsystems Inc.
> OS name:
>    Windows 7
> OS architecture:
>    x86
> OS version:
>    6.1
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+260968395190
>



-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+260968395190