dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Hi Jason

On 29 December 2010 11:36, Jason Pickering <jason.p.pickering@xxxxxxxxx> wrote:
> Well, of course, as was the case with my previous mail, I saw my
> mistake immediately after I sent the mail, which was an incorrect
> attribute in the dataValue block  (periodid instead of period).

I was going to say after reading your previous mail that an invalid
attribute name tends to result in this error about number format.

>After
> several more tedious hours of looking at XML, I have something that
> works, and plan to share my experience with the group once I am done.

I also saw your blueprint about validation ... I think many of the
common errors might be best caught with schematron rules rather than
traditional relaxng ot xml schema.   Having also mispent many glorious
hours going through the sort of process you have been going through I
agree that some validations would help.

> This conversion of legacy data is always something that needs to be
> done, but up until now, it has been a process that we have not spent
> so much time documenting, especially via the route from a legacy
> database to DXF.
>
> A few more impressions  and comments as well.
>
> 1) Periods...DHIS2 will happily import periods which are actually not
> quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
> 1998. It would be a good to put validation on this.  Of course, the
> dates should be valid to begin with, but perhaps we cannot always
> trust legacy data.

I have never been too happy about monthly, weekly, quarterly, annual
etc periods being represented by start and end date.  Of course for
uniformity of storage they could and perhaps should be persisted like
that, but in the api and dxf it might be better to construct them from
1997-01, 1197-W1, 1997-Q1, 1997 etc.  Specifying start-end dates for
months and the like is really looking for validation errors - which
means of course we should test them :-)  Period validation blueprint
required.

>
> I made another mistake as well, and had an end date of "1997-09-31"
> which is not a valid date. I kept getting an NPE, but could not figure
> out what the heck was causing it. The date itself was actually
> invalid, so the enddate field ended up being transformed to NULL,
> which violates the NOT NULL constraint on the enddate in periods, and
> thus the NPE.
>
>  A more informative error message would have been helpful. In general,
> trying to untangle the error messages that were provided during the
> workup to a functional XML file resulted in a small migraine.

Some schematron rules would help here.

>
> 2) Users...When considering how to import a big block of data into an
> empty database, I could not really figure out how to specify a user,
> which is used in the data value attribute "storedBy"
>
> <dataValue dataElement="2" period="37" source="6795" value="17"
> storedBy="14" timeStamp="" comment="PATH Harvard Dataset"
> categoryOptionCombo="11"></dataValue>

I am not sure we should allow storedBy as an attribute at all.  There
isn't obvious use case to import/export users.  At least not in the
common data exchange scenarios.  I think the most sensible approach
would be to remove the attribute (or at least make it optional, which
I think it actually is but the schema may not reflect this) and set
storedBy to the user performing the import.

>
> Looking at the DXF XSD it is pretty clear...
>
>  <xs:attribute name="storedBy" type="xs:integer"/>
>  that the ID of the user has to be provided, but there does not appear
> to be a way to specify a user actually anywhere else in the XML. I
> know in DHIS 1.4, users can be transmitted from one DB to the other. I
> was not really sure how to handle this, so I had DHIS2 create the DB
> first, and then I had a peek in the users table to see what the
> "admin" user's ID actually was. Again..a bit troublesome if you want
> to do a bulk import of data from XML, without having to mess around
> with the DB.

The dxf 1 schema needs a bit of tidying up to reflect some of the
minor changes over the past few months.  I've not given it the
attention it deserves because I am still hoping to sift all these
learnings into a new look dxf 2 one of these fine days.  Perhaps 2011
will be it ... I know Jo has also been rumbling in this regard.

Cheers
Bob

>
>
> I plan on adding a bit of a case study to the documentation once I am
> done, and will inform the devs once I have had time to do this.
>
> Regards,
> Jason
>
>
>
>
> On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering
> <jason.p.pickering@xxxxxxxxx> wrote:
>> Hi again,
>> I had sent a message to the group earlier regarding importation of
>> data from a legacy source by DXF. Previously, I have used direct
>> inserts into the DHIS2 database, but because of the nature of this
>> data source, I want to try and use DXF. I am having problems however,
>> and could use some pointers.
>>
>> The DXF file is here (382kB)
>>
>> http://dl.dropbox.com/u/11802917/test_dxf_export8.7z
>>
>>
>> Basically, everything runs smoothly, up until the last stage when the
>> data values are to be imported.. When I get the error message posted
>> at the end of the mail.
>>
>> If I import the same DXF without the <dataValues> block, it imports
>> just fine. I really cannot decipher the log message however to
>> determine what the exact problem is. Any help appreciated.
>>
>> Thanks,
>> Jason
>>
>>
>>
>>
>> * INFO  08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
>> Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
>> _00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
>> * INFO  08:08:50,782 Internal process started (AbstractStatementInternalProcess.
>> java [Thread-32])
>> * INFO  08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
>> * INFO  08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
>> ead-32])
>> * INFO  08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
>> * WARN  08:08:50,836 Resource Leaking:  Could not remove uploaded file 'C:\Users
>> \jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
>> d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
>> * INFO  08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
>> ead-32])
>> * INFO  08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
>> 2])
>> * INFO  08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
>> ad-32])
>> * INFO  08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
>>  [Thread-32])
>> * INFO  08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
>> java [Thread-32])
>> * INFO  08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
>> ava [Thread-32])
>> * INFO  08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
>> * INFO  08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
>> * INFO  08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
>> [Thread-32])
>> * INFO  08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
>> 32])
>> * INFO  08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
>> Thread-32])
>> * INFO  08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
>> ad-32])
>> * INFO  08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
>> a [Thread-32])
>> * INFO  08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
>> 32])
>> * ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
>> ava [Thread-32])
>> java.lang.NumberFormatException: null
>>        at java.lang.Integer.parseInt(Integer.java:417)
>>        at java.lang.Integer.parseInt(Integer.java:499)
>>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
>> ValueConverter.java:195)
>>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
>> (DefaultConverterInvoker.java:45)
>>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
>> on(AopUtils.java:309)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
>> inpoint(ReflectiveMethodInvocation.java:183)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>> ReflectiveMethodInvocation.java:150)
>>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
>> voke(TransactionInterceptor.java:110)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>> ReflectiveMethodInvocation.java:172)
>>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
>> cAopProxy.java:202)
>>        at $Proxy49.invokeRead(Unknown Source)
>>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
>> er.java:968)
>>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
>> ortService.java:185)
>>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
>> portInternalProcess.java:89)
>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>> (AbstractStatementInternalProcess.java:91)
>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>> (AbstractStatementInternalProcess.java:40)
>>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
>> 25)
>>        at java.lang.Thread.run(Thread.java:662)
>> java.lang.NumberFormatException: null
>>        at java.lang.Integer.parseInt(Integer.java:417)
>>        at java.lang.Integer.parseInt(Integer.java:499)
>>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
>> ValueConverter.java:195)
>>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
>> (DefaultConverterInvoker.java:45)
>>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
>> on(AopUtils.java:309)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
>> inpoint(ReflectiveMethodInvocation.java:183)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>> ReflectiveMethodInvocation.java:150)
>>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
>> voke(TransactionInterceptor.java:110)
>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>> ReflectiveMethodInvocation.java:172)
>>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
>> cAopProxy.java:202)
>>        at $Proxy49.invokeRead(Unknown Source)
>>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
>> er.java:968)
>>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
>> ortService.java:185)
>>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
>> portInternalProcess.java:89)
>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>> (AbstractStatementInternalProcess.java:91)
>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>> (AbstractStatementInternalProcess.java:40)
>>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
>> 25)
>>        at java.lang.Thread.run(Thread.java:662)
>> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
>> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
>> ewForm.vm [line 92, column 157]
>> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
>> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
>> ewForm.vm [line 93, column 161]
>>
>>
>>
>> Current user:
>>    admin
>> Version:
>>    2.0.5
>> Build revision:
>>    2095
>> Build date:
>>    2010-11-25 10:47
>> User agent:
>>    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
>> Gecko/20101203 Firefox/3.6.13
>> External configuration directory:
>>    C:\Program Files\DHIS2-2.0.5\conf
>> Environment variable:
>>    DHIS2_HOME
>> Database type:
>>    PostgreSQL
>> Database name:
>>    harvard
>> Database user:
>>    postgres
>> Java Opts:
>> Java version:
>>    1.6.0_22
>> Java vendor:
>>    Sun Microsystems Inc.
>> OS name:
>>    Windows 7
>> OS architecture:
>>    x86
>> OS version:
>>    6.1
>> --
>> Jason P. Pickering
>> email: jason.p.pickering@xxxxxxxxx
>> tel:+260968395190
>>
>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+260968395190
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>