dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Bob,
Completely agree that the schematron rules would be a good idea. This
is also somewhat related to my blueprint on attribute level validation
(https://blueprints.launchpad.net/dhis2/+spec/regex-validation) but
not exactly.  I must confess I have not done much with schematrons,
and do not really understand their full potential. However, there is
clearly a need for rules both within an attribute itself, as well as
the relationships between them. If a schematron could do this, or part
of it, it would certainly be helpful. The biggest problem I see at the
moment for people aspiring to import data via DXF or other formats is
the extraordinarily arcane error messages that leaves one either
peeking at the source code to attempt to decipher what the problem
might be or desiring to open a beer instead. If these could somehow be
improved, it sure would make this process better. In general, the
entire import process seems rather fragile (my experience with DHIS
1.4 imports as well). If anything is wrong, the whole process bombs
with a bizarre message (which when using DHIS Live, is not even clear
where the log actually is???)

Just a short rebuttal on the importation of users. I actually would
consider the import of users to be quite important. You are correct of
course (as seen in the database) that the storedBy attribute is not
required. However, it is useful to see who has done what, and who
imported the data. It is especially useful in the context of pulling
data in from remote sources, where the contact details of the person
who has transmitted the file is present, so that if follow up is
necessary, it is easily apparent who to contact. I would argue for
more info in this regard, a la DHIS 1.4. Of course, there are security
issues here that need to be sorted out in the context of DHIS2. Just
because a user would get imported into the DB, does not mean they
would have the right to do anything. Sounds like another blueprint,
but I will not bother unless there is a need beyond my own wish list.

Regards,
Jason




On Wed, Dec 29, 2010 at 4:20 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
> Hi Jason
>
> On 29 December 2010 11:36, Jason Pickering <jason.p.pickering@xxxxxxxxx> wrote:
>> Well, of course, as was the case with my previous mail, I saw my
>> mistake immediately after I sent the mail, which was an incorrect
>> attribute in the dataValue block  (periodid instead of period).
>
> I was going to say after reading your previous mail that an invalid
> attribute name tends to result in this error about number format.
>
>>After
>> several more tedious hours of looking at XML, I have something that
>> works, and plan to share my experience with the group once I am done.
>
> I also saw your blueprint about validation ... I think many of the
> common errors might be best caught with schematron rules rather than
> traditional relaxng ot xml schema.   Having also mispent many glorious
> hours going through the sort of process you have been going through I
> agree that some validations would help.
>
>> This conversion of legacy data is always something that needs to be
>> done, but up until now, it has been a process that we have not spent
>> so much time documenting, especially via the route from a legacy
>> database to DXF.
>>
>> A few more impressions  and comments as well.
>>
>> 1) Periods...DHIS2 will happily import periods which are actually not
>> quarters, such as with a startdate of Jan 1 1997 and end date of Jan 1
>> 1998. It would be a good to put validation on this.  Of course, the
>> dates should be valid to begin with, but perhaps we cannot always
>> trust legacy data.
>
> I have never been too happy about monthly, weekly, quarterly, annual
> etc periods being represented by start and end date.  Of course for
> uniformity of storage they could and perhaps should be persisted like
> that, but in the api and dxf it might be better to construct them from
> 1997-01, 1197-W1, 1997-Q1, 1997 etc.  Specifying start-end dates for
> months and the like is really looking for validation errors - which
> means of course we should test them :-)  Period validation blueprint
> required.
>
>>
>> I made another mistake as well, and had an end date of "1997-09-31"
>> which is not a valid date. I kept getting an NPE, but could not figure
>> out what the heck was causing it. The date itself was actually
>> invalid, so the enddate field ended up being transformed to NULL,
>> which violates the NOT NULL constraint on the enddate in periods, and
>> thus the NPE.
>>
>>  A more informative error message would have been helpful. In general,
>> trying to untangle the error messages that were provided during the
>> workup to a functional XML file resulted in a small migraine.
>
> Some schematron rules would help here.
>
>>
>> 2) Users...When considering how to import a big block of data into an
>> empty database, I could not really figure out how to specify a user,
>> which is used in the data value attribute "storedBy"
>>
>> <dataValue dataElement="2" period="37" source="6795" value="17"
>> storedBy="14" timeStamp="" comment="PATH Harvard Dataset"
>> categoryOptionCombo="11"></dataValue>
>
> I am not sure we should allow storedBy as an attribute at all.  There
> isn't obvious use case to import/export users.  At least not in the
> common data exchange scenarios.  I think the most sensible approach
> would be to remove the attribute (or at least make it optional, which
> I think it actually is but the schema may not reflect this) and set
> storedBy to the user performing the import.
>
>>
>> Looking at the DXF XSD it is pretty clear...
>>
>>  <xs:attribute name="storedBy" type="xs:integer"/>
>>  that the ID of the user has to be provided, but there does not appear
>> to be a way to specify a user actually anywhere else in the XML. I
>> know in DHIS 1.4, users can be transmitted from one DB to the other. I
>> was not really sure how to handle this, so I had DHIS2 create the DB
>> first, and then I had a peek in the users table to see what the
>> "admin" user's ID actually was. Again..a bit troublesome if you want
>> to do a bulk import of data from XML, without having to mess around
>> with the DB.
>
> The dxf 1 schema needs a bit of tidying up to reflect some of the
> minor changes over the past few months.  I've not given it the
> attention it deserves because I am still hoping to sift all these
> learnings into a new look dxf 2 one of these fine days.  Perhaps 2011
> will be it ... I know Jo has also been rumbling in this regard.
>
> Cheers
> Bob
>
>>
>>
>> I plan on adding a bit of a case study to the documentation once I am
>> done, and will inform the devs once I have had time to do this.
>>
>> Regards,
>> Jason
>>
>>
>>
>>
>> On Wed, Dec 29, 2010 at 8:41 AM, Jason Pickering
>> <jason.p.pickering@xxxxxxxxx> wrote:
>>> Hi again,
>>> I had sent a message to the group earlier regarding importation of
>>> data from a legacy source by DXF. Previously, I have used direct
>>> inserts into the DHIS2 database, but because of the nature of this
>>> data source, I want to try and use DXF. I am having problems however,
>>> and could use some pointers.
>>>
>>> The DXF file is here (382kB)
>>>
>>> http://dl.dropbox.com/u/11802917/test_dxf_export8.7z
>>>
>>>
>>> Basically, everything runs smoothly, up until the last stage when the
>>> data values are to be imported.. When I get the error message posted
>>> at the end of the mail.
>>>
>>> If I import the same DXF without the <dataValues> block, it imports
>>> just fine. I really cannot decipher the log message however to
>>> determine what the exact problem is. Any help appreciated.
>>>
>>> Thanks,
>>> Jason
>>>
>>>
>>>
>>>
>>> * INFO  08:08:50,741 Content-type: text/plain, filename: C:\Users\jason\AppData\
>>> Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12d30795f48__8000
>>> _00000026.tmp (ImportAction.java [2779391@qtp-1027049-15])
>>> * INFO  08:08:50,782 Internal process started (AbstractStatementInternalProcess.
>>> java [Thread-32])
>>> * INFO  08:08:50,782 Importing stream (DefaultImportService.java [Thread-32])
>>> * INFO  08:08:50,784 Importing DXF native stream (DefaultImportService.java [Thr
>>> ead-32])
>>> * INFO  08:08:50,791 Deleted previewed objects (DXFConverter.java [Thread-32])
>>> * WARN  08:08:50,836 Resource Leaking:  Could not remove uploaded file 'C:\Users
>>> \jason\AppData\Local\Temp\Jetty_0_0_0_0_8888_dhis____.al3jm9\upload__59f6bd88_12
>>> d30795f48__8000_00000026.tmp'. (CommonsLogger.java [2779391@qtp-1027049-15])
>>> * INFO  08:08:50,857 Imported DataElementCategoryOptions (DXFConverter.java [Thr
>>> ead-32])
>>> * INFO  08:08:50,907 Imported DataElementCategories (DXFConverter.java [Thread-3
>>> 2])
>>> * INFO  08:08:50,956 Imported DataElementCategoryCombos (DXFConverter.java [Thre
>>> ad-32])
>>> * INFO  08:08:51,054 Imported DataElementCategoryOptionCombos (DXFConverter.java
>>>  [Thread-32])
>>> * INFO  08:08:51,195 Imported CategoryCategoryOption associations (DXFConverter.
>>> java [Thread-32])
>>> * INFO  08:08:51,308 Imported CategoryComboCategory associations (DXFConverter.j
>>> ava [Thread-32])
>>> * INFO  08:08:52,568 Imported DataElements (DXFConverter.java [Thread-32])
>>> * INFO  08:22:23,426 Imported OrganisationUnits (DXFConverter.java [Thread-32])
>>> * INFO  08:22:40,145 Imported OrganisationUnit relationships (DXFConverter.java
>>> [Thread-32])
>>> * INFO  08:22:40,466 Imported OrganisationUnitGroups (DXFConverter.java [Thread-
>>> 32])
>>> * INFO  08:23:54,931 Imported OrganisationUnitGroup members (DXFConverter.java [
>>> Thread-32])
>>> * INFO  08:23:55,005 Imported OrganisationUnitGroupSets (DXFConverter.java [Thre
>>> ad-32])
>>> * INFO  08:23:55,175 Imported OrganisationUnitGroupSet members (DXFConverter.jav
>>> a [Thread-32])
>>> * INFO  08:23:55,255 Imported OrganisationUnitLevels (DXFConverter.java [Thread-
>>> 32])
>>> * ERROR 08:23:55,419 Internal process failed (AbstractStatementInternalProcess.j
>>> ava [Thread-32])
>>> java.lang.NumberFormatException: null
>>>        at java.lang.Integer.parseInt(Integer.java:417)
>>>        at java.lang.Integer.parseInt(Integer.java:499)
>>>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
>>> ValueConverter.java:195)
>>>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
>>> (DefaultConverterInvoker.java:45)
>>>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> sorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
>>> on(AopUtils.java:309)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
>>> inpoint(ReflectiveMethodInvocation.java:183)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>>> ReflectiveMethodInvocation.java:150)
>>>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
>>> voke(TransactionInterceptor.java:110)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>>> ReflectiveMethodInvocation.java:172)
>>>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
>>> cAopProxy.java:202)
>>>        at $Proxy49.invokeRead(Unknown Source)
>>>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
>>> er.java:968)
>>>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
>>> ortService.java:185)
>>>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
>>> portInternalProcess.java:89)
>>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>>> (AbstractStatementInternalProcess.java:91)
>>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>>> (AbstractStatementInternalProcess.java:40)
>>>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
>>> 25)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> java.lang.NumberFormatException: null
>>>        at java.lang.Integer.parseInt(Integer.java:417)
>>>        at java.lang.Integer.parseInt(Integer.java:499)
>>>        at org.hisp.dhis.importexport.dxf.converter.DataValueConverter.read(Data
>>> ValueConverter.java:195)
>>>        at org.hisp.dhis.importexport.invoker.DefaultConverterInvoker.invokeRead
>>> (DefaultConverterInvoker.java:45)
>>>        at sun.reflect.GeneratedMethodAccessor163.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> sorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflecti
>>> on(AopUtils.java:309)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJo
>>> inpoint(ReflectiveMethodInvocation.java:183)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>>> ReflectiveMethodInvocation.java:150)
>>>        at org.springframework.transaction.interceptor.TransactionInterceptor.in
>>> voke(TransactionInterceptor.java:110)
>>>        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>>> ReflectiveMethodInvocation.java:172)
>>>        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynami
>>> cAopProxy.java:202)
>>>        at $Proxy49.invokeRead(Unknown Source)
>>>        at org.hisp.dhis.importexport.dxf.converter.DXFConverter.read(DXFConvert
>>> er.java:968)
>>>        at org.hisp.dhis.importexport.DefaultImportService.importData(DefaultImp
>>> ortService.java:185)
>>>        at org.hisp.dhis.importexport.ImportInternalProcess.executeStatements(Im
>>> portInternalProcess.java:89)
>>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>>> (AbstractStatementInternalProcess.java:91)
>>>        at org.hisp.dhis.system.process.AbstractStatementInternalProcess.execute
>>> (AbstractStatementInternalProcess.java:40)
>>>        at org.amplecode.cave.process.ProcessExecutor.run(ProcessExecutor.java:1
>>> 25)
>>>        at java.lang.Thread.run(Thread.java:662)
>>> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
>>> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>>>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
>>> ewForm.vm [line 92, column 157]
>>> Dec 29, 2010 8:23:57 AM org.apache.velocity.runtime.log.JdkLogChute log
>>> SEVERE: Left side ($status) of '==' operation has null value. If a reference, it
>>>  may not be in the context. Operation not possible. /dhis-web-importexport/previ
>>> ewForm.vm [line 93, column 161]
>>>
>>>
>>>
>>> Current user:
>>>    admin
>>> Version:
>>>    2.0.5
>>> Build revision:
>>>    2095
>>> Build date:
>>>    2010-11-25 10:47
>>> User agent:
>>>    Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13)
>>> Gecko/20101203 Firefox/3.6.13
>>> External configuration directory:
>>>    C:\Program Files\DHIS2-2.0.5\conf
>>> Environment variable:
>>>    DHIS2_HOME
>>> Database type:
>>>    PostgreSQL
>>> Database name:
>>>    harvard
>>> Database user:
>>>    postgres
>>> Java Opts:
>>> Java version:
>>>    1.6.0_22
>>> Java vendor:
>>>    Sun Microsystems Inc.
>>> OS name:
>>>    Windows 7
>>> OS architecture:
>>>    x86
>>> OS version:
>>>    6.1
>>> --
>>> Jason P. Pickering
>>> email: jason.p.pickering@xxxxxxxxx
>>> tel:+260968395190
>>>
>>
>>
>>
>> --
>> Jason P. Pickering
>> email: jason.p.pickering@xxxxxxxxx
>> tel:+260968395190
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>



-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+260968395190