dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

Halvdan,

Firstly, I have just done a test import of the za_CTRY.gml file using the
latest 2.20 trunk - and IT WORKS. I just hope using 2.20 for importing
those files and then reverting back to our customised 2.19 won't mess up
anything (NOTE: for the time being, we have to use a customised version of
2.19 due to the inclusion of 1.4 conversion code, the new database
Synchronisation Manager enabling synching between many different DHIS2
instances, and the small piece of code enabling vertical cursor movement
when capturing daily data in the custom "pivot" data entry form).

Secondly, I do get some warning messages when importing - not sure if they
are relevant/expected (see tomcat log extract below).

I really appreciate the effort you've put in on this, especially during
those hot mid-summer days in Oslo :-)

Regards
Calle

Tomcat log:
04-Jul-2015 01:33:26.270 INFO [main]
org.apache.catalina.startup.Catalina.start Server startup in 78200 ms
* WARN  2015-07-04 01:33:39,997 Authentication event
AuthenticationSuccessEvent: Calle_Hedberg; details:
org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8:
RemoteIpAddress: 0
0-exec-5])
* WARN  2015-07-04 01:33:39,998 Authentication event
SessionFixationProtectionEvent: Calle_Hedberg; details:
org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8:
RemoteIpAddres
-8080-exec-5])
* WARN  2015-07-04 01:33:39,999 Authentication event
InteractiveAuthenticationSuccessEvent: Calle_Hedberg; details:
org.springframework.security.web.authentication.WebAuthenticationDetails@fffbcba8:
RemoteI
ttp-nio-8080-exec-5])
Warning:  org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser: Property '
http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
recognized.
Compiler warnings:
  WARNING:  'org.apache.xerces.jaxp.SAXParserImpl: Property '
http://javax.xml.XMLConstants/property/accessExternalDTD' is not
recognized.'
Warning:  org.apache.xerces.parsers.SAXParser: Feature '
http://javax.xml.XMLConstants/feature/secure-processing' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property '
http://javax.xml.XMLConstants/property/accessExternalDTD' is not recognized.
Warning:  org.apache.xerces.parsers.SAXParser: Property '
http://www.oracle.com/xml/jaxp/properties/entityExpansionLimit' is not
recognized.
* INFO  2015-07-04 01:34:39,988 User 'Calle_Hedberg' started import at Sat
Jul 04 01:34:39 CAT 2015 (DefaultImportService.java [taskScheduler-1])
* INFO  2015-07-04 01:34:39,988 [Level: INFO, category: METADATA_IMPORT,
time: Sat Jul 04 01:34:39 CAT 2015, message: Importing meta-data]
(InMemoryNotifier.java [taskScheduler-1])
* INFO  2015-07-04 01:34:39,998 Building object-bridge maps (preheatCache:
true). (DefaultObjectBridge.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,235 Building object-bridge maps took 3.24
seconds. (DefaultObjectBridge.java [taskScheduler-1])
* WARN  2015-07-04 01:34:43,235 Can not find getter for 'oAuth2Clients'.
(DefaultImportService.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,235 [Level: INFO, category: METADATA_IMPORT,
time: Sat Jul 04 01:34:43 CAT 2015, message: Importing 1 OrganisationUnits]
(InMemoryNotifier.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,245 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,245 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,245 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* WARN  2015-07-04 01:34:43,265 Can not find getter for
'programRuleVariables'. (DefaultImportService.java [taskScheduler-1])
* WARN  2015-07-04 01:34:43,265 Can not find getter for 'programRules'.
(DefaultImportService.java [taskScheduler-1])
* WARN  2015-07-04 01:34:43,265 Can not find getter for
'programRuleActions'. (DefaultImportService.java [taskScheduler-1])
* INFO  2015-07-04 01:34:43,275 [Level: INFO, category: METADATA_IMPORT,
time: Sat Jul 04 01:34:43 CAT 2015, message: Import done. Completed in 3.28
seconds.] (InMemoryNotifier.java [taskScheduler-1])


On 4 July 2015 at 01:12, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:

> Oh, and for the record: the stack overflow was caused by cirular calls to
> toString between AttributeValue and Attribute, respectively.
>
> 2015-07-04 0:56 GMT+02:00 Halvdan Grelland <halvdanhg@xxxxxxxxx>:
>
>> Yes, that is the plan, ultimately.
>> It's a pretty major change though, making it non-trivial to port the fix
>> (it's more of a refactor). I'll look into it after the weekend (or during
>> the weekend if the sun stops shining in Oslo).
>>
>> I did make (and backport) some changes which fix the stack overflow
>> issue, however. In practice this means that GML import should work for you
>> on the latest 2.19, but any attribute values will unfortunately be deleted
>> in the process (!), or you might even experience the importer halting if
>> there are attribute values for the orgunits. Not ideal, in other words, and
>> something you should be aware of.
>>
>> Halvdan
>>
>> 2015-07-03 18:55 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>
>>> Halvdan,
>>>
>>> I see you have committed a significant fix to 2.20 trunk - which I hope
>>> will also fix my GML import issue.
>>>
>>> Will you port this fix to 2.19 also, please?
>>>
>>> Regards
>>> Calle
>>>
>>> On 1 July 2015 at 15:01, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:
>>>
>>>> Ok, so I've looked further into this. The only issue I am able to
>>>> consistently reproduce happens only in the GML import process. The stack
>>>> trace can be found here
>>>> <https://gist.github.com/grelland/45acadfbe475422889ec>
>>>> Note that this might be unrelated to the looping/recursion issue.,
>>>>
>>>> This does not happen if I pre-stage the meta data payload (ie write it
>>>> to a file and import using the metadata importer). My suspicion that the
>>>> conflict occurs because of the clear/re-add of attributevalues in the
>>>> orgunit.mergeWith() method (which I'm using to merge geodata updates into
>>>> the orgunit), which happens within a transactional scope. I'm gonna have to
>>>> experiment further to find a solution. Any suggestions are much appreciated.
>>>>
>>>>
>>>> 2015-07-01 14:39 GMT+02:00 Bob Jolliffe <bobjolliffe@xxxxxxxxx>:
>>>>
>>>>> Morten this is a long shot, but do you think there is any possibility
>>>>> the removal of xerces exclusions in 2.19 might be causing these
>>>>> strange unpredictable problems?  Weird StackOverflowErrors and Xerces
>>>>> problems seem to go hand in hand - google the two terms :-(  I guess
>>>>> the nub is that it becomes unpredictable which xercesimpl the
>>>>> classloader uses.
>>>>>
>>>>> The original reason for adding the xercesImpl exclusions was that it
>>>>> seemed to make the java xml parsing more stable.  I am not sure how it
>>>>> easy it would be test this as you'd have to build without openid4java
>>>>> .. which is the library which caused us to remove the exclusions.
>>>>>
>>>>> On 1 July 2015 at 10:16, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>>>>> wrote:
>>>>> >
>>>>> > On Wed, Jul 1, 2015 at 4:12 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>>>>> wrote:
>>>>> >>
>>>>> >> Having said that, this stack overflow business might carry some
>>>>> clue.
>>>>> >> @Morten you mention elsewhere in this thread  "It seems to be
>>>>> looping in
>>>>> >> Attribute.toString()".  Are you sure it is looping or is there
>>>>> something
>>>>> >> recursive going on here?  (note that xslt with very large nodesets
>>>>> could
>>>>> >> indeed also cause recursion problems)
>>>>> >
>>>>> >
>>>>> > Well, it reminded me of an earlier issue we had, but that have since
>>>>> been
>>>>> > fixed.
>>>>> >
>>>>> > As I can import the sample XML file from Halvdan without any issues
>>>>> in 2.19
>>>>> > / 2.20 (but he could not...) I might not be reproducing it properly,
>>>>> so I'm
>>>>> > waiting for input from him.
>>>>> >
>>>>> > --
>>>>> > Morten
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *******************************************
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472
>>>
>>> Cell: +27-82-853-5352
>>>
>>> Iridium SatPhone: +8816-315-19274
>>>
>>> Email: calle.hedberg@xxxxxxxxx
>>>
>>> Skype: calle_hedberg
>>>
>>> *******************************************
>>>
>>>
>>
>


-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************