dhis2-devs team mailing list archive

[Morten Olav Hansen <mortenoh@xxxxxxxxx>]

Hi guys

I've been trying to reproduce your bug this morning, but I'm not able to. I
restored the db, and imported the xml without any issues, I tested on both
2.19 and 2.20, so I will need more information to reproduce.

This was the command I used to import:
curl -X POST -d @dxfstream.xml -H "Content-Type: application/xml" -v -u
admin:district http://localhost:8080/api/metadata

--
Morten

On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland <halvdanhg@xxxxxxxxx>
wrote:

> So. The stack size thing was just a qualified guess derived from the stack
> trace. It's pretty obviously not the cause. I'll wait on feedback from
> Morten before going any further on this.
> On Jun 27, 2015 6:50 PM, "Calle Hedberg" <calle.hedberg@xxxxxxxxx> wrote:
>
>> Halvdan,
>>
>> I understand from your last email that the root cause of the problem is a
>> 2.19 software issue - the meta data importer in 2.19 is not able to handle
>> OrgUnit attribute values when present. That makes sense, and it is likely
>> that the root cause of this bug / design flaw is a 2.19 change (since the
>> same import worked fine in 2.18). Morten and/or Lars will hopefully know
>> what change(s) caused this.
>>
>> With regard to JVM running out of heap space, and in relation to the
>> (outdated - very!) chapter 8.2.8 you referenced:
>>
>> Firstly, with regard to the 150+ instances we are hosting on our data
>> centre servers, we do have some constraints at the moment: Those servers
>> have 32GB RAM only, and most instances are in reality running on 1-3GB.
>> That will only change from next week, when we start moving to new servers
>> with (initially) 128GB RAM. That should on average provide 6-8GB per
>> instance (varying between 2 and 32GB).
>>
>> Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64
>> bits) memory allocations on my Windows notebook (32GB RAM) has been a
>> nightmare - the people writing up java and tomcat documentation are
>> useless, they seem unable to give any quantified advice at all where any
>> normal person can find it. (In comparison, chapter 8.2.8 although outdated
>> is super-clear). I am currently using the following setting for Java:
>>
>> JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m
>>
>> For some reason I/we do not understand, it seems Tomcat 8.23 does not use
>> CATALINA_OPTS settings, so I ended up using the "Configure Tomcat" and then
>> specify the following under the "Java" tab:
>> Java options:
>> -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0
>> -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0
>> -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat
>> 8.0\endorsed
>> -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat
>> 8.0\temp
>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>> -Djava.util.logging.config.file=C:\Program Files\Apache Software
>> Foundation\Tomcat 8.0\conf\logging.properties
>> -XX:MaxPermSize=4096m
>>
>> (NOTE: the -XX value here is different from what I currently have in the
>> JAVA_OPTS environmental variable - I cannot find any info on which of them
>> takes precedence
>>
>> Also under that same "Java" tab on "Configure Tomcat":
>> Initial memory pool: 4096MB
>> Maximum memory pool: 8192MB
>> Thread stack size: 4096KB (4MB)
>>
>> Based on your initial advice to increase the stack size, I tried to
>> increase it to 8MB (GML import still crashing), 64MB (GML import still
>> crashing), and 132MB (GML import went completely haywire). Which is
>> understandable, given your subsequent finding that the root cause of the
>> problem was not related to heap/stack settings.
>>
>> So I've set the thread stack size back to 4096KB (4MB) - even if I don't
>> know whether that's correct or not (seems high to my layman eyes, but maybe
>> DHIS2 require it?)
>>
>> If anybody actually KNOWS what memory/stack sizes that I optimal for a
>> 32GB Windows machine, I would appreciate the advice...
>>
>> Regards
>> Calle
>>
>> On 26 June 2015 at 20:26, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:
>>
>>> Ok, I could have a look, but am preoccupied at the moment. Will get back
>>> to you.
>>>
>>> In the meantime, it seems that you're facing a stack overflow error,
>>> which could be caused by the JVM running out of heap space. Could you
>>> please check the allocated heap for tomcat? If it is not being set (in the
>>> CATALINA_OPTS env variable), please set it to something quite large and try
>>> again.
>>>
>>> Under chapter 8.2.8 at the following link is a brief guide
>>> https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html
>>>
>>> Please let me know how this works out for you. :)
>>>
>>> 2015-06-26 11:44 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>
>>>> Halvdan,
>>>>
>>>> Here's a stripped down version of the Free State instance (no records
>>>> in the data value table). Restore it as za_fs_s (psql -U dhis
>>>> -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)
>>>>
>>>> You have got that single-record country GML file already.
>>>>
>>>> I would appreciate any help in figuring out what's happening. Just note
>>>> that I'm using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.
>>>>
>>>> Best regards
>>>> Calle
>>>>
>>>> On 26 June 2015 at 11:17, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I ran it again (revision 19247) and pulled out the first part of the
>>>>> log error:
>>>>>
>>>>> * INFO  2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import at
>>>>> Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:18,348 [Level: INFO, category:
>>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing
>>>>> meta-data] (InMemoryNotifier.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:18,348 Building object-bridge maps
>>>>> (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:21,141 Building object-bridge maps took 2.79
>>>>> seconds. (DefaultObjectBridge.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:21,141 [Level: INFO, category:
>>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1
>>>>> OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with object
>>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with object
>>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with object
>>>>> of type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
>>>>> * ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled
>>>>> task. (TaskUtils.java [taskScheduler-1])
>>>>> java.lang.StackOverflowError
>>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
>>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
>>>>> at java.util.regex.Pattern$BitClass.<init>(Unknown Source)
>>>>> at java.util.regex.Pattern.clazz(Unknown Source)
>>>>> at java.util.regex.Pattern.sequence(Unknown Source)
>>>>> at java.util.regex.Pattern.expr(Unknown Source)
>>>>> at java.util.regex.Pattern.compile(Unknown Source)
>>>>> at java.util.regex.Pattern.<init>(Unknown Source)
>>>>> at java.util.regex.Pattern.compile(Unknown Source)
>>>>> at java.lang.String.replaceAll(Unknown Source)
>>>>> at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)
>>>>> at
>>>>> com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)
>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)
>>>>> at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>> at
>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)
>>>>> at
>>>>> org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>>> at
>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Regards
>>>>> Calle
>>>>>
>>>>> On 26 June 2015 at 10:12, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Firstly, yes we are using a custom revision - although it is SUPPOSED
>>>>>> to be based on a 2.19 version from a few days ago. I will check that.
>>>>>>
>>>>>> BUT I have tried the import using revision 19247 too, and the
>>>>>> problem/behaviour is exactly the same - it hangs. Whereas if I import into
>>>>>> an older version of the same provincial instance that still runs on 2.18,
>>>>>> it imports fine.
>>>>>>
>>>>>> The fact that the same GML file imports in the Sierra Leone demo - I
>>>>>> cannot explain that, and the tomcat log only shows that
>>>>>> Attribute.toString() looping...
>>>>>>
>>>>>> Can I create a stripped down version of the instance and upload it
>>>>>> for you to test?
>>>>>>
>>>>>> Regards
>>>>>> Calle
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 26 June 2015 at 04:09, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> It seems to be looping in Attribute.toString(), I remember this was
>>>>>>> an issue at some point.. but that was a long long time ago.. so maybe you
>>>>>>> need a revision bump
>>>>>>>
>>>>>>> --
>>>>>>> Morten
>>>>>>>
>>>>>>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <
>>>>>>> halvdanhg@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> I tried your example GML on both our demo servers (dev on 2.20 and
>>>>>>>> demo on 2.19, same rev as your server) and it works just fine.
>>>>>>>>
>>>>>>>> If you're experiencing it hanging and not finishing you might be
>>>>>>>> experiencing some kind of performance and/or memory issue. The coordinate
>>>>>>>> set is fairly large (though it works well on our demos, but does take some
>>>>>>>> time to finish). You can verify by trying yourself to create the orgunit
>>>>>>>> (with the exact name) and import the GML on the Sierra Leone demo.
>>>>>>>>
>>>>>>>> The fact that the dry run seems to "work" for you (as well as the
>>>>>>>> stack trace you provided) also points at the issue not residing in the GML
>>>>>>>> processing but rather in the actual import stage (the GML importer uses the
>>>>>>>> metadata importer as it's backend).
>>>>>>>>
>>>>>>>> Another thing: The instance you linked to is using a revision of 16783
>>>>>>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247).
>>>>>>>> Are you using a custom build? Does it contain customizations to the core in
>>>>>>>> any way? That would also be a possible source of issues, obviously.
>>>>>>>>
>>>>>>>> You could try deploying this war
>>>>>>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war
>>>>>>>> and replicate the issue using the za South Africa orgunit.
>>>>>>>>
>>>>>>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>>>>>
>>>>>>>>> Halvdan,
>>>>>>>>>
>>>>>>>>> The GML files I'm trying to import using 2.19 is the same GML
>>>>>>>>> files I've been using successfully for 2.17, 2.18 (so it's using NAME).
>>>>>>>>> Something in the import has changed:
>>>>>>>>>
>>>>>>>>> I have attached the small GML file I'm trying to import - it has
>>>>>>>>> the international boundary of SA, so there is only ONE GIS record (top node
>>>>>>>>> in all our OrgHierarchies). It imports with no problems in 2.18.
>>>>>>>>>
>>>>>>>>> In 2.19, if I choose YES for dry run, it will complete with usual
>>>>>>>>> import summary etc.
>>>>>>>>>
>>>>>>>>> If I choose NO for dry run, I get
>>>>>>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits  2015-06-25
>>>>>>>>> 22:18:11Importing meta-data
>>>>>>>>> and there it hangs. Nothing gets imported.
>>>>>>>>>
>>>>>>>>> The online instance you can try to import into is
>>>>>>>>> http://fs.dhis.dhmis.org/staging/
>>>>>>>>>
>>>>>>>>> When I try to import into a local instance, it seems to enter some
>>>>>>>>> kind of endless loop based on the tomcat log:
>>>>>>>>>   at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>> Source)
>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>>>>>>   at
>>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at java.util.AbstractCollection.toString(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>>>>>   at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>> Source)
>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>>>>>>   at
>>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at java.util.AbstractCollection.toString(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>   at
>>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>>>>>   at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>> Source)
>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>
>>>>>>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Calle,
>>>>>>>>>>
>>>>>>>>>> The commit you've replied to implements (but not fully, therefore
>>>>>>>>>> 'WIP', there's another commit following it pretty soon after) better error
>>>>>>>>>> reporting feedback to the user through the interface. I.e. not having to
>>>>>>>>>> see the stacktrace on the server to determine if and why something went
>>>>>>>>>> wrong. I believe you have been among the people requesting this too :)
>>>>>>>>>>
>>>>>>>>>> Also, as you're referencing, we've made some other changes to GML
>>>>>>>>>> import. In particular the importer will now match on uid, code or name, as
>>>>>>>>>> oppsed to the former mode of matching only on name. This is not reflected
>>>>>>>>>> in the UI as it is a backend enhancement only, and does not change the
>>>>>>>>>> import options at all. It IS, however, documented and ready for you to read
>>>>>>>>>> about. Short version: it matches on a prioritized list of uid -> code ->
>>>>>>>>>> name. Long version:
>>>>>>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html
>>>>>>>>>>
>>>>>>>>>> Be adviced that the commit you've referenced is towards
>>>>>>>>>> development trunk, and is not backported to 2.19, the match on multiple
>>>>>>>>>> identifiers, however, was introduced in 2.19 and should work in current
>>>>>>>>>> 2.19 stable. If you're having problems with it you can send me a full stack
>>>>>>>>>> trace and preferably the GML input data.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx
>>>>>>>>>> >:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I'm unable to import GML files using 2.19 (the same files import
>>>>>>>>>>> fine in 2.18) - the import simply hangs (endless rolling progress bar icon).
>>>>>>>>>>>
>>>>>>>>>>> Will the above bug-fix sort out that problem - I do not
>>>>>>>>>>> understand what the above actually means, or if the GML format has changed
>>>>>>>>>>> in some way.
>>>>>>>>>>>
>>>>>>>>>>> Note also that the final 2.19 release was supposed to include a
>>>>>>>>>>> more diversified GML import - but I cannot see WHERE to do that (I still
>>>>>>>>>>> see the same two buttons: choose file and import, either dry run or not)
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Calle
>>>>>>>>>>>
>>>>>>>>>>> On 5 June 2015 at 14:57, <noreply@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>> revno: 19296
>>>>>>>>>>>> committer: Halvdan Hoem Grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>>>>> branch nick: dhis2
>>>>>>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200
>>>>>>>>>>>> message:
>>>>>>>>>>>>   WIP proper error reporting on failed GML parsing during import
>>>>>>>>>>>> added:
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>> modified:
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>>
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> lp:dhis2
>>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>>>>>>>>>
>>>>>>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>>>>>>>>>> To unsubscribe from this branch go to
>>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>   2015-06-03 15:00:19 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>   2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -32,6 +32,8 @@
>>>>>>>>>>>>  import com.google.common.base.Strings;
>>>>>>>>>>>>  import com.google.common.collect.Iterators;
>>>>>>>>>>>>  import com.google.common.collect.Maps;
>>>>>>>>>>>> +import org.apache.commons.io.IOUtils;
>>>>>>>>>>>> +import org.hibernate.Hibernate;
>>>>>>>>>>>>  import org.hisp.dhis.common.IdentifiableObjectManager;
>>>>>>>>>>>>  import org.hisp.dhis.common.IdentifiableProperty;
>>>>>>>>>>>>  import org.hisp.dhis.common.MergeStrategy;
>>>>>>>>>>>> @@ -88,13 +90,91 @@
>>>>>>>>>>>>      // GmlImportService implementation
>>>>>>>>>>>>      //
>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> +    @Transactional
>>>>>>>>>>>> +    @Override
>>>>>>>>>>>> +    public GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>>>>>> inputStream )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        InputStream dxfStream = null;
>>>>>>>>>>>> +        MetaData metaData = null;
>>>>>>>>>>>> +
>>>>>>>>>>>> +        try
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            dxfStream = transformGml( inputStream );
>>>>>>>>>>>> +            metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +        catch ( IOException | TransformerException e )
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            return GmlPreProcessingResult.failure( e );
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +        finally
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            IOUtils.closeQuietly( dxfStream );
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>> +        Map<String, OrganisationUnit> uidMap  =
>>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>>>>>> +
>>>>>>>>>>>> +        matchAndFilterOnIdentifiers(
>>>>>>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedUidMap  =
>>>>>>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedCodeMap =
>>>>>>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedNameMap =
>>>>>>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        Iterator<OrganisationUnit> persistedIterator =
>>>>>>>>>>>> Iterators.concat( persistedUidMap.values().iterator(),
>>>>>>>>>>>> +            persistedCodeMap.values().iterator(),
>>>>>>>>>>>> persistedNameMap.values().iterator() );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        while ( persistedIterator.hasNext() )
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            OrganisationUnit persisted =
>>>>>>>>>>>> persistedIterator.next(), imported = null;
>>>>>>>>>>>> +
>>>>>>>>>>>> +            if ( !Strings.isNullOrEmpty( persisted.getUid() )
>>>>>>>>>>>> && uidMap.containsKey( persisted.getUid() ) )
>>>>>>>>>>>> +            {
>>>>>>>>>>>> +                imported = uidMap.get( persisted.getUid() );
>>>>>>>>>>>> +            }
>>>>>>>>>>>> +            else if ( !Strings.isNullOrEmpty(
>>>>>>>>>>>> persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
>>>>>>>>>>>> +            {
>>>>>>>>>>>> +                imported = codeMap.get( persisted.getCode() );
>>>>>>>>>>>> +            }
>>>>>>>>>>>> +            else if ( !Strings.isNullOrEmpty(
>>>>>>>>>>>> persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
>>>>>>>>>>>> +            {
>>>>>>>>>>>> +                imported = nameMap.get( persisted.getName() );
>>>>>>>>>>>> +            }
>>>>>>>>>>>> +
>>>>>>>>>>>> +            if ( imported == null || imported.getCoordinates()
>>>>>>>>>>>> == null || imported.getFeatureType() == null )
>>>>>>>>>>>> +            {
>>>>>>>>>>>> +                continue; // Failed to dereference a persisted
>>>>>>>>>>>> entity for this org unit or geo data incomplete/missing, therefore ignore
>>>>>>>>>>>> +            }
>>>>>>>>>>>> +
>>>>>>>>>>>> +            mergeNonGeoData( persisted, imported );
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>> +        return GmlPreProcessingResult.success( metaData );
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>>      @Override
>>>>>>>>>>>>      public MetaData fromGml( InputStream inputStream )
>>>>>>>>>>>>          throws IOException, TransformerException
>>>>>>>>>>>>      {
>>>>>>>>>>>> -        InputStream dxfStream = transformGml( inputStream );
>>>>>>>>>>>> -        MetaData metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>> -        dxfStream.close();
>>>>>>>>>>>> +        InputStream dxfStream;
>>>>>>>>>>>> +        MetaData metaData;
>>>>>>>>>>>> +
>>>>>>>>>>>> +        try
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            dxfStream = transformGml( inputStream );
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +        catch (Exception e)
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            dxfStream = null;
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>> +        if(dxfStream != null)
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>> +            dxfStream.close();
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +        else
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            return null;
>>>>>>>>>>>> +        }
>>>>>>>>>>>>
>>>>>>>>>>>>          Map<String, OrganisationUnit> uidMap  =
>>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>>>>>>
>>>>>>>>>>>> @@ -143,6 +223,7 @@
>>>>>>>>>>>>          importService.importMetaData( userUid, fromGml(
>>>>>>>>>>>> inputStream ), importOptions, taskId );
>>>>>>>>>>>>      }
>>>>>>>>>>>>
>>>>>>>>>>>> +    @Transactional
>>>>>>>>>>>>      @Override
>>>>>>>>>>>>      public void importGml( MetaData metaData, String userUid,
>>>>>>>>>>>> ImportOptions importOptions, TaskId taskId )
>>>>>>>>>>>>      {
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>  2015-06-03 15:00:19 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>  2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -62,6 +62,24 @@
>>>>>>>>>>>>          throws IOException, TransformerException;
>>>>>>>>>>>>
>>>>>>>>>>>>      /**
>>>>>>>>>>>> +     * Pre-process a GML document. The process, in short,
>>>>>>>>>>>> entails the following:
>>>>>>>>>>>> +     * <ol>
>>>>>>>>>>>> +     *     <li>Parse the GML payload and transform it into
>>>>>>>>>>>> DXF2 format</li>
>>>>>>>>>>>> +     *     <li>Get the given identifiers (uid, code or name)
>>>>>>>>>>>> from the parsed payload and fetch
>>>>>>>>>>>> +     *     the corresponding entities from the DB</li>
>>>>>>>>>>>> +     *     <li>Merge the geospatial data given in the input
>>>>>>>>>>>> GML into DB entities and return</li>
>>>>>>>>>>>> +     * </ol>
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * The result of this process in returned in a {@link
>>>>>>>>>>>> GmlPreProcessingResult} which
>>>>>>>>>>>> +     * encapsulates the returned {@link MetaData} object or
>>>>>>>>>>>> the exception in cause of parse
>>>>>>>>>>>> +     * failure due to IO errors or malformed input.
>>>>>>>>>>>> +     *
>>>>>>>>>>>> +     * @param gmlInputStream the InputStream providing the GML
>>>>>>>>>>>> input.
>>>>>>>>>>>> +     * @return a GmlPreProcessingResult representing the end
>>>>>>>>>>>> result of the process.
>>>>>>>>>>>> +     */
>>>>>>>>>>>> +    GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>>>>>> gmlInputStream );
>>>>>>>>>>>> +
>>>>>>>>>>>> +    /**
>>>>>>>>>>>>       * Imports GML data and merges the geospatial data updates
>>>>>>>>>>>> into the database.
>>>>>>>>>>>>       * See {@link #fromGml(InputStream)} for details on the
>>>>>>>>>>>> underlying process.
>>>>>>>>>>>>       *
>>>>>>>>>>>>
>>>>>>>>>>>> === added file
>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>>    1970-01-01 00:00:00 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>>    2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -0,0 +1,70 @@
>>>>>>>>>>>> +package org.hisp.dhis.dxf2.gml;
>>>>>>>>>>>> +
>>>>>>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData;
>>>>>>>>>>>> +
>>>>>>>>>>>> +import java.io.InputStream;
>>>>>>>>>>>> +
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * Wraps the result of {@link
>>>>>>>>>>>> GmlImportService#preProcessGml(InputStream)}.
>>>>>>>>>>>> + * This is necessary when performing GML import on a context
>>>>>>>>>>>> where exceptions
>>>>>>>>>>>> + * due to malformed input cannot be caught, thus leaving the
>>>>>>>>>>>> user uninformed of
>>>>>>>>>>>> + * the error. This class will wrap the failure and provide the
>>>>>>>>>>>> Throwable to the
>>>>>>>>>>>> + * consuming class on error or the resulting MetaData object
>>>>>>>>>>>> on success.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * @author Halvdan Hoem grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>>>>> + */
>>>>>>>>>>>> +public final class GmlPreProcessingResult
>>>>>>>>>>>> +{
>>>>>>>>>>>> +    private boolean isSuccess;
>>>>>>>>>>>> +    private MetaData resultMetaData;
>>>>>>>>>>>> +    private Throwable throwable;
>>>>>>>>>>>> +
>>>>>>>>>>>> +    private GmlPreProcessingResult(){}
>>>>>>>>>>>> +
>>>>>>>>>>>> +    public static GmlPreProcessingResult success( MetaData
>>>>>>>>>>>> resultMetaData ) {
>>>>>>>>>>>> +        GmlPreProcessingResult result = new
>>>>>>>>>>>> GmlPreProcessingResult();
>>>>>>>>>>>> +        result.setResultMetaData( resultMetaData );
>>>>>>>>>>>> +        result.setSuccess( true );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        return result;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    public static GmlPreProcessingResult failure( Throwable
>>>>>>>>>>>> throwable )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        GmlPreProcessingResult result = new
>>>>>>>>>>>> GmlPreProcessingResult();
>>>>>>>>>>>> +        result.setThrowable( throwable );
>>>>>>>>>>>> +        result.setSuccess( false );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        return result;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    private void setSuccess( boolean isSuccess )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        this.isSuccess = isSuccess;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    public boolean isSuccess()
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        return isSuccess;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    private void setResultMetaData( MetaData resultMetaData )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        this.resultMetaData = resultMetaData;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    public MetaData getResultMetaData()
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        return resultMetaData;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    private void setThrowable( Throwable throwable )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        this.throwable = throwable;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    public Throwable getThrowable()
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        return throwable;
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +}
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>      2015-01-17 07:41:26 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>      2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -117,4 +117,6 @@
>>>>>>>>>>>>          assertEquals( 1, units.get(
>>>>>>>>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
>>>>>>>>>>>>          assertEquals( 76, units.get( "Forskningsparken"
>>>>>>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
>>>>>>>>>>>>      }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    // TODO Add test for
>>>>>>>>>>>> GmlImportService#preProcessGml(InputStream)
>>>>>>>>>>>>  }
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>> 2015-05-30 13:36:07 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -77,7 +77,7 @@
>>>>>>>>>>>>      {
>>>>>>>>>>>>          taskExecutor.execute( task );
>>>>>>>>>>>>      }
>>>>>>>>>>>> -
>>>>>>>>>>>> +
>>>>>>>>>>>>      @Override
>>>>>>>>>>>>      public boolean scheduleTask( String key, Runnable task,
>>>>>>>>>>>> String cronExpr )
>>>>>>>>>>>>      {
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>       2015-05-28 16:10:07 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>       2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -187,7 +187,7 @@
>>>>>>>>>>>>          }
>>>>>>>>>>>>          else if ( "gml".equals( importFormat ) )
>>>>>>>>>>>>          {
>>>>>>>>>>>> -            scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>>>>>> userId, gmlImportService, importOptions, in, taskId ) );
>>>>>>>>>>>> +            scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) );
>>>>>>>>>>>>          }
>>>>>>>>>>>>          else if ( "json".equals( importFormat ) ||
>>>>>>>>>>>> "xml".equals( importFormat ) )
>>>>>>>>>>>>          {
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>      2015-04-11 14:06:51 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>      2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -28,14 +28,18 @@
>>>>>>>>>>>>   * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>>>>>>>>>>   */
>>>>>>>>>>>>
>>>>>>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils;
>>>>>>>>>>>>  import org.apache.commons.logging.Log;
>>>>>>>>>>>>  import org.apache.commons.logging.LogFactory;
>>>>>>>>>>>>  import org.hisp.dhis.dxf2.common.ImportOptions;
>>>>>>>>>>>>  import org.hisp.dhis.dxf2.gml.GmlImportService;
>>>>>>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
>>>>>>>>>>>>  import org.hisp.dhis.scheduling.TaskId;
>>>>>>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel;
>>>>>>>>>>>> +import org.hisp.dhis.system.notification.Notifier;
>>>>>>>>>>>> +import org.springframework.web.util.HtmlUtils;
>>>>>>>>>>>> +import org.xml.sax.SAXParseException;
>>>>>>>>>>>>
>>>>>>>>>>>> -import javax.xml.transform.TransformerException;
>>>>>>>>>>>> -import java.io.IOException;
>>>>>>>>>>>>  import java.io.InputStream;
>>>>>>>>>>>>
>>>>>>>>>>>>  /**
>>>>>>>>>>>> @@ -50,12 +54,15 @@
>>>>>>>>>>>>
>>>>>>>>>>>>      private String userUid;
>>>>>>>>>>>>
>>>>>>>>>>>> +
>>>>>>>>>>>>      //
>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>      // Dependencies
>>>>>>>>>>>>      //
>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>      private GmlImportService gmlImportService;
>>>>>>>>>>>>
>>>>>>>>>>>> +    private Notifier notifier;
>>>>>>>>>>>> +
>>>>>>>>>>>>      private ImportOptions importOptions;
>>>>>>>>>>>>
>>>>>>>>>>>>      private InputStream inputStream;
>>>>>>>>>>>> @@ -64,11 +71,12 @@
>>>>>>>>>>>>      // Constructors
>>>>>>>>>>>>      //
>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>> -    public ImportMetaDataGmlTask( String userUid,
>>>>>>>>>>>> GmlImportService gmlImportService,
>>>>>>>>>>>> +    public ImportMetaDataGmlTask( String userUid,
>>>>>>>>>>>> GmlImportService gmlImportService, Notifier notifier,
>>>>>>>>>>>>          ImportOptions importOptions, InputStream inputStream,
>>>>>>>>>>>> TaskId taskId )
>>>>>>>>>>>>      {
>>>>>>>>>>>>          this.userUid = userUid;
>>>>>>>>>>>>          this.gmlImportService = gmlImportService;
>>>>>>>>>>>> +        this.notifier = notifier;
>>>>>>>>>>>>          this.importOptions = importOptions;
>>>>>>>>>>>>          this.inputStream = inputStream;
>>>>>>>>>>>>          this.taskId = taskId;
>>>>>>>>>>>> @@ -83,15 +91,37 @@
>>>>>>>>>>>>      {
>>>>>>>>>>>>          importOptions.setImportStrategy( "update" ); // Force
>>>>>>>>>>>> update only for GML import
>>>>>>>>>>>>
>>>>>>>>>>>> -        try
>>>>>>>>>>>> -        {
>>>>>>>>>>>> -            gmlImportService.importGml( inputStream, userUid,
>>>>>>>>>>>> importOptions, taskId );
>>>>>>>>>>>> -        }
>>>>>>>>>>>> -        catch ( IOException | TransformerException e )
>>>>>>>>>>>> -        {
>>>>>>>>>>>> -            log.error( "Unable to read GML data from input
>>>>>>>>>>>> stream", e );
>>>>>>>>>>>> -
>>>>>>>>>>>> -            throw new RuntimeException( "Failed to parse GML
>>>>>>>>>>>> input stream", e );
>>>>>>>>>>>> -        }
>>>>>>>>>>>> +        GmlPreProcessingResult gmlPreProcessingResult =
>>>>>>>>>>>> gmlImportService.preProcessGml( inputStream );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        if ( !gmlPreProcessingResult.isSuccess() )
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            Throwable throwable =
>>>>>>>>>>>> gmlPreProcessingResult.getThrowable();
>>>>>>>>>>>> +            String message = createErrorMessage( throwable );
>>>>>>>>>>>> +
>>>>>>>>>>>> +            notifier.notify( taskId, NotificationLevel.ERROR,
>>>>>>>>>>>> message, false );
>>>>>>>>>>>> +            log.error( "GML import failed: " + message,
>>>>>>>>>>>> throwable );
>>>>>>>>>>>> +
>>>>>>>>>>>> +            return;
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>> +        gmlImportService.importGml(
>>>>>>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId
>>>>>>>>>>>> );
>>>>>>>>>>>> +    }
>>>>>>>>>>>> +
>>>>>>>>>>>> +    private String createErrorMessage( Throwable throwable )
>>>>>>>>>>>> +    {
>>>>>>>>>>>> +        String message = "";
>>>>>>>>>>>> +        Throwable rootThrowable = ExceptionUtils.getRootCause(
>>>>>>>>>>>> throwable );
>>>>>>>>>>>> +
>>>>>>>>>>>> +        if ( rootThrowable instanceof SAXParseException )
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            SAXParseException e = (SAXParseException)
>>>>>>>>>>>> rootThrowable;
>>>>>>>>>>>> +            message += "Syntax error on line " +
>>>>>>>>>>>> e.getLineNumber() + ". " + e.getMessage();
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +        else
>>>>>>>>>>>> +        {
>>>>>>>>>>>> +            message += rootThrowable.getMessage();
>>>>>>>>>>>> +        }
>>>>>>>>>>>> +
>>>>>>>>>>>> +        return HtmlUtils.htmlEscape( message );
>>>>>>>>>>>>      }
>>>>>>>>>>>>  }
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>> 2015-02-20 11:17:31 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -300,6 +300,7 @@
>>>>>>>>>>>>  ignored=Ignored
>>>>>>>>>>>>  conflicts=Conflicts
>>>>>>>>>>>>  no_conflicts_found=No conflicts found
>>>>>>>>>>>> +no_import_summary_available=No summary available
>>>>>>>>>>>>  type=Type
>>>>>>>>>>>>  count=Count
>>>>>>>>>>>>  export_as_xml=Export as XML
>>>>>>>>>>>>
>>>>>>>>>>>> === modified file
>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm'
>>>>>>>>>>>> ---
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>       2013-03-21 09:15:24 +0000
>>>>>>>>>>>> +++
>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>       2015-06-05 12:56:34 +0000
>>>>>>>>>>>> @@ -1,4 +1,6 @@
>>>>>>>>>>>>  <h3>$i18n.getString( "import_summary" )</h3>
>>>>>>>>>>>> +
>>>>>>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML
>>>>>>>>>>>> import (pre-processing fails)
>>>>>>>>>>>>  <h4>$i18n.getString( "import_count" )</h4>
>>>>>>>>>>>>
>>>>>>>>>>>>  $summary.importCount.imported Imported<br/>
>>>>>>>>>>>> @@ -54,3 +56,6 @@
>>>>>>>>>>>>  #else
>>>>>>>>>>>>  <p>$i18n.getString( "no_conflicts_found" )</p>
>>>>>>>>>>>>  #end
>>>>>>>>>>>> +#else
>>>>>>>>>>>> +    $i18n.getString( "no_summary_available" )
>>>>>>>>>>>> +#end
>>>>>>>>>>>> \ No newline at end of file
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> *******************************************
>>>>>>>>>>>
>>>>>>>>>>> Calle Hedberg
>>>>>>>>>>>
>>>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>>>>>
>>>>>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>>>>>
>>>>>>>>>>> Cell: +27-82-853-5352
>>>>>>>>>>>
>>>>>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>>>>>
>>>>>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>>>>>
>>>>>>>>>>> Skype: calle_hedberg
>>>>>>>>>>>
>>>>>>>>>>> *******************************************
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> *******************************************
>>>>>>>>>
>>>>>>>>> Calle Hedberg
>>>>>>>>>
>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>>>
>>>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>>>
>>>>>>>>> Cell: +27-82-853-5352
>>>>>>>>>
>>>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>>>
>>>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>>>
>>>>>>>>> Skype: calle_hedberg
>>>>>>>>>
>>>>>>>>> *******************************************
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>> Calle Hedberg
>>>>>>
>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>
>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>
>>>>>> Cell: +27-82-853-5352
>>>>>>
>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>
>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>
>>>>>> Skype: calle_hedberg
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *******************************************
>>>>>
>>>>> Calle Hedberg
>>>>>
>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>
>>>>> Tel/fax (home): +27-21-685-6472
>>>>>
>>>>> Cell: +27-82-853-5352
>>>>>
>>>>> Iridium SatPhone: +8816-315-19274
>>>>>
>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>
>>>>> Skype: calle_hedberg
>>>>>
>>>>> *******************************************
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *******************************************
>>>>
>>>> Calle Hedberg
>>>>
>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>
>>>> Tel/fax (home): +27-21-685-6472
>>>>
>>>> Cell: +27-82-853-5352
>>>>
>>>> Iridium SatPhone: +8816-315-19274
>>>>
>>>> Email: calle.hedberg@xxxxxxxxx
>>>>
>>>> Skype: calle_hedberg
>>>>
>>>> *******************************************
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>>