dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

Morten,

I'm not using curl to import GML files - I'm using the normal user
interface (apps -> import/export -> import GML file

Regards
Calle

On 1 July 2015 at 06:57, Morten Olav Hansen <mortenoh@xxxxxxxxx> wrote:

> Hi guys
>
> I've been trying to reproduce your bug this morning, but I'm not able to.
> I restored the db, and imported the xml without any issues, I tested on
> both 2.19 and 2.20, so I will need more information to reproduce.
>
> This was the command I used to import:
> curl -X POST -d @dxfstream.xml -H "Content-Type: application/xml" -v -u
> admin:district http://localhost:8080/api/metadata
>
> --
> Morten
>
> On Sun, Jun 28, 2015 at 5:16 AM, Halvdan Grelland <halvdanhg@xxxxxxxxx>
> wrote:
>
>> So. The stack size thing was just a qualified guess derived from the
>> stack trace. It's pretty obviously not the cause. I'll wait on feedback
>> from Morten before going any further on this.
>> On Jun 27, 2015 6:50 PM, "Calle Hedberg" <calle.hedberg@xxxxxxxxx> wrote:
>>
>>> Halvdan,
>>>
>>> I understand from your last email that the root cause of the problem is
>>> a 2.19 software issue - the meta data importer in 2.19 is not able to
>>> handle OrgUnit attribute values when present. That makes sense, and it is
>>> likely that the root cause of this bug / design flaw is a 2.19 change
>>> (since the same import worked fine in 2.18). Morten and/or Lars will
>>> hopefully know what change(s) caused this.
>>>
>>> With regard to JVM running out of heap space, and in relation to the
>>> (outdated - very!) chapter 8.2.8 you referenced:
>>>
>>> Firstly, with regard to the 150+ instances we are hosting on our data
>>> centre servers, we do have some constraints at the moment: Those servers
>>> have 32GB RAM only, and most instances are in reality running on 1-3GB.
>>> That will only change from next week, when we start moving to new servers
>>> with (initially) 128GB RAM. That should on average provide 6-8GB per
>>> instance (varying between 2 and 32GB).
>>>
>>> Secondly, finding the correct Java 7 and 8 (64 bits) and Tomcat 8.23 (64
>>> bits) memory allocations on my Windows notebook (32GB RAM) has been a
>>> nightmare - the people writing up java and tomcat documentation are
>>> useless, they seem unable to give any quantified advice at all where any
>>> normal person can find it. (In comparison, chapter 8.2.8 although outdated
>>> is super-clear). I am currently using the following setting for Java:
>>>
>>> JAVA_OPTS: -Xms4096m -Xmx8192m -XX:PermSize=512m -XX:MaxPermSize=1024m
>>>
>>> For some reason I/we do not understand, it seems Tomcat 8.23 does not
>>> use CATALINA_OPTS settings, so I ended up using the "Configure Tomcat" and
>>> then specify the following under the "Java" tab:
>>> Java options:
>>> -Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat 8.0
>>> -Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat 8.0
>>> -Djava.endorsed.dirs=C:\Program Files\Apache Software Foundation\Tomcat
>>> 8.0\endorsed
>>> -Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat
>>> 8.0\temp
>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>> -Djava.util.logging.config.file=C:\Program Files\Apache Software
>>> Foundation\Tomcat 8.0\conf\logging.properties
>>> -XX:MaxPermSize=4096m
>>>
>>> (NOTE: the -XX value here is different from what I currently have in the
>>> JAVA_OPTS environmental variable - I cannot find any info on which of them
>>> takes precedence
>>>
>>> Also under that same "Java" tab on "Configure Tomcat":
>>> Initial memory pool: 4096MB
>>> Maximum memory pool: 8192MB
>>> Thread stack size: 4096KB (4MB)
>>>
>>> Based on your initial advice to increase the stack size, I tried to
>>> increase it to 8MB (GML import still crashing), 64MB (GML import still
>>> crashing), and 132MB (GML import went completely haywire). Which is
>>> understandable, given your subsequent finding that the root cause of the
>>> problem was not related to heap/stack settings.
>>>
>>> So I've set the thread stack size back to 4096KB (4MB) - even if I don't
>>> know whether that's correct or not (seems high to my layman eyes, but maybe
>>> DHIS2 require it?)
>>>
>>> If anybody actually KNOWS what memory/stack sizes that I optimal for a
>>> 32GB Windows machine, I would appreciate the advice...
>>>
>>> Regards
>>> Calle
>>>
>>> On 26 June 2015 at 20:26, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:
>>>
>>>> Ok, I could have a look, but am preoccupied at the moment. Will get
>>>> back to you.
>>>>
>>>> In the meantime, it seems that you're facing a stack overflow error,
>>>> which could be caused by the JVM running out of heap space. Could you
>>>> please check the allocated heap for tomcat? If it is not being set (in the
>>>> CATALINA_OPTS env variable), please set it to something quite large and try
>>>> again.
>>>>
>>>> Under chapter 8.2.8 at the following link is a brief guide
>>>> https://www.dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html
>>>>
>>>> Please let me know how this works out for you. :)
>>>>
>>>> 2015-06-26 11:44 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>
>>>>> Halvdan,
>>>>>
>>>>> Here's a stripped down version of the Free State instance (no records
>>>>> in the data value table). Restore it as za_fs_s (psql -U dhis
>>>>> -f za_fs_s_26062015ext.sql -d za_fs_s, password dhis)
>>>>>
>>>>> You have got that single-record country GML file already.
>>>>>
>>>>> I would appreciate any help in figuring out what's happening. Just
>>>>> note that I'm using Postgresql 9.4, Java 7.79 64bits, and tomcat 8.23.
>>>>>
>>>>> Best regards
>>>>> Calle
>>>>>
>>>>> On 26 June 2015 at 11:17, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I ran it again (revision 19247) and pulled out the first part of the
>>>>>> log error:
>>>>>>
>>>>>> * INFO  2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import
>>>>>> at Fri Jun 26 11:04:18 CAT 2015 (DefaultImportService.java
>>>>>> [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:18,348 [Level: INFO, category:
>>>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:18 CAT 2015, message: Importing
>>>>>> meta-data] (InMemoryNotifier.java [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:18,348 Building object-bridge maps
>>>>>> (preheatCache: true). (DefaultObjectBridge.java [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:21,141 Building object-bridge maps took 2.79
>>>>>> seconds. (DefaultObjectBridge.java [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:21,141 [Level: INFO, category:
>>>>>> METADATA_IMPORT, time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1
>>>>>> OrganisationUnits] (InMemoryNotifier.java [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with
>>>>>> object of type AttributeValue (DefaultDeletionManager.java
>>>>>> [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with
>>>>>> object of type AttributeValue (DefaultDeletionManager.java
>>>>>> [taskScheduler-1])
>>>>>> * INFO  2015-06-26 11:04:21,151 Deleted objects associated with
>>>>>> object of type AttributeValue (DefaultDeletionManager.java
>>>>>> [taskScheduler-1])
>>>>>> * ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in
>>>>>> scheduled task. (TaskUtils.java [taskScheduler-1])
>>>>>> java.lang.StackOverflowError
>>>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
>>>>>> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
>>>>>> at java.util.regex.Pattern$BitClass.<init>(Unknown Source)
>>>>>> at java.util.regex.Pattern.clazz(Unknown Source)
>>>>>> at java.util.regex.Pattern.sequence(Unknown Source)
>>>>>> at java.util.regex.Pattern.expr(Unknown Source)
>>>>>> at java.util.regex.Pattern.compile(Unknown Source)
>>>>>> at java.util.regex.Pattern.<init>(Unknown Source)
>>>>>> at java.util.regex.Pattern.compile(Unknown Source)
>>>>>> at java.lang.String.replaceAll(Unknown Source)
>>>>>> at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)
>>>>>> at
>>>>>> com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)
>>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)
>>>>>> at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>> at
>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)
>>>>>> at
>>>>>> org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)
>>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>>> at
>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)
>>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>>>> at
>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>>> at
>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Regards
>>>>>> Calle
>>>>>>
>>>>>> On 26 June 2015 at 10:12, Calle Hedberg <calle.hedberg@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Firstly, yes we are using a custom revision - although it is
>>>>>>> SUPPOSED to be based on a 2.19 version from a few days ago. I will check
>>>>>>> that.
>>>>>>>
>>>>>>> BUT I have tried the import using revision 19247 too, and the
>>>>>>> problem/behaviour is exactly the same - it hangs. Whereas if I import into
>>>>>>> an older version of the same provincial instance that still runs on 2.18,
>>>>>>> it imports fine.
>>>>>>>
>>>>>>> The fact that the same GML file imports in the Sierra Leone demo - I
>>>>>>> cannot explain that, and the tomcat log only shows that
>>>>>>> Attribute.toString() looping...
>>>>>>>
>>>>>>> Can I create a stripped down version of the instance and upload it
>>>>>>> for you to test?
>>>>>>>
>>>>>>> Regards
>>>>>>> Calle
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 26 June 2015 at 04:09, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> It seems to be looping in Attribute.toString(), I remember this was
>>>>>>>> an issue at some point.. but that was a long long time ago.. so maybe you
>>>>>>>> need a revision bump
>>>>>>>>
>>>>>>>> --
>>>>>>>> Morten
>>>>>>>>
>>>>>>>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <
>>>>>>>> halvdanhg@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> I tried your example GML on both our demo servers (dev on 2.20 and
>>>>>>>>> demo on 2.19, same rev as your server) and it works just fine.
>>>>>>>>>
>>>>>>>>> If you're experiencing it hanging and not finishing you might be
>>>>>>>>> experiencing some kind of performance and/or memory issue. The coordinate
>>>>>>>>> set is fairly large (though it works well on our demos, but does take some
>>>>>>>>> time to finish). You can verify by trying yourself to create the orgunit
>>>>>>>>> (with the exact name) and import the GML on the Sierra Leone demo.
>>>>>>>>>
>>>>>>>>> The fact that the dry run seems to "work" for you (as well as the
>>>>>>>>> stack trace you provided) also points at the issue not residing in the GML
>>>>>>>>> processing but rather in the actual import stage (the GML importer uses the
>>>>>>>>> metadata importer as it's backend).
>>>>>>>>>
>>>>>>>>> Another thing: The instance you linked to is using a revision of 16783
>>>>>>>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247).
>>>>>>>>> Are you using a custom build? Does it contain customizations to the core in
>>>>>>>>> any way? That would also be a possible source of issues, obviously.
>>>>>>>>>
>>>>>>>>> You could try deploying this war
>>>>>>>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war
>>>>>>>>> and replicate the issue using the za South Africa orgunit.
>>>>>>>>>
>>>>>>>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>>>>>>
>>>>>>>>>> Halvdan,
>>>>>>>>>>
>>>>>>>>>> The GML files I'm trying to import using 2.19 is the same GML
>>>>>>>>>> files I've been using successfully for 2.17, 2.18 (so it's using NAME).
>>>>>>>>>> Something in the import has changed:
>>>>>>>>>>
>>>>>>>>>> I have attached the small GML file I'm trying to import - it has
>>>>>>>>>> the international boundary of SA, so there is only ONE GIS record (top node
>>>>>>>>>> in all our OrgHierarchies). It imports with no problems in 2.18.
>>>>>>>>>>
>>>>>>>>>> In 2.19, if I choose YES for dry run, it will complete with usual
>>>>>>>>>> import summary etc.
>>>>>>>>>>
>>>>>>>>>> If I choose NO for dry run, I get
>>>>>>>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits  2015-06-25
>>>>>>>>>> 22:18:11Importing meta-data
>>>>>>>>>> and there it hangs. Nothing gets imported.
>>>>>>>>>>
>>>>>>>>>> The online instance you can try to import into is
>>>>>>>>>> http://fs.dhis.dhmis.org/staging/
>>>>>>>>>>
>>>>>>>>>> When I try to import into a local instance, it seems to enter
>>>>>>>>>> some kind of endless loop based on the tomcat log:
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>>> Source)
>>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at java.util.AbstractCollection.toString(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>>> Source)
>>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at java.util.AbstractCollection.toString(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>>>>>>>   at java.lang.String.valueOf(Unknown Source)
>>>>>>>>>>   at java.lang.StringBuilder.append(Unknown Source)
>>>>>>>>>>   at
>>>>>>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>>>>>>>   at
>>>>>>>>>> org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>>>>>>>   at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>>>>>>>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>>>>>>>>>> Source)
>>>>>>>>>>   at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>>>>>>
>>>>>>>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Calle,
>>>>>>>>>>>
>>>>>>>>>>> The commit you've replied to implements (but not fully,
>>>>>>>>>>> therefore 'WIP', there's another commit following it pretty soon after)
>>>>>>>>>>> better error reporting feedback to the user through the interface. I.e. not
>>>>>>>>>>> having to see the stacktrace on the server to determine if and why
>>>>>>>>>>> something went wrong. I believe you have been among the people requesting
>>>>>>>>>>> this too :)
>>>>>>>>>>>
>>>>>>>>>>> Also, as you're referencing, we've made some other changes to
>>>>>>>>>>> GML import. In particular the importer will now match on uid, code or name,
>>>>>>>>>>> as oppsed to the former mode of matching only on name. This is not
>>>>>>>>>>> reflected in the UI as it is a backend enhancement only, and does not
>>>>>>>>>>> change the import options at all. It IS, however, documented and ready for
>>>>>>>>>>> you to read about. Short version: it matches on a prioritized list of uid
>>>>>>>>>>> -> code -> name. Long version:
>>>>>>>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html
>>>>>>>>>>>
>>>>>>>>>>> Be adviced that the commit you've referenced is towards
>>>>>>>>>>> development trunk, and is not backported to 2.19, the match on multiple
>>>>>>>>>>> identifiers, however, was introduced in 2.19 and should work in current
>>>>>>>>>>> 2.19 stable. If you're having problems with it you can send me a full stack
>>>>>>>>>>> trace and preferably the GML input data.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <
>>>>>>>>>>> calle.hedberg@xxxxxxxxx>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm unable to import GML files using 2.19 (the same files
>>>>>>>>>>>> import fine in 2.18) - the import simply hangs (endless rolling progress
>>>>>>>>>>>> bar icon).
>>>>>>>>>>>>
>>>>>>>>>>>> Will the above bug-fix sort out that problem - I do not
>>>>>>>>>>>> understand what the above actually means, or if the GML format has changed
>>>>>>>>>>>> in some way.
>>>>>>>>>>>>
>>>>>>>>>>>> Note also that the final 2.19 release was supposed to include a
>>>>>>>>>>>> more diversified GML import - but I cannot see WHERE to do that (I still
>>>>>>>>>>>> see the same two buttons: choose file and import, either dry run or not)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Calle
>>>>>>>>>>>>
>>>>>>>>>>>> On 5 June 2015 at 14:57, <noreply@xxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> ------------------------------------------------------------
>>>>>>>>>>>>> revno: 19296
>>>>>>>>>>>>> committer: Halvdan Hoem Grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>>>>>> branch nick: dhis2
>>>>>>>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200
>>>>>>>>>>>>> message:
>>>>>>>>>>>>>   WIP proper error reporting on failed GML parsing during
>>>>>>>>>>>>> import
>>>>>>>>>>>>> added:
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>>> modified:
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>>>
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> lp:dhis2
>>>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>>>>>>>>>>> To unsubscribe from this branch go to
>>>>>>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>>   2015-06-03 15:00:19 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>>>>>>   2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -32,6 +32,8 @@
>>>>>>>>>>>>>  import com.google.common.base.Strings;
>>>>>>>>>>>>>  import com.google.common.collect.Iterators;
>>>>>>>>>>>>>  import com.google.common.collect.Maps;
>>>>>>>>>>>>> +import org.apache.commons.io.IOUtils;
>>>>>>>>>>>>> +import org.hibernate.Hibernate;
>>>>>>>>>>>>>  import org.hisp.dhis.common.IdentifiableObjectManager;
>>>>>>>>>>>>>  import org.hisp.dhis.common.IdentifiableProperty;
>>>>>>>>>>>>>  import org.hisp.dhis.common.MergeStrategy;
>>>>>>>>>>>>> @@ -88,13 +90,91 @@
>>>>>>>>>>>>>      // GmlImportService implementation
>>>>>>>>>>>>>      //
>>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> +    @Transactional
>>>>>>>>>>>>> +    @Override
>>>>>>>>>>>>> +    public GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>>>>>>> inputStream )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        InputStream dxfStream = null;
>>>>>>>>>>>>> +        MetaData metaData = null;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        try
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            dxfStream = transformGml( inputStream );
>>>>>>>>>>>>> +            metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +        catch ( IOException | TransformerException e )
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            return GmlPreProcessingResult.failure( e );
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +        finally
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            IOUtils.closeQuietly( dxfStream );
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        Map<String, OrganisationUnit> uidMap  =
>>>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        matchAndFilterOnIdentifiers(
>>>>>>>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedUidMap  =
>>>>>>>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(),  IdentifiableProperty.UID );
>>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedCodeMap =
>>>>>>>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
>>>>>>>>>>>>> +        Map<String, OrganisationUnit> persistedNameMap =
>>>>>>>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        Iterator<OrganisationUnit> persistedIterator =
>>>>>>>>>>>>> Iterators.concat( persistedUidMap.values().iterator(),
>>>>>>>>>>>>> +            persistedCodeMap.values().iterator(),
>>>>>>>>>>>>> persistedNameMap.values().iterator() );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        while ( persistedIterator.hasNext() )
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            OrganisationUnit persisted =
>>>>>>>>>>>>> persistedIterator.next(), imported = null;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +            if ( !Strings.isNullOrEmpty( persisted.getUid() )
>>>>>>>>>>>>> && uidMap.containsKey( persisted.getUid() ) )
>>>>>>>>>>>>> +            {
>>>>>>>>>>>>> +                imported = uidMap.get( persisted.getUid() );
>>>>>>>>>>>>> +            }
>>>>>>>>>>>>> +            else if ( !Strings.isNullOrEmpty(
>>>>>>>>>>>>> persisted.getCode() ) && codeMap.containsKey( persisted.getCode() ) )
>>>>>>>>>>>>> +            {
>>>>>>>>>>>>> +                imported = codeMap.get( persisted.getCode() );
>>>>>>>>>>>>> +            }
>>>>>>>>>>>>> +            else if ( !Strings.isNullOrEmpty(
>>>>>>>>>>>>> persisted.getName() ) && nameMap.containsKey( persisted.getName() ) )
>>>>>>>>>>>>> +            {
>>>>>>>>>>>>> +                imported = nameMap.get( persisted.getName() );
>>>>>>>>>>>>> +            }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +            if ( imported == null ||
>>>>>>>>>>>>> imported.getCoordinates() == null || imported.getFeatureType() == null )
>>>>>>>>>>>>> +            {
>>>>>>>>>>>>> +                continue; // Failed to dereference a
>>>>>>>>>>>>> persisted entity for this org unit or geo data incomplete/missing,
>>>>>>>>>>>>> therefore ignore
>>>>>>>>>>>>> +            }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +            mergeNonGeoData( persisted, imported );
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        return GmlPreProcessingResult.success( metaData );
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      @Override
>>>>>>>>>>>>>      public MetaData fromGml( InputStream inputStream )
>>>>>>>>>>>>>          throws IOException, TransformerException
>>>>>>>>>>>>>      {
>>>>>>>>>>>>> -        InputStream dxfStream = transformGml( inputStream );
>>>>>>>>>>>>> -        MetaData metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>>> -        dxfStream.close();
>>>>>>>>>>>>> +        InputStream dxfStream;
>>>>>>>>>>>>> +        MetaData metaData;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        try
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            dxfStream = transformGml( inputStream );
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +        catch (Exception e)
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            dxfStream = null;
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        if(dxfStream != null)
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            metaData = renderService.fromXml( dxfStream,
>>>>>>>>>>>>> MetaData.class );
>>>>>>>>>>>>> +            dxfStream.close();
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +        else
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            return null;
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>>
>>>>>>>>>>>>>          Map<String, OrganisationUnit> uidMap  =
>>>>>>>>>>>>> Maps.newHashMap(), codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>>>>>>>
>>>>>>>>>>>>> @@ -143,6 +223,7 @@
>>>>>>>>>>>>>          importService.importMetaData( userUid, fromGml(
>>>>>>>>>>>>> inputStream ), importOptions, taskId );
>>>>>>>>>>>>>      }
>>>>>>>>>>>>>
>>>>>>>>>>>>> +    @Transactional
>>>>>>>>>>>>>      @Override
>>>>>>>>>>>>>      public void importGml( MetaData metaData, String userUid,
>>>>>>>>>>>>> ImportOptions importOptions, TaskId taskId )
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>>  2015-06-03 15:00:19 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>>>>>>  2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -62,6 +62,24 @@
>>>>>>>>>>>>>          throws IOException, TransformerException;
>>>>>>>>>>>>>
>>>>>>>>>>>>>      /**
>>>>>>>>>>>>> +     * Pre-process a GML document. The process, in short,
>>>>>>>>>>>>> entails the following:
>>>>>>>>>>>>> +     * <ol>
>>>>>>>>>>>>> +     *     <li>Parse the GML payload and transform it into
>>>>>>>>>>>>> DXF2 format</li>
>>>>>>>>>>>>> +     *     <li>Get the given identifiers (uid, code or name)
>>>>>>>>>>>>> from the parsed payload and fetch
>>>>>>>>>>>>> +     *     the corresponding entities from the DB</li>
>>>>>>>>>>>>> +     *     <li>Merge the geospatial data given in the input
>>>>>>>>>>>>> GML into DB entities and return</li>
>>>>>>>>>>>>> +     * </ol>
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * The result of this process in returned in a {@link
>>>>>>>>>>>>> GmlPreProcessingResult} which
>>>>>>>>>>>>> +     * encapsulates the returned {@link MetaData} object or
>>>>>>>>>>>>> the exception in cause of parse
>>>>>>>>>>>>> +     * failure due to IO errors or malformed input.
>>>>>>>>>>>>> +     *
>>>>>>>>>>>>> +     * @param gmlInputStream the InputStream providing the
>>>>>>>>>>>>> GML input.
>>>>>>>>>>>>> +     * @return a GmlPreProcessingResult representing the end
>>>>>>>>>>>>> result of the process.
>>>>>>>>>>>>> +     */
>>>>>>>>>>>>> +    GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>>>>>>> gmlInputStream );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    /**
>>>>>>>>>>>>>       * Imports GML data and merges the geospatial data
>>>>>>>>>>>>> updates into the database.
>>>>>>>>>>>>>       * See {@link #fromGml(InputStream)} for details on the
>>>>>>>>>>>>> underlying process.
>>>>>>>>>>>>>       *
>>>>>>>>>>>>>
>>>>>>>>>>>>> === added file
>>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>>>    1970-01-01 00:00:00 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>>>>>>>    2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -0,0 +1,70 @@
>>>>>>>>>>>>> +package org.hisp.dhis.dxf2.gml;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +import java.io.InputStream;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * Wraps the result of {@link
>>>>>>>>>>>>> GmlImportService#preProcessGml(InputStream)}.
>>>>>>>>>>>>> + * This is necessary when performing GML import on a context
>>>>>>>>>>>>> where exceptions
>>>>>>>>>>>>> + * due to malformed input cannot be caught, thus leaving the
>>>>>>>>>>>>> user uninformed of
>>>>>>>>>>>>> + * the error. This class will wrap the failure and provide
>>>>>>>>>>>>> the Throwable to the
>>>>>>>>>>>>> + * consuming class on error or the resulting MetaData object
>>>>>>>>>>>>> on success.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * @author Halvdan Hoem grelland <halvdanhg@xxxxxxxxx>
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +public final class GmlPreProcessingResult
>>>>>>>>>>>>> +{
>>>>>>>>>>>>> +    private boolean isSuccess;
>>>>>>>>>>>>> +    private MetaData resultMetaData;
>>>>>>>>>>>>> +    private Throwable throwable;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    private GmlPreProcessingResult(){}
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    public static GmlPreProcessingResult success( MetaData
>>>>>>>>>>>>> resultMetaData ) {
>>>>>>>>>>>>> +        GmlPreProcessingResult result = new
>>>>>>>>>>>>> GmlPreProcessingResult();
>>>>>>>>>>>>> +        result.setResultMetaData( resultMetaData );
>>>>>>>>>>>>> +        result.setSuccess( true );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        return result;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    public static GmlPreProcessingResult failure( Throwable
>>>>>>>>>>>>> throwable )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        GmlPreProcessingResult result = new
>>>>>>>>>>>>> GmlPreProcessingResult();
>>>>>>>>>>>>> +        result.setThrowable( throwable );
>>>>>>>>>>>>> +        result.setSuccess( false );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        return result;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    private void setSuccess( boolean isSuccess )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        this.isSuccess = isSuccess;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    public boolean isSuccess()
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        return isSuccess;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    private void setResultMetaData( MetaData resultMetaData )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        this.resultMetaData = resultMetaData;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    public MetaData getResultMetaData()
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        return resultMetaData;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    private void setThrowable( Throwable throwable )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        this.throwable = throwable;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    public Throwable getThrowable()
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        return throwable;
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +}
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>>      2015-01-17 07:41:26 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>>>>>>      2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -117,4 +117,6 @@
>>>>>>>>>>>>>          assertEquals( 1, units.get(
>>>>>>>>>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
>>>>>>>>>>>>>          assertEquals( 76, units.get( "Forskningsparken"
>>>>>>>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
>>>>>>>>>>>>>      }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    // TODO Add test for
>>>>>>>>>>>>> GmlImportService#preProcessGml(InputStream)
>>>>>>>>>>>>>  }
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>>> 2015-05-30 13:36:07 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -77,7 +77,7 @@
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>          taskExecutor.execute( task );
>>>>>>>>>>>>>      }
>>>>>>>>>>>>> -
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      @Override
>>>>>>>>>>>>>      public boolean scheduleTask( String key, Runnable task,
>>>>>>>>>>>>> String cronExpr )
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>>       2015-05-28 16:10:07 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>>>>>>       2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -187,7 +187,7 @@
>>>>>>>>>>>>>          }
>>>>>>>>>>>>>          else if ( "gml".equals( importFormat ) )
>>>>>>>>>>>>>          {
>>>>>>>>>>>>> -            scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>>>>>>> userId, gmlImportService, importOptions, in, taskId ) );
>>>>>>>>>>>>> +            scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) );
>>>>>>>>>>>>>          }
>>>>>>>>>>>>>          else if ( "json".equals( importFormat ) ||
>>>>>>>>>>>>> "xml".equals( importFormat ) )
>>>>>>>>>>>>>          {
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>>      2015-04-11 14:06:51 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>>>>>>      2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -28,14 +28,18 @@
>>>>>>>>>>>>>   * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
>>>>>>>>>>>>> DAMAGE.
>>>>>>>>>>>>>   */
>>>>>>>>>>>>>
>>>>>>>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils;
>>>>>>>>>>>>>  import org.apache.commons.logging.Log;
>>>>>>>>>>>>>  import org.apache.commons.logging.LogFactory;
>>>>>>>>>>>>>  import org.hisp.dhis.dxf2.common.ImportOptions;
>>>>>>>>>>>>>  import org.hisp.dhis.dxf2.gml.GmlImportService;
>>>>>>>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
>>>>>>>>>>>>>  import org.hisp.dhis.scheduling.TaskId;
>>>>>>>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel;
>>>>>>>>>>>>> +import org.hisp.dhis.system.notification.Notifier;
>>>>>>>>>>>>> +import org.springframework.web.util.HtmlUtils;
>>>>>>>>>>>>> +import org.xml.sax.SAXParseException;
>>>>>>>>>>>>>
>>>>>>>>>>>>> -import javax.xml.transform.TransformerException;
>>>>>>>>>>>>> -import java.io.IOException;
>>>>>>>>>>>>>  import java.io.InputStream;
>>>>>>>>>>>>>
>>>>>>>>>>>>>  /**
>>>>>>>>>>>>> @@ -50,12 +54,15 @@
>>>>>>>>>>>>>
>>>>>>>>>>>>>      private String userUid;
>>>>>>>>>>>>>
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      //
>>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>>      // Dependencies
>>>>>>>>>>>>>      //
>>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>>      private GmlImportService gmlImportService;
>>>>>>>>>>>>>
>>>>>>>>>>>>> +    private Notifier notifier;
>>>>>>>>>>>>> +
>>>>>>>>>>>>>      private ImportOptions importOptions;
>>>>>>>>>>>>>
>>>>>>>>>>>>>      private InputStream inputStream;
>>>>>>>>>>>>> @@ -64,11 +71,12 @@
>>>>>>>>>>>>>      // Constructors
>>>>>>>>>>>>>      //
>>>>>>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>> -    public ImportMetaDataGmlTask( String userUid,
>>>>>>>>>>>>> GmlImportService gmlImportService,
>>>>>>>>>>>>> +    public ImportMetaDataGmlTask( String userUid,
>>>>>>>>>>>>> GmlImportService gmlImportService, Notifier notifier,
>>>>>>>>>>>>>          ImportOptions importOptions, InputStream inputStream,
>>>>>>>>>>>>> TaskId taskId )
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>          this.userUid = userUid;
>>>>>>>>>>>>>          this.gmlImportService = gmlImportService;
>>>>>>>>>>>>> +        this.notifier = notifier;
>>>>>>>>>>>>>          this.importOptions = importOptions;
>>>>>>>>>>>>>          this.inputStream = inputStream;
>>>>>>>>>>>>>          this.taskId = taskId;
>>>>>>>>>>>>> @@ -83,15 +91,37 @@
>>>>>>>>>>>>>      {
>>>>>>>>>>>>>          importOptions.setImportStrategy( "update" ); // Force
>>>>>>>>>>>>> update only for GML import
>>>>>>>>>>>>>
>>>>>>>>>>>>> -        try
>>>>>>>>>>>>> -        {
>>>>>>>>>>>>> -            gmlImportService.importGml( inputStream, userUid,
>>>>>>>>>>>>> importOptions, taskId );
>>>>>>>>>>>>> -        }
>>>>>>>>>>>>> -        catch ( IOException | TransformerException e )
>>>>>>>>>>>>> -        {
>>>>>>>>>>>>> -            log.error( "Unable to read GML data from input
>>>>>>>>>>>>> stream", e );
>>>>>>>>>>>>> -
>>>>>>>>>>>>> -            throw new RuntimeException( "Failed to parse GML
>>>>>>>>>>>>> input stream", e );
>>>>>>>>>>>>> -        }
>>>>>>>>>>>>> +        GmlPreProcessingResult gmlPreProcessingResult =
>>>>>>>>>>>>> gmlImportService.preProcessGml( inputStream );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        if ( !gmlPreProcessingResult.isSuccess() )
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            Throwable throwable =
>>>>>>>>>>>>> gmlPreProcessingResult.getThrowable();
>>>>>>>>>>>>> +            String message = createErrorMessage( throwable );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +            notifier.notify( taskId, NotificationLevel.ERROR,
>>>>>>>>>>>>> message, false );
>>>>>>>>>>>>> +            log.error( "GML import failed: " + message,
>>>>>>>>>>>>> throwable );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +            return;
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        gmlImportService.importGml(
>>>>>>>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId
>>>>>>>>>>>>> );
>>>>>>>>>>>>> +    }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    private String createErrorMessage( Throwable throwable )
>>>>>>>>>>>>> +    {
>>>>>>>>>>>>> +        String message = "";
>>>>>>>>>>>>> +        Throwable rootThrowable =
>>>>>>>>>>>>> ExceptionUtils.getRootCause( throwable );
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        if ( rootThrowable instanceof SAXParseException )
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            SAXParseException e = (SAXParseException)
>>>>>>>>>>>>> rootThrowable;
>>>>>>>>>>>>> +            message += "Syntax error on line " +
>>>>>>>>>>>>> e.getLineNumber() + ". " + e.getMessage();
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +        else
>>>>>>>>>>>>> +        {
>>>>>>>>>>>>> +            message += rootThrowable.getMessage();
>>>>>>>>>>>>> +        }
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +        return HtmlUtils.htmlEscape( message );
>>>>>>>>>>>>>      }
>>>>>>>>>>>>>  }
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>>> 2015-02-20 11:17:31 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -300,6 +300,7 @@
>>>>>>>>>>>>>  ignored=Ignored
>>>>>>>>>>>>>  conflicts=Conflicts
>>>>>>>>>>>>>  no_conflicts_found=No conflicts found
>>>>>>>>>>>>> +no_import_summary_available=No summary available
>>>>>>>>>>>>>  type=Type
>>>>>>>>>>>>>  count=Count
>>>>>>>>>>>>>  export_as_xml=Export as XML
>>>>>>>>>>>>>
>>>>>>>>>>>>> === modified file
>>>>>>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm'
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>>       2013-03-21 09:15:24 +0000
>>>>>>>>>>>>> +++
>>>>>>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>>>>>>       2015-06-05 12:56:34 +0000
>>>>>>>>>>>>> @@ -1,4 +1,6 @@
>>>>>>>>>>>>>  <h3>$i18n.getString( "import_summary" )</h3>
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML
>>>>>>>>>>>>> import (pre-processing fails)
>>>>>>>>>>>>>  <h4>$i18n.getString( "import_count" )</h4>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  $summary.importCount.imported Imported<br/>
>>>>>>>>>>>>> @@ -54,3 +56,6 @@
>>>>>>>>>>>>>  #else
>>>>>>>>>>>>>  <p>$i18n.getString( "no_conflicts_found" )</p>
>>>>>>>>>>>>>  #end
>>>>>>>>>>>>> +#else
>>>>>>>>>>>>> +    $i18n.getString( "no_summary_available" )
>>>>>>>>>>>>> +#end
>>>>>>>>>>>>> \ No newline at end of file
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>>
>>>>>>>>>>>> *******************************************
>>>>>>>>>>>>
>>>>>>>>>>>> Calle Hedberg
>>>>>>>>>>>>
>>>>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>>>>>>
>>>>>>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>>>>>>
>>>>>>>>>>>> Cell: +27-82-853-5352
>>>>>>>>>>>>
>>>>>>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>>>>>>
>>>>>>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>>>>>>
>>>>>>>>>>>> Skype: calle_hedberg
>>>>>>>>>>>>
>>>>>>>>>>>> *******************************************
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> *******************************************
>>>>>>>>>>
>>>>>>>>>> Calle Hedberg
>>>>>>>>>>
>>>>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>>>>
>>>>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>>>>
>>>>>>>>>> Cell: +27-82-853-5352
>>>>>>>>>>
>>>>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>>>>
>>>>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>>>>
>>>>>>>>>> Skype: calle_hedberg
>>>>>>>>>>
>>>>>>>>>> *******************************************
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>> Calle Hedberg
>>>>>>>
>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>
>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>
>>>>>>> Cell: +27-82-853-5352
>>>>>>>
>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>
>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>
>>>>>>> Skype: calle_hedberg
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>> Calle Hedberg
>>>>>>
>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>
>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>
>>>>>> Cell: +27-82-853-5352
>>>>>>
>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>
>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>
>>>>>> Skype: calle_hedberg
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *******************************************
>>>>>
>>>>> Calle Hedberg
>>>>>
>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>
>>>>> Tel/fax (home): +27-21-685-6472
>>>>>
>>>>> Cell: +27-82-853-5352
>>>>>
>>>>> Iridium SatPhone: +8816-315-19274
>>>>>
>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>
>>>>> Skype: calle_hedberg
>>>>>
>>>>> *******************************************
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> *******************************************
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472
>>>
>>> Cell: +27-82-853-5352
>>>
>>> Iridium SatPhone: +8816-315-19274
>>>
>>> Email: calle.hedberg@xxxxxxxxx
>>>
>>> Skype: calle_hedberg
>>>
>>> *******************************************
>>>
>>>
>


-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************