dhis2-devs team mailing list archive
-
dhis2-devs team
-
Mailing list archive
-
Message #38256
Re: [Branch ~dhis2-devs-core/dhis2/trunk] Rev 19296: WIP proper error reporting on failed GML parsing during import
Hi,
I ran it again (revision 19247) and pulled out the first part of the log
error:
* INFO 2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import at Fri
Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])
* INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT,
time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data]
(InMemoryNotifier.java [taskScheduler-1])
* INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache:
true). (DefaultObjectBridge.java [taskScheduler-1])
* INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79
seconds. (DefaultObjectBridge.java [taskScheduler-1])
* INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT,
time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits]
(InMemoryNotifier.java [taskScheduler-1])
* INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
* ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled
task. (TaskUtils.java [taskScheduler-1])
java.lang.StackOverflowError
at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
at java.util.regex.Pattern$BitClass.<init>(Unknown Source)
at java.util.regex.Pattern.clazz(Unknown Source)
at java.util.regex.Pattern.sequence(Unknown Source)
at java.util.regex.Pattern.expr(Unknown Source)
at java.util.regex.Pattern.compile(Unknown Source)
at java.util.regex.Pattern.<init>(Unknown Source)
at java.util.regex.Pattern.compile(Unknown Source)
at java.lang.String.replaceAll(Unknown Source)
at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)
at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)
at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)
at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)
at
org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)
at java.lang.String.valueOf(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)
at java.lang.String.valueOf(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at java.util.AbstractCollection.toString(Unknown Source)
at
org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
at java.lang.String.valueOf(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at
com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
Any ideas?
Regards
Calle
On 26 June 2015 at 10:12, Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:
> Hi,
>
> Firstly, yes we are using a custom revision - although it is SUPPOSED to
> be based on a 2.19 version from a few days ago. I will check that.
>
> BUT I have tried the import using revision 19247 too, and the
> problem/behaviour is exactly the same - it hangs. Whereas if I import into
> an older version of the same provincial instance that still runs on 2.18,
> it imports fine.
>
> The fact that the same GML file imports in the Sierra Leone demo - I
> cannot explain that, and the tomcat log only shows that
> Attribute.toString() looping...
>
> Can I create a stripped down version of the instance and upload it for you
> to test?
>
> Regards
> Calle
>
>
>
> On 26 June 2015 at 04:09, Morten Olav Hansen <mortenoh@xxxxxxxxx> wrote:
>
>> It seems to be looping in Attribute.toString(), I remember this was an
>> issue at some point.. but that was a long long time ago.. so maybe you need
>> a revision bump
>>
>> --
>> Morten
>>
>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>> wrote:
>>
>>> I tried your example GML on both our demo servers (dev on 2.20 and demo
>>> on 2.19, same rev as your server) and it works just fine.
>>>
>>> If you're experiencing it hanging and not finishing you might be
>>> experiencing some kind of performance and/or memory issue. The coordinate
>>> set is fairly large (though it works well on our demos, but does take some
>>> time to finish). You can verify by trying yourself to create the orgunit
>>> (with the exact name) and import the GML on the Sierra Leone demo.
>>>
>>> The fact that the dry run seems to "work" for you (as well as the stack
>>> trace you provided) also points at the issue not residing in the GML
>>> processing but rather in the actual import stage (the GML importer uses the
>>> metadata importer as it's backend).
>>>
>>> Another thing: The instance you linked to is using a revision of 16783
>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are
>>> you using a custom build? Does it contain customizations to the core in any
>>> way? That would also be a possible source of issues, obviously.
>>>
>>> You could try deploying this war
>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war
>>> and replicate the issue using the za South Africa orgunit.
>>>
>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>
>>>> Halvdan,
>>>>
>>>> The GML files I'm trying to import using 2.19 is the same GML files
>>>> I've been using successfully for 2.17, 2.18 (so it's using NAME). Something
>>>> in the import has changed:
>>>>
>>>> I have attached the small GML file I'm trying to import - it has the
>>>> international boundary of SA, so there is only ONE GIS record (top node in
>>>> all our OrgHierarchies). It imports with no problems in 2.18.
>>>>
>>>> In 2.19, if I choose YES for dry run, it will complete with usual
>>>> import summary etc.
>>>>
>>>> If I choose NO for dry run, I get
>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits 2015-06-25 22:18:11Importing
>>>> meta-data
>>>> and there it hangs. Nothing gets imported.
>>>>
>>>> The online instance you can try to import into is
>>>> http://fs.dhis.dhmis.org/staging/
>>>>
>>>> When I try to import into a local instance, it seems to enter some kind
>>>> of endless loop based on the tomcat log:
>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>> at
>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>> at
>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at
>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>> at
>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at
>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>> at
>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>> at
>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at
>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>> at
>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>> at java.lang.String.valueOf(Unknown Source)
>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>> at
>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>
>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Calle,
>>>>>
>>>>> The commit you've replied to implements (but not fully, therefore
>>>>> 'WIP', there's another commit following it pretty soon after) better error
>>>>> reporting feedback to the user through the interface. I.e. not having to
>>>>> see the stacktrace on the server to determine if and why something went
>>>>> wrong. I believe you have been among the people requesting this too :)
>>>>>
>>>>> Also, as you're referencing, we've made some other changes to GML
>>>>> import. In particular the importer will now match on uid, code or name, as
>>>>> oppsed to the former mode of matching only on name. This is not reflected
>>>>> in the UI as it is a backend enhancement only, and does not change the
>>>>> import options at all. It IS, however, documented and ready for you to read
>>>>> about. Short version: it matches on a prioritized list of uid -> code ->
>>>>> name. Long version:
>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html
>>>>>
>>>>> Be adviced that the commit you've referenced is towards development
>>>>> trunk, and is not backported to 2.19, the match on multiple identifiers,
>>>>> however, was introduced in 2.19 and should work in current 2.19 stable. If
>>>>> you're having problems with it you can send me a full stack trace and
>>>>> preferably the GML input data.
>>>>>
>>>>>
>>>>>
>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm unable to import GML files using 2.19 (the same files import fine
>>>>>> in 2.18) - the import simply hangs (endless rolling progress bar icon).
>>>>>>
>>>>>> Will the above bug-fix sort out that problem - I do not understand
>>>>>> what the above actually means, or if the GML format has changed in some way.
>>>>>>
>>>>>> Note also that the final 2.19 release was supposed to include a more
>>>>>> diversified GML import - but I cannot see WHERE to do that (I still see the
>>>>>> same two buttons: choose file and import, either dry run or not)
>>>>>>
>>>>>> Regards
>>>>>> Calle
>>>>>>
>>>>>> On 5 June 2015 at 14:57, <noreply@xxxxxxxxxxxxx> wrote:
>>>>>>
>>>>>>> ------------------------------------------------------------
>>>>>>> revno: 19296
>>>>>>> committer: Halvdan Hoem Grelland <halvdanhg@xxxxxxxxx>
>>>>>>> branch nick: dhis2
>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200
>>>>>>> message:
>>>>>>> WIP proper error reporting on failed GML parsing during import
>>>>>>> added:
>>>>>>>
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>> modified:
>>>>>>>
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>
>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> lp:dhis2
>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>>>>
>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>>>>> To unsubscribe from this branch go to
>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>> 2015-06-03 15:00:19 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -32,6 +32,8 @@
>>>>>>> import com.google.common.base.Strings;
>>>>>>> import com.google.common.collect.Iterators;
>>>>>>> import com.google.common.collect.Maps;
>>>>>>> +import org.apache.commons.io.IOUtils;
>>>>>>> +import org.hibernate.Hibernate;
>>>>>>> import org.hisp.dhis.common.IdentifiableObjectManager;
>>>>>>> import org.hisp.dhis.common.IdentifiableProperty;
>>>>>>> import org.hisp.dhis.common.MergeStrategy;
>>>>>>> @@ -88,13 +90,91 @@
>>>>>>> // GmlImportService implementation
>>>>>>> //
>>>>>>> -------------------------------------------------------------------------
>>>>>>>
>>>>>>> + @Transactional
>>>>>>> + @Override
>>>>>>> + public GmlPreProcessingResult preProcessGml( InputStream
>>>>>>> inputStream )
>>>>>>> + {
>>>>>>> + InputStream dxfStream = null;
>>>>>>> + MetaData metaData = null;
>>>>>>> +
>>>>>>> + try
>>>>>>> + {
>>>>>>> + dxfStream = transformGml( inputStream );
>>>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>>>> MetaData.class );
>>>>>>> + }
>>>>>>> + catch ( IOException | TransformerException e )
>>>>>>> + {
>>>>>>> + return GmlPreProcessingResult.failure( e );
>>>>>>> + }
>>>>>>> + finally
>>>>>>> + {
>>>>>>> + IOUtils.closeQuietly( dxfStream );
>>>>>>> + }
>>>>>>> +
>>>>>>> + Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>> +
>>>>>>> + matchAndFilterOnIdentifiers(
>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
>>>>>>> +
>>>>>>> + Map<String, OrganisationUnit> persistedUidMap =
>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(), IdentifiableProperty.UID );
>>>>>>> + Map<String, OrganisationUnit> persistedCodeMap =
>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
>>>>>>> + Map<String, OrganisationUnit> persistedNameMap =
>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
>>>>>>> +
>>>>>>> + Iterator<OrganisationUnit> persistedIterator =
>>>>>>> Iterators.concat( persistedUidMap.values().iterator(),
>>>>>>> + persistedCodeMap.values().iterator(),
>>>>>>> persistedNameMap.values().iterator() );
>>>>>>> +
>>>>>>> + while ( persistedIterator.hasNext() )
>>>>>>> + {
>>>>>>> + OrganisationUnit persisted = persistedIterator.next(),
>>>>>>> imported = null;
>>>>>>> +
>>>>>>> + if ( !Strings.isNullOrEmpty( persisted.getUid() ) &&
>>>>>>> uidMap.containsKey( persisted.getUid() ) )
>>>>>>> + {
>>>>>>> + imported = uidMap.get( persisted.getUid() );
>>>>>>> + }
>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getCode() )
>>>>>>> && codeMap.containsKey( persisted.getCode() ) )
>>>>>>> + {
>>>>>>> + imported = codeMap.get( persisted.getCode() );
>>>>>>> + }
>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getName() )
>>>>>>> && nameMap.containsKey( persisted.getName() ) )
>>>>>>> + {
>>>>>>> + imported = nameMap.get( persisted.getName() );
>>>>>>> + }
>>>>>>> +
>>>>>>> + if ( imported == null || imported.getCoordinates() ==
>>>>>>> null || imported.getFeatureType() == null )
>>>>>>> + {
>>>>>>> + continue; // Failed to dereference a persisted
>>>>>>> entity for this org unit or geo data incomplete/missing, therefore ignore
>>>>>>> + }
>>>>>>> +
>>>>>>> + mergeNonGeoData( persisted, imported );
>>>>>>> + }
>>>>>>> +
>>>>>>> + return GmlPreProcessingResult.success( metaData );
>>>>>>> + }
>>>>>>> +
>>>>>>> @Override
>>>>>>> public MetaData fromGml( InputStream inputStream )
>>>>>>> throws IOException, TransformerException
>>>>>>> {
>>>>>>> - InputStream dxfStream = transformGml( inputStream );
>>>>>>> - MetaData metaData = renderService.fromXml( dxfStream,
>>>>>>> MetaData.class );
>>>>>>> - dxfStream.close();
>>>>>>> + InputStream dxfStream;
>>>>>>> + MetaData metaData;
>>>>>>> +
>>>>>>> + try
>>>>>>> + {
>>>>>>> + dxfStream = transformGml( inputStream );
>>>>>>> + }
>>>>>>> + catch (Exception e)
>>>>>>> + {
>>>>>>> + dxfStream = null;
>>>>>>> + }
>>>>>>> +
>>>>>>> + if(dxfStream != null)
>>>>>>> + {
>>>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>>>> MetaData.class );
>>>>>>> + dxfStream.close();
>>>>>>> + }
>>>>>>> + else
>>>>>>> + {
>>>>>>> + return null;
>>>>>>> + }
>>>>>>>
>>>>>>> Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>
>>>>>>> @@ -143,6 +223,7 @@
>>>>>>> importService.importMetaData( userUid, fromGml( inputStream
>>>>>>> ), importOptions, taskId );
>>>>>>> }
>>>>>>>
>>>>>>> + @Transactional
>>>>>>> @Override
>>>>>>> public void importGml( MetaData metaData, String userUid,
>>>>>>> ImportOptions importOptions, TaskId taskId )
>>>>>>> {
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>> 2015-06-03 15:00:19 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -62,6 +62,24 @@
>>>>>>> throws IOException, TransformerException;
>>>>>>>
>>>>>>> /**
>>>>>>> + * Pre-process a GML document. The process, in short, entails
>>>>>>> the following:
>>>>>>> + * <ol>
>>>>>>> + * <li>Parse the GML payload and transform it into DXF2
>>>>>>> format</li>
>>>>>>> + * <li>Get the given identifiers (uid, code or name) from
>>>>>>> the parsed payload and fetch
>>>>>>> + * the corresponding entities from the DB</li>
>>>>>>> + * <li>Merge the geospatial data given in the input GML
>>>>>>> into DB entities and return</li>
>>>>>>> + * </ol>
>>>>>>> + *
>>>>>>> + * The result of this process in returned in a {@link
>>>>>>> GmlPreProcessingResult} which
>>>>>>> + * encapsulates the returned {@link MetaData} object or the
>>>>>>> exception in cause of parse
>>>>>>> + * failure due to IO errors or malformed input.
>>>>>>> + *
>>>>>>> + * @param gmlInputStream the InputStream providing the GML
>>>>>>> input.
>>>>>>> + * @return a GmlPreProcessingResult representing the end result
>>>>>>> of the process.
>>>>>>> + */
>>>>>>> + GmlPreProcessingResult preProcessGml( InputStream
>>>>>>> gmlInputStream );
>>>>>>> +
>>>>>>> + /**
>>>>>>> * Imports GML data and merges the geospatial data updates into
>>>>>>> the database.
>>>>>>> * See {@link #fromGml(InputStream)} for details on the
>>>>>>> underlying process.
>>>>>>> *
>>>>>>>
>>>>>>> === added file
>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>> 1970-01-01 00:00:00 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -0,0 +1,70 @@
>>>>>>> +package org.hisp.dhis.dxf2.gml;
>>>>>>> +
>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData;
>>>>>>> +
>>>>>>> +import java.io.InputStream;
>>>>>>> +
>>>>>>> +/**
>>>>>>> + * Wraps the result of {@link
>>>>>>> GmlImportService#preProcessGml(InputStream)}.
>>>>>>> + * This is necessary when performing GML import on a context where
>>>>>>> exceptions
>>>>>>> + * due to malformed input cannot be caught, thus leaving the user
>>>>>>> uninformed of
>>>>>>> + * the error. This class will wrap the failure and provide the
>>>>>>> Throwable to the
>>>>>>> + * consuming class on error or the resulting MetaData object on
>>>>>>> success.
>>>>>>> + *
>>>>>>> + * @author Halvdan Hoem grelland <halvdanhg@xxxxxxxxx>
>>>>>>> + */
>>>>>>> +public final class GmlPreProcessingResult
>>>>>>> +{
>>>>>>> + private boolean isSuccess;
>>>>>>> + private MetaData resultMetaData;
>>>>>>> + private Throwable throwable;
>>>>>>> +
>>>>>>> + private GmlPreProcessingResult(){}
>>>>>>> +
>>>>>>> + public static GmlPreProcessingResult success( MetaData
>>>>>>> resultMetaData ) {
>>>>>>> + GmlPreProcessingResult result = new
>>>>>>> GmlPreProcessingResult();
>>>>>>> + result.setResultMetaData( resultMetaData );
>>>>>>> + result.setSuccess( true );
>>>>>>> +
>>>>>>> + return result;
>>>>>>> + }
>>>>>>> +
>>>>>>> + public static GmlPreProcessingResult failure( Throwable
>>>>>>> throwable )
>>>>>>> + {
>>>>>>> + GmlPreProcessingResult result = new
>>>>>>> GmlPreProcessingResult();
>>>>>>> + result.setThrowable( throwable );
>>>>>>> + result.setSuccess( false );
>>>>>>> +
>>>>>>> + return result;
>>>>>>> + }
>>>>>>> +
>>>>>>> + private void setSuccess( boolean isSuccess )
>>>>>>> + {
>>>>>>> + this.isSuccess = isSuccess;
>>>>>>> + }
>>>>>>> +
>>>>>>> + public boolean isSuccess()
>>>>>>> + {
>>>>>>> + return isSuccess;
>>>>>>> + }
>>>>>>> +
>>>>>>> + private void setResultMetaData( MetaData resultMetaData )
>>>>>>> + {
>>>>>>> + this.resultMetaData = resultMetaData;
>>>>>>> + }
>>>>>>> +
>>>>>>> + public MetaData getResultMetaData()
>>>>>>> + {
>>>>>>> + return resultMetaData;
>>>>>>> + }
>>>>>>> +
>>>>>>> + private void setThrowable( Throwable throwable )
>>>>>>> + {
>>>>>>> + this.throwable = throwable;
>>>>>>> + }
>>>>>>> +
>>>>>>> + public Throwable getThrowable()
>>>>>>> + {
>>>>>>> + return throwable;
>>>>>>> + }
>>>>>>> +}
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>> 2015-01-17 07:41:26 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -117,4 +117,6 @@
>>>>>>> assertEquals( 1, units.get(
>>>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
>>>>>>> assertEquals( 76, units.get( "Forskningsparken"
>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
>>>>>>> }
>>>>>>> +
>>>>>>> + // TODO Add test for GmlImportService#preProcessGml(InputStream)
>>>>>>> }
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>> 2015-05-30 13:36:07 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -77,7 +77,7 @@
>>>>>>> {
>>>>>>> taskExecutor.execute( task );
>>>>>>> }
>>>>>>> -
>>>>>>> +
>>>>>>> @Override
>>>>>>> public boolean scheduleTask( String key, Runnable task, String
>>>>>>> cronExpr )
>>>>>>> {
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>> 2015-05-28 16:10:07 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -187,7 +187,7 @@
>>>>>>> }
>>>>>>> else if ( "gml".equals( importFormat ) )
>>>>>>> {
>>>>>>> - scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>> userId, gmlImportService, importOptions, in, taskId ) );
>>>>>>> + scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) );
>>>>>>> }
>>>>>>> else if ( "json".equals( importFormat ) || "xml".equals(
>>>>>>> importFormat ) )
>>>>>>> {
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java'
>>>>>>> ---
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>> 2015-04-11 14:06:51 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -28,14 +28,18 @@
>>>>>>> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>>>>> */
>>>>>>>
>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils;
>>>>>>> import org.apache.commons.logging.Log;
>>>>>>> import org.apache.commons.logging.LogFactory;
>>>>>>> import org.hisp.dhis.dxf2.common.ImportOptions;
>>>>>>> import org.hisp.dhis.dxf2.gml.GmlImportService;
>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
>>>>>>> import org.hisp.dhis.scheduling.TaskId;
>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel;
>>>>>>> +import org.hisp.dhis.system.notification.Notifier;
>>>>>>> +import org.springframework.web.util.HtmlUtils;
>>>>>>> +import org.xml.sax.SAXParseException;
>>>>>>>
>>>>>>> -import javax.xml.transform.TransformerException;
>>>>>>> -import java.io.IOException;
>>>>>>> import java.io.InputStream;
>>>>>>>
>>>>>>> /**
>>>>>>> @@ -50,12 +54,15 @@
>>>>>>>
>>>>>>> private String userUid;
>>>>>>>
>>>>>>> +
>>>>>>> //
>>>>>>> -------------------------------------------------------------------------
>>>>>>> // Dependencies
>>>>>>> //
>>>>>>> -------------------------------------------------------------------------
>>>>>>>
>>>>>>> private GmlImportService gmlImportService;
>>>>>>>
>>>>>>> + private Notifier notifier;
>>>>>>> +
>>>>>>> private ImportOptions importOptions;
>>>>>>>
>>>>>>> private InputStream inputStream;
>>>>>>> @@ -64,11 +71,12 @@
>>>>>>> // Constructors
>>>>>>> //
>>>>>>> -------------------------------------------------------------------------
>>>>>>>
>>>>>>> - public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>>>> gmlImportService,
>>>>>>> + public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>>>> gmlImportService, Notifier notifier,
>>>>>>> ImportOptions importOptions, InputStream inputStream,
>>>>>>> TaskId taskId )
>>>>>>> {
>>>>>>> this.userUid = userUid;
>>>>>>> this.gmlImportService = gmlImportService;
>>>>>>> + this.notifier = notifier;
>>>>>>> this.importOptions = importOptions;
>>>>>>> this.inputStream = inputStream;
>>>>>>> this.taskId = taskId;
>>>>>>> @@ -83,15 +91,37 @@
>>>>>>> {
>>>>>>> importOptions.setImportStrategy( "update" ); // Force
>>>>>>> update only for GML import
>>>>>>>
>>>>>>> - try
>>>>>>> - {
>>>>>>> - gmlImportService.importGml( inputStream, userUid,
>>>>>>> importOptions, taskId );
>>>>>>> - }
>>>>>>> - catch ( IOException | TransformerException e )
>>>>>>> - {
>>>>>>> - log.error( "Unable to read GML data from input stream",
>>>>>>> e );
>>>>>>> -
>>>>>>> - throw new RuntimeException( "Failed to parse GML input
>>>>>>> stream", e );
>>>>>>> - }
>>>>>>> + GmlPreProcessingResult gmlPreProcessingResult =
>>>>>>> gmlImportService.preProcessGml( inputStream );
>>>>>>> +
>>>>>>> + if ( !gmlPreProcessingResult.isSuccess() )
>>>>>>> + {
>>>>>>> + Throwable throwable =
>>>>>>> gmlPreProcessingResult.getThrowable();
>>>>>>> + String message = createErrorMessage( throwable );
>>>>>>> +
>>>>>>> + notifier.notify( taskId, NotificationLevel.ERROR,
>>>>>>> message, false );
>>>>>>> + log.error( "GML import failed: " + message, throwable );
>>>>>>> +
>>>>>>> + return;
>>>>>>> + }
>>>>>>> +
>>>>>>> + gmlImportService.importGml(
>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId
>>>>>>> );
>>>>>>> + }
>>>>>>> +
>>>>>>> + private String createErrorMessage( Throwable throwable )
>>>>>>> + {
>>>>>>> + String message = "";
>>>>>>> + Throwable rootThrowable = ExceptionUtils.getRootCause(
>>>>>>> throwable );
>>>>>>> +
>>>>>>> + if ( rootThrowable instanceof SAXParseException )
>>>>>>> + {
>>>>>>> + SAXParseException e = (SAXParseException) rootThrowable;
>>>>>>> + message += "Syntax error on line " + e.getLineNumber()
>>>>>>> + ". " + e.getMessage();
>>>>>>> + }
>>>>>>> + else
>>>>>>> + {
>>>>>>> + message += rootThrowable.getMessage();
>>>>>>> + }
>>>>>>> +
>>>>>>> + return HtmlUtils.htmlEscape( message );
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties'
>>>>>>> ---
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>> 2015-02-20 11:17:31 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -300,6 +300,7 @@
>>>>>>> ignored=Ignored
>>>>>>> conflicts=Conflicts
>>>>>>> no_conflicts_found=No conflicts found
>>>>>>> +no_import_summary_available=No summary available
>>>>>>> type=Type
>>>>>>> count=Count
>>>>>>> export_as_xml=Export as XML
>>>>>>>
>>>>>>> === modified file
>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm'
>>>>>>> ---
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>> 2013-03-21 09:15:24 +0000
>>>>>>> +++
>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>> @@ -1,4 +1,6 @@
>>>>>>> <h3>$i18n.getString( "import_summary" )</h3>
>>>>>>> +
>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML import
>>>>>>> (pre-processing fails)
>>>>>>> <h4>$i18n.getString( "import_count" )</h4>
>>>>>>>
>>>>>>> $summary.importCount.imported Imported<br/>
>>>>>>> @@ -54,3 +56,6 @@
>>>>>>> #else
>>>>>>> <p>$i18n.getString( "no_conflicts_found" )</p>
>>>>>>> #end
>>>>>>> +#else
>>>>>>> + $i18n.getString( "no_summary_available" )
>>>>>>> +#end
>>>>>>> \ No newline at end of file
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>> Calle Hedberg
>>>>>>
>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>
>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>
>>>>>> Cell: +27-82-853-5352
>>>>>>
>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>
>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>
>>>>>> Skype: calle_hedberg
>>>>>>
>>>>>> *******************************************
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *******************************************
>>>>
>>>> Calle Hedberg
>>>>
>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>
>>>> Tel/fax (home): +27-21-685-6472
>>>>
>>>> Cell: +27-82-853-5352
>>>>
>>>> Iridium SatPhone: +8816-315-19274
>>>>
>>>> Email: calle.hedberg@xxxxxxxxx
>>>>
>>>> Skype: calle_hedberg
>>>>
>>>> *******************************************
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19274
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
>
--
*******************************************
Calle Hedberg
46D Alma Road, 7700 Rosebank, SOUTH AFRICA
Tel/fax (home): +27-21-685-6472
Cell: +27-82-853-5352
Iridium SatPhone: +8816-315-19274
Email: calle.hedberg@xxxxxxxxx
Skype: calle_hedberg
*******************************************
Follow ups
References
