dhis2-devs team mailing list archive
-
dhis2-devs team
-
Mailing list archive
-
Message #38258
Re: [Branch ~dhis2-devs-core/dhis2/trunk] Rev 19296: WIP proper error reporting on failed GML parsing during import
I would recommend running your shapefile through Mapshaper to simplify it
before conversion to GML
On Fri, Jun 26, 2015 at 2:17 PM, Dan Cocos <dan@xxxxxxxxxxxx> wrote:
> I’ve seen this error in the past on import of GML the cause was an Org
> Unit had a lot of coordinates (as in greater than 200 it was every turn of
> Lake Victoria) and this caused the importer to hang.
>
> On Jun 26, 2015, at 5:17 AM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
> wrote:
>
> Hi,
>
> I ran it again (revision 19247) and pulled out the first part of the log
> error:
>
> * INFO 2015-06-26 11:04:18,338 User 'Calle_Hedberg' started import at Fri
> Jun 26 11:04:18 CAT 2015 (DefaultImportService.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:18,348 [Level: INFO, category: METADATA_IMPORT,
> time: Fri Jun 26 11:04:18 CAT 2015, message: Importing meta-data]
> (InMemoryNotifier.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:18,348 Building object-bridge maps (preheatCache:
> true). (DefaultObjectBridge.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:21,141 Building object-bridge maps took 2.79
> seconds. (DefaultObjectBridge.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:21,141 [Level: INFO, category: METADATA_IMPORT,
> time: Fri Jun 26 11:04:21 CAT 2015, message: Importing 1 OrganisationUnits]
> (InMemoryNotifier.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
> type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
> type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
> * INFO 2015-06-26 11:04:21,151 Deleted objects associated with object of
> type AttributeValue (DefaultDeletionManager.java [taskScheduler-1])
> * ERROR 2015-06-26 11:04:21,432 Unexpected error occurred in scheduled
> task. (TaskUtils.java [taskScheduler-1])
> java.lang.StackOverflowError
> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
> at java.util.regex.Pattern$BmpCharProperty.<init>(Unknown Source)
> at java.util.regex.Pattern$BitClass.<init>(Unknown Source)
> at java.util.regex.Pattern.clazz(Unknown Source)
> at java.util.regex.Pattern.sequence(Unknown Source)
> at java.util.regex.Pattern.expr(Unknown Source)
> at java.util.regex.Pattern.compile(Unknown Source)
> at java.util.regex.Pattern.<init>(Unknown Source)
> at java.util.regex.Pattern.compile(Unknown Source)
> at java.lang.String.replaceAll(Unknown Source)
> at com.google.common.base.MoreObjects.simpleName(MoreObjects.java:134)
> at com.google.common.base.MoreObjects.toStringHelper(MoreObjects.java:95)
> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:396)
> at sun.reflect.GeneratedMethodAccessor626.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:196)
> at
> org.hisp.dhis.attribute.Attribute_$$_jvst804_10.toString(Attribute_$$_jvst804_10.java)
> at java.lang.String.valueOf(Unknown Source)
> at java.lang.StringBuilder.append(Unknown Source)
> at org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:122)
> at java.lang.String.valueOf(Unknown Source)
> at java.lang.StringBuilder.append(Unknown Source)
> at java.util.AbstractCollection.toString(Unknown Source)
> at
> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
> at java.lang.String.valueOf(Unknown Source)
> at java.lang.StringBuilder.append(Unknown Source)
> at
> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>
> Any ideas?
>
> Regards
> Calle
>
> On 26 June 2015 at 10:12, Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:
>
>> Hi,
>>
>> Firstly, yes we are using a custom revision - although it is SUPPOSED to
>> be based on a 2.19 version from a few days ago. I will check that.
>>
>> BUT I have tried the import using revision 19247 too, and the
>> problem/behaviour is exactly the same - it hangs. Whereas if I import into
>> an older version of the same provincial instance that still runs on 2.18,
>> it imports fine.
>>
>> The fact that the same GML file imports in the Sierra Leone demo - I
>> cannot explain that, and the tomcat log only shows that
>> Attribute.toString() looping...
>>
>> Can I create a stripped down version of the instance and upload it for
>> you to test?
>>
>> Regards
>> Calle
>>
>>
>>
>> On 26 June 2015 at 04:09, Morten Olav Hansen <mortenoh@xxxxxxxxx> wrote:
>>
>>> It seems to be looping in Attribute.toString(), I remember this was an
>>> issue at some point.. but that was a long long time ago.. so maybe you need
>>> a revision bump
>>>
>>> --
>>> Morten
>>>
>>> On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>>> wrote:
>>>
>>>> I tried your example GML on both our demo servers (dev on 2.20 and demo
>>>> on 2.19, same rev as your server) and it works just fine.
>>>>
>>>> If you're experiencing it hanging and not finishing you might be
>>>> experiencing some kind of performance and/or memory issue. The coordinate
>>>> set is fairly large (though it works well on our demos, but does take some
>>>> time to finish). You can verify by trying yourself to create the orgunit
>>>> (with the exact name) and import the GML on the Sierra Leone demo.
>>>>
>>>> The fact that the dry run seems to "work" for you (as well as the stack
>>>> trace you provided) also points at the issue not residing in the GML
>>>> processing but rather in the actual import stage (the GML importer uses the
>>>> metadata importer as it's backend).
>>>>
>>>> Another thing: The instance you linked to is using a revision of 16783
>>>> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are
>>>> you using a custom build? Does it contain customizations to the core in any
>>>> way? That would also be a possible source of issues, obviously.
>>>>
>>>> You could try deploying this war
>>>> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war
>>>> and replicate the issue using the za South Africa orgunit.
>>>>
>>>> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>
>>>>> Halvdan,
>>>>>
>>>>> The GML files I'm trying to import using 2.19 is the same GML files
>>>>> I've been using successfully for 2.17, 2.18 (so it's using NAME). Something
>>>>> in the import has changed:
>>>>>
>>>>> I have attached the small GML file I'm trying to import - it has the
>>>>> international boundary of SA, so there is only ONE GIS record (top node in
>>>>> all our OrgHierarchies). It imports with no problems in 2.18.
>>>>>
>>>>> In 2.19, if I choose YES for dry run, it will complete with usual
>>>>> import summary etc.
>>>>>
>>>>> If I choose NO for dry run, I get
>>>>> 2015-06-25 22:18:15Importing 1 OrganisationUnits 2015-06-25 22:18:11Importing
>>>>> meta-data
>>>>> and there it hangs. Nothing gets imported.
>>>>>
>>>>> The online instance you can try to import into is
>>>>> http://fs.dhis.dhmis.org/staging/
>>>>>
>>>>> When I try to import into a local instance, it seems to enter some
>>>>> kind of endless loop based on the tomcat log:
>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>> at
>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>> at
>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>>> at
>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>> at
>>>>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>>>>> at
>>>>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at java.util.AbstractCollection.toString(Unknown Source)
>>>>> at
>>>>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>>>>> at java.lang.String.valueOf(Unknown Source)
>>>>> at java.lang.StringBuilder.append(Unknown Source)
>>>>> at
>>>>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>>>>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>>>>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>>>>> at java.lang.reflect.Method.invoke(Unknown Source)
>>>>>
>>>>> On 25 June 2015 at 18:16, Halvdan Grelland <halvdanhg@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Calle,
>>>>>>
>>>>>> The commit you've replied to implements (but not fully, therefore
>>>>>> 'WIP', there's another commit following it pretty soon after) better error
>>>>>> reporting feedback to the user through the interface. I.e. not having to
>>>>>> see the stacktrace on the server to determine if and why something went
>>>>>> wrong. I believe you have been among the people requesting this too :)
>>>>>>
>>>>>> Also, as you're referencing, we've made some other changes to GML
>>>>>> import. In particular the importer will now match on uid, code or name, as
>>>>>> oppsed to the former mode of matching only on name. This is not reflected
>>>>>> in the UI as it is a backend enhancement only, and does not change the
>>>>>> import options at all. It IS, however, documented and ready for you to read
>>>>>> about. Short version: it matches on a prioritized list of uid -> code ->
>>>>>> name. Long version:
>>>>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html
>>>>>>
>>>>>> Be adviced that the commit you've referenced is towards development
>>>>>> trunk, and is not backported to 2.19, the match on multiple identifiers,
>>>>>> however, was introduced in 2.19 and should work in current 2.19 stable. If
>>>>>> you're having problems with it you can send me a full stack trace and
>>>>>> preferably the GML input data.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm unable to import GML files using 2.19 (the same files import
>>>>>>> fine in 2.18) - the import simply hangs (endless rolling progress bar icon).
>>>>>>>
>>>>>>> Will the above bug-fix sort out that problem - I do not understand
>>>>>>> what the above actually means, or if the GML format has changed in some way.
>>>>>>>
>>>>>>> Note also that the final 2.19 release was supposed to include a more
>>>>>>> diversified GML import - but I cannot see WHERE to do that (I still see the
>>>>>>> same two buttons: choose file and import, either dry run or not)
>>>>>>>
>>>>>>> Regards
>>>>>>> Calle
>>>>>>>
>>>>>>> On 5 June 2015 at 14:57, <noreply@xxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> ------------------------------------------------------------
>>>>>>>> revno: 19296
>>>>>>>> committer: Halvdan Hoem Grelland <halvdanhg@xxxxxxxxx>
>>>>>>>> branch nick: dhis2
>>>>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200
>>>>>>>> message:
>>>>>>>> WIP proper error reporting on failed GML parsing during import
>>>>>>>> added:
>>>>>>>>
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>> modified:
>>>>>>>>
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>>
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>>
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> lp:dhis2
>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>>>>>
>>>>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>>>>>> To unsubscribe from this branch go to
>>>>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>> 2015-06-03 15:00:19 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -32,6 +32,8 @@
>>>>>>>> import com.google.common.base.Strings;
>>>>>>>> import com.google.common.collect.Iterators;
>>>>>>>> import com.google.common.collect.Maps;
>>>>>>>> +import org.apache.commons.io.IOUtils;
>>>>>>>> +import org.hibernate.Hibernate;
>>>>>>>> import org.hisp.dhis.common.IdentifiableObjectManager;
>>>>>>>> import org.hisp.dhis.common.IdentifiableProperty;
>>>>>>>> import org.hisp.dhis.common.MergeStrategy;
>>>>>>>> @@ -88,13 +90,91 @@
>>>>>>>> // GmlImportService implementation
>>>>>>>> //
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> + @Transactional
>>>>>>>> + @Override
>>>>>>>> + public GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>> inputStream )
>>>>>>>> + {
>>>>>>>> + InputStream dxfStream = null;
>>>>>>>> + MetaData metaData = null;
>>>>>>>> +
>>>>>>>> + try
>>>>>>>> + {
>>>>>>>> + dxfStream = transformGml( inputStream );
>>>>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>>>>> MetaData.class );
>>>>>>>> + }
>>>>>>>> + catch ( IOException | TransformerException e )
>>>>>>>> + {
>>>>>>>> + return GmlPreProcessingResult.failure( e );
>>>>>>>> + }
>>>>>>>> + finally
>>>>>>>> + {
>>>>>>>> + IOUtils.closeQuietly( dxfStream );
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>> +
>>>>>>>> + matchAndFilterOnIdentifiers(
>>>>>>>> metaData.getOrganisationUnits(), uidMap, codeMap, nameMap );
>>>>>>>> +
>>>>>>>> + Map<String, OrganisationUnit> persistedUidMap =
>>>>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(), IdentifiableProperty.UID );
>>>>>>>> + Map<String, OrganisationUnit> persistedCodeMap =
>>>>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
>>>>>>>> + Map<String, OrganisationUnit> persistedNameMap =
>>>>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
>>>>>>>> +
>>>>>>>> + Iterator<OrganisationUnit> persistedIterator =
>>>>>>>> Iterators.concat( persistedUidMap.values().iterator(),
>>>>>>>> + persistedCodeMap.values().iterator(),
>>>>>>>> persistedNameMap.values().iterator() );
>>>>>>>> +
>>>>>>>> + while ( persistedIterator.hasNext() )
>>>>>>>> + {
>>>>>>>> + OrganisationUnit persisted = persistedIterator.next(),
>>>>>>>> imported = null;
>>>>>>>> +
>>>>>>>> + if ( !Strings.isNullOrEmpty( persisted.getUid() ) &&
>>>>>>>> uidMap.containsKey( persisted.getUid() ) )
>>>>>>>> + {
>>>>>>>> + imported = uidMap.get( persisted.getUid() );
>>>>>>>> + }
>>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getCode()
>>>>>>>> ) && codeMap.containsKey( persisted.getCode() ) )
>>>>>>>> + {
>>>>>>>> + imported = codeMap.get( persisted.getCode() );
>>>>>>>> + }
>>>>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getName()
>>>>>>>> ) && nameMap.containsKey( persisted.getName() ) )
>>>>>>>> + {
>>>>>>>> + imported = nameMap.get( persisted.getName() );
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + if ( imported == null || imported.getCoordinates() ==
>>>>>>>> null || imported.getFeatureType() == null )
>>>>>>>> + {
>>>>>>>> + continue; // Failed to dereference a persisted
>>>>>>>> entity for this org unit or geo data incomplete/missing, therefore ignore
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + mergeNonGeoData( persisted, imported );
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + return GmlPreProcessingResult.success( metaData );
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> @Override
>>>>>>>> public MetaData fromGml( InputStream inputStream )
>>>>>>>> throws IOException, TransformerException
>>>>>>>> {
>>>>>>>> - InputStream dxfStream = transformGml( inputStream );
>>>>>>>> - MetaData metaData = renderService.fromXml( dxfStream,
>>>>>>>> MetaData.class );
>>>>>>>> - dxfStream.close();
>>>>>>>> + InputStream dxfStream;
>>>>>>>> + MetaData metaData;
>>>>>>>> +
>>>>>>>> + try
>>>>>>>> + {
>>>>>>>> + dxfStream = transformGml( inputStream );
>>>>>>>> + }
>>>>>>>> + catch (Exception e)
>>>>>>>> + {
>>>>>>>> + dxfStream = null;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + if(dxfStream != null)
>>>>>>>> + {
>>>>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>>>>> MetaData.class );
>>>>>>>> + dxfStream.close();
>>>>>>>> + }
>>>>>>>> + else
>>>>>>>> + {
>>>>>>>> + return null;
>>>>>>>> + }
>>>>>>>>
>>>>>>>> Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>>>>
>>>>>>>> @@ -143,6 +223,7 @@
>>>>>>>> importService.importMetaData( userUid, fromGml(
>>>>>>>> inputStream ), importOptions, taskId );
>>>>>>>> }
>>>>>>>>
>>>>>>>> + @Transactional
>>>>>>>> @Override
>>>>>>>> public void importGml( MetaData metaData, String userUid,
>>>>>>>> ImportOptions importOptions, TaskId taskId )
>>>>>>>> {
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>> 2015-06-03 15:00:19 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -62,6 +62,24 @@
>>>>>>>> throws IOException, TransformerException;
>>>>>>>>
>>>>>>>> /**
>>>>>>>> + * Pre-process a GML document. The process, in short, entails
>>>>>>>> the following:
>>>>>>>> + * <ol>
>>>>>>>> + * <li>Parse the GML payload and transform it into DXF2
>>>>>>>> format</li>
>>>>>>>> + * <li>Get the given identifiers (uid, code or name) from
>>>>>>>> the parsed payload and fetch
>>>>>>>> + * the corresponding entities from the DB</li>
>>>>>>>> + * <li>Merge the geospatial data given in the input GML
>>>>>>>> into DB entities and return</li>
>>>>>>>> + * </ol>
>>>>>>>> + *
>>>>>>>> + * The result of this process in returned in a {@link
>>>>>>>> GmlPreProcessingResult} which
>>>>>>>> + * encapsulates the returned {@link MetaData} object or the
>>>>>>>> exception in cause of parse
>>>>>>>> + * failure due to IO errors or malformed input.
>>>>>>>> + *
>>>>>>>> + * @param gmlInputStream the InputStream providing the GML
>>>>>>>> input.
>>>>>>>> + * @return a GmlPreProcessingResult representing the end
>>>>>>>> result of the process.
>>>>>>>> + */
>>>>>>>> + GmlPreProcessingResult preProcessGml( InputStream
>>>>>>>> gmlInputStream );
>>>>>>>> +
>>>>>>>> + /**
>>>>>>>> * Imports GML data and merges the geospatial data updates
>>>>>>>> into the database.
>>>>>>>> * See {@link #fromGml(InputStream)} for details on the
>>>>>>>> underlying process.
>>>>>>>> *
>>>>>>>>
>>>>>>>> === added file
>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>> 1970-01-01 00:00:00 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -0,0 +1,70 @@
>>>>>>>> +package org.hisp.dhis.dxf2.gml;
>>>>>>>> +
>>>>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData;
>>>>>>>> +
>>>>>>>> +import java.io.InputStream;
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * Wraps the result of {@link
>>>>>>>> GmlImportService#preProcessGml(InputStream)}.
>>>>>>>> + * This is necessary when performing GML import on a context where
>>>>>>>> exceptions
>>>>>>>> + * due to malformed input cannot be caught, thus leaving the user
>>>>>>>> uninformed of
>>>>>>>> + * the error. This class will wrap the failure and provide the
>>>>>>>> Throwable to the
>>>>>>>> + * consuming class on error or the resulting MetaData object on
>>>>>>>> success.
>>>>>>>> + *
>>>>>>>> + * @author Halvdan Hoem grelland <halvdanhg@xxxxxxxxx>
>>>>>>>> + */
>>>>>>>> +public final class GmlPreProcessingResult
>>>>>>>> +{
>>>>>>>> + private boolean isSuccess;
>>>>>>>> + private MetaData resultMetaData;
>>>>>>>> + private Throwable throwable;
>>>>>>>> +
>>>>>>>> + private GmlPreProcessingResult(){}
>>>>>>>> +
>>>>>>>> + public static GmlPreProcessingResult success( MetaData
>>>>>>>> resultMetaData ) {
>>>>>>>> + GmlPreProcessingResult result = new
>>>>>>>> GmlPreProcessingResult();
>>>>>>>> + result.setResultMetaData( resultMetaData );
>>>>>>>> + result.setSuccess( true );
>>>>>>>> +
>>>>>>>> + return result;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + public static GmlPreProcessingResult failure( Throwable
>>>>>>>> throwable )
>>>>>>>> + {
>>>>>>>> + GmlPreProcessingResult result = new
>>>>>>>> GmlPreProcessingResult();
>>>>>>>> + result.setThrowable( throwable );
>>>>>>>> + result.setSuccess( false );
>>>>>>>> +
>>>>>>>> + return result;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + private void setSuccess( boolean isSuccess )
>>>>>>>> + {
>>>>>>>> + this.isSuccess = isSuccess;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + public boolean isSuccess()
>>>>>>>> + {
>>>>>>>> + return isSuccess;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + private void setResultMetaData( MetaData resultMetaData )
>>>>>>>> + {
>>>>>>>> + this.resultMetaData = resultMetaData;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + public MetaData getResultMetaData()
>>>>>>>> + {
>>>>>>>> + return resultMetaData;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + private void setThrowable( Throwable throwable )
>>>>>>>> + {
>>>>>>>> + this.throwable = throwable;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + public Throwable getThrowable()
>>>>>>>> + {
>>>>>>>> + return throwable;
>>>>>>>> + }
>>>>>>>> +}
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>> 2015-01-17 07:41:26 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -117,4 +117,6 @@
>>>>>>>> assertEquals( 1, units.get(
>>>>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
>>>>>>>> assertEquals( 76, units.get( "Forskningsparken"
>>>>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
>>>>>>>> }
>>>>>>>> +
>>>>>>>> + // TODO Add test for
>>>>>>>> GmlImportService#preProcessGml(InputStream)
>>>>>>>> }
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>> 2015-05-30 13:36:07 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -77,7 +77,7 @@
>>>>>>>> {
>>>>>>>> taskExecutor.execute( task );
>>>>>>>> }
>>>>>>>> -
>>>>>>>> +
>>>>>>>> @Override
>>>>>>>> public boolean scheduleTask( String key, Runnable task, String
>>>>>>>> cronExpr )
>>>>>>>> {
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>> 2015-05-28 16:10:07 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -187,7 +187,7 @@
>>>>>>>> }
>>>>>>>> else if ( "gml".equals( importFormat ) )
>>>>>>>> {
>>>>>>>> - scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>> userId, gmlImportService, importOptions, in, taskId ) );
>>>>>>>> + scheduler.executeTask( new ImportMetaDataGmlTask(
>>>>>>>> userId, gmlImportService, notifier, importOptions, in, taskId ) );
>>>>>>>> }
>>>>>>>> else if ( "json".equals( importFormat ) || "xml".equals(
>>>>>>>> importFormat ) )
>>>>>>>> {
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>> 2015-04-11 14:06:51 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -28,14 +28,18 @@
>>>>>>>> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>>>>>> */
>>>>>>>>
>>>>>>>> +import org.apache.commons.lang.exception.ExceptionUtils;
>>>>>>>> import org.apache.commons.logging.Log;
>>>>>>>> import org.apache.commons.logging.LogFactory;
>>>>>>>> import org.hisp.dhis.dxf2.common.ImportOptions;
>>>>>>>> import org.hisp.dhis.dxf2.gml.GmlImportService;
>>>>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
>>>>>>>> import org.hisp.dhis.scheduling.TaskId;
>>>>>>>> +import org.hisp.dhis.system.notification.NotificationLevel;
>>>>>>>> +import org.hisp.dhis.system.notification.Notifier;
>>>>>>>> +import org.springframework.web.util.HtmlUtils;
>>>>>>>> +import org.xml.sax.SAXParseException;
>>>>>>>>
>>>>>>>> -import javax.xml.transform.TransformerException;
>>>>>>>> -import java.io.IOException;
>>>>>>>> import java.io.InputStream;
>>>>>>>>
>>>>>>>> /**
>>>>>>>> @@ -50,12 +54,15 @@
>>>>>>>>
>>>>>>>> private String userUid;
>>>>>>>>
>>>>>>>> +
>>>>>>>> //
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>> // Dependencies
>>>>>>>> //
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> private GmlImportService gmlImportService;
>>>>>>>>
>>>>>>>> + private Notifier notifier;
>>>>>>>> +
>>>>>>>> private ImportOptions importOptions;
>>>>>>>>
>>>>>>>> private InputStream inputStream;
>>>>>>>> @@ -64,11 +71,12 @@
>>>>>>>> // Constructors
>>>>>>>> //
>>>>>>>> -------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> - public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>>>>> gmlImportService,
>>>>>>>> + public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>>>>> gmlImportService, Notifier notifier,
>>>>>>>> ImportOptions importOptions, InputStream inputStream,
>>>>>>>> TaskId taskId )
>>>>>>>> {
>>>>>>>> this.userUid = userUid;
>>>>>>>> this.gmlImportService = gmlImportService;
>>>>>>>> + this.notifier = notifier;
>>>>>>>> this.importOptions = importOptions;
>>>>>>>> this.inputStream = inputStream;
>>>>>>>> this.taskId = taskId;
>>>>>>>> @@ -83,15 +91,37 @@
>>>>>>>> {
>>>>>>>> importOptions.setImportStrategy( "update" ); // Force
>>>>>>>> update only for GML import
>>>>>>>>
>>>>>>>> - try
>>>>>>>> - {
>>>>>>>> - gmlImportService.importGml( inputStream, userUid,
>>>>>>>> importOptions, taskId );
>>>>>>>> - }
>>>>>>>> - catch ( IOException | TransformerException e )
>>>>>>>> - {
>>>>>>>> - log.error( "Unable to read GML data from input
>>>>>>>> stream", e );
>>>>>>>> -
>>>>>>>> - throw new RuntimeException( "Failed to parse GML input
>>>>>>>> stream", e );
>>>>>>>> - }
>>>>>>>> + GmlPreProcessingResult gmlPreProcessingResult =
>>>>>>>> gmlImportService.preProcessGml( inputStream );
>>>>>>>> +
>>>>>>>> + if ( !gmlPreProcessingResult.isSuccess() )
>>>>>>>> + {
>>>>>>>> + Throwable throwable =
>>>>>>>> gmlPreProcessingResult.getThrowable();
>>>>>>>> + String message = createErrorMessage( throwable );
>>>>>>>> +
>>>>>>>> + notifier.notify( taskId, NotificationLevel.ERROR,
>>>>>>>> message, false );
>>>>>>>> + log.error( "GML import failed: " + message, throwable
>>>>>>>> );
>>>>>>>> +
>>>>>>>> + return;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + gmlImportService.importGml(
>>>>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId
>>>>>>>> );
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + private String createErrorMessage( Throwable throwable )
>>>>>>>> + {
>>>>>>>> + String message = "";
>>>>>>>> + Throwable rootThrowable = ExceptionUtils.getRootCause(
>>>>>>>> throwable );
>>>>>>>> +
>>>>>>>> + if ( rootThrowable instanceof SAXParseException )
>>>>>>>> + {
>>>>>>>> + SAXParseException e = (SAXParseException)
>>>>>>>> rootThrowable;
>>>>>>>> + message += "Syntax error on line " + e.getLineNumber()
>>>>>>>> + ". " + e.getMessage();
>>>>>>>> + }
>>>>>>>> + else
>>>>>>>> + {
>>>>>>>> + message += rootThrowable.getMessage();
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + return HtmlUtils.htmlEscape( message );
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>> 2015-02-20 11:17:31 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -300,6 +300,7 @@
>>>>>>>> ignored=Ignored
>>>>>>>> conflicts=Conflicts
>>>>>>>> no_conflicts_found=No conflicts found
>>>>>>>> +no_import_summary_available=No summary available
>>>>>>>> type=Type
>>>>>>>> count=Count
>>>>>>>> export_as_xml=Export as XML
>>>>>>>>
>>>>>>>> === modified file
>>>>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm'
>>>>>>>> ---
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>> 2013-03-21 09:15:24 +0000
>>>>>>>> +++
>>>>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>>>> 2015-06-05 12:56:34 +0000
>>>>>>>> @@ -1,4 +1,6 @@
>>>>>>>> <h3>$i18n.getString( "import_summary" )</h3>
>>>>>>>> +
>>>>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML import
>>>>>>>> (pre-processing fails)
>>>>>>>> <h4>$i18n.getString( "import_count" )</h4>
>>>>>>>>
>>>>>>>> $summary.importCount.imported Imported<br/>
>>>>>>>> @@ -54,3 +56,6 @@
>>>>>>>> #else
>>>>>>>> <p>$i18n.getString( "no_conflicts_found" )</p>
>>>>>>>> #end
>>>>>>>> +#else
>>>>>>>> + $i18n.getString( "no_summary_available" )
>>>>>>>> +#end
>>>>>>>> \ No newline at end of file
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>> Calle Hedberg
>>>>>>>
>>>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>>>
>>>>>>> Tel/fax (home): +27-21-685-6472
>>>>>>>
>>>>>>> Cell: +27-82-853-5352
>>>>>>>
>>>>>>> Iridium SatPhone: +8816-315-19274
>>>>>>>
>>>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>>>
>>>>>>> Skype: calle_hedberg
>>>>>>>
>>>>>>> *******************************************
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *******************************************
>>>>>
>>>>> Calle Hedberg
>>>>>
>>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>>
>>>>> Tel/fax (home): +27-21-685-6472
>>>>>
>>>>> Cell: +27-82-853-5352
>>>>>
>>>>> Iridium SatPhone: +8816-315-19274
>>>>>
>>>>> Email: calle.hedberg@xxxxxxxxx
>>>>>
>>>>> Skype: calle_hedberg
>>>>>
>>>>> *******************************************
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472
>
> Cell: +27-82-853-5352
>
> Iridium SatPhone: +8816-315-19274
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>
--
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar
http://dhis2.org
Follow ups
References
