dhis2-devs team mailing list archive
-
dhis2-devs team
-
Mailing list archive
-
Message #38243
Re: [Branch ~dhis2-devs-core/dhis2/trunk] Rev 19296: WIP proper error reporting on failed GML parsing during import
It seems to be looping in Attribute.toString(), I remember this was an
issue at some point.. but that was a long long time ago.. so maybe you need
a revision bump
--
Morten
On Fri, Jun 26, 2015 at 6:52 AM, Halvdan Grelland <halvdanhg@xxxxxxxxx>
wrote:
> I tried your example GML on both our demo servers (dev on 2.20 and demo on
> 2.19, same rev as your server) and it works just fine.
>
> If you're experiencing it hanging and not finishing you might be
> experiencing some kind of performance and/or memory issue. The coordinate
> set is fairly large (though it works well on our demos, but does take some
> time to finish). You can verify by trying yourself to create the orgunit
> (with the exact name) and import the GML on the Sierra Leone demo.
>
> The fact that the dry run seems to "work" for you (as well as the stack
> trace you provided) also points at the issue not residing in the GML
> processing but rather in the actual import stage (the GML importer uses the
> metadata importer as it's backend).
>
> Another thing: The instance you linked to is using a revision of 16783
> and ver. 2.19, which is sept. 2014 (current is 2.19 is at 19247). Are you
> using a custom build? Does it contain customizations to the core in any
> way? That would also be a possible source of issues, obviously.
>
> You could try deploying this war
> https://apps.dhis2.org/ci/job/dhis-2.19/lastBuild/artifact/dhis-2/dhis-web/dhis-web-portal/target/dhis.war
> and replicate the issue using the za South Africa orgunit.
>
> 2015-06-26 0:35 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>
>> Halvdan,
>>
>> The GML files I'm trying to import using 2.19 is the same GML files I've
>> been using successfully for 2.17, 2.18 (so it's using NAME). Something in
>> the import has changed:
>>
>> I have attached the small GML file I'm trying to import - it has the
>> international boundary of SA, so there is only ONE GIS record (top node in
>> all our OrgHierarchies). It imports with no problems in 2.18.
>>
>> In 2.19, if I choose YES for dry run, it will complete with usual import
>> summary etc.
>>
>> If I choose NO for dry run, I get
>> 2015-06-25 22:18:15Importing 1 OrganisationUnits 2015-06-25 22:18:11Importing
>> meta-data
>> and there it hangs. Nothing gets imported.
>>
>> The online instance you can try to import into is
>> http://fs.dhis.dhmis.org/staging/
>>
>> When I try to import into a local instance, it seems to enter some kind
>> of endless loop based on the tomcat log:
>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at
>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>> at
>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at
>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at java.util.AbstractCollection.toString(Unknown Source)
>> at
>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at
>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at
>> org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:192)
>> at
>> org.hisp.dhis.attribute.Attribute_$$_javassist_16.toString(Attribute_$$_javassist_16.java)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at
>> org.hisp.dhis.attribute.AttributeValue.toString(AttributeValue.java:136)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at java.util.AbstractCollection.toString(Unknown Source)
>> at
>> org.hibernate.collection.internal.PersistentSet.toString(PersistentSet.java:327)
>> at java.lang.String.valueOf(Unknown Source)
>> at java.lang.StringBuilder.append(Unknown Source)
>> at
>> com.google.common.base.MoreObjects$ToStringHelper.toString(MoreObjects.java:359)
>> at org.hisp.dhis.attribute.Attribute.toString(Attribute.java:397)
>> at sun.reflect.GeneratedMethodAccessor723.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>>
>> On 25 June 2015 at 18:16, Halvdan Grelland <halvdanhg@xxxxxxxxx> wrote:
>>
>>> Hi Calle,
>>>
>>> The commit you've replied to implements (but not fully, therefore 'WIP',
>>> there's another commit following it pretty soon after) better error
>>> reporting feedback to the user through the interface. I.e. not having to
>>> see the stacktrace on the server to determine if and why something went
>>> wrong. I believe you have been among the people requesting this too :)
>>>
>>> Also, as you're referencing, we've made some other changes to GML
>>> import. In particular the importer will now match on uid, code or name, as
>>> oppsed to the former mode of matching only on name. This is not reflected
>>> in the UI as it is a backend enhancement only, and does not change the
>>> import options at all. It IS, however, documented and ready for you to read
>>> about. Short version: it matches on a prioritized list of uid -> code ->
>>> name. Long version:
>>> https://www.dhis2.org/doc/snapshot/en/user/html/ch18s02.html
>>>
>>> Be adviced that the commit you've referenced is towards development
>>> trunk, and is not backported to 2.19, the match on multiple identifiers,
>>> however, was introduced in 2.19 and should work in current 2.19 stable. If
>>> you're having problems with it you can send me a full stack trace and
>>> preferably the GML input data.
>>>
>>>
>>>
>>> 2015-06-25 17:54 GMT+02:00 Calle Hedberg <calle.hedberg@xxxxxxxxx>:
>>>
>>>> Hi,
>>>>
>>>> I'm unable to import GML files using 2.19 (the same files import fine
>>>> in 2.18) - the import simply hangs (endless rolling progress bar icon).
>>>>
>>>> Will the above bug-fix sort out that problem - I do not understand what
>>>> the above actually means, or if the GML format has changed in some way.
>>>>
>>>> Note also that the final 2.19 release was supposed to include a more
>>>> diversified GML import - but I cannot see WHERE to do that (I still see the
>>>> same two buttons: choose file and import, either dry run or not)
>>>>
>>>> Regards
>>>> Calle
>>>>
>>>> On 5 June 2015 at 14:57, <noreply@xxxxxxxxxxxxx> wrote:
>>>>
>>>>> ------------------------------------------------------------
>>>>> revno: 19296
>>>>> committer: Halvdan Hoem Grelland <halvdanhg@xxxxxxxxx>
>>>>> branch nick: dhis2
>>>>> timestamp: Fri 2015-06-05 14:56:34 +0200
>>>>> message:
>>>>> WIP proper error reporting on failed GML parsing during import
>>>>> added:
>>>>>
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>> modified:
>>>>>
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>>
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>>
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>>
>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>>
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>>
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>>
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>>
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>>
>>>>>
>>>>> --
>>>>> lp:dhis2
>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk
>>>>>
>>>>> Your team DHIS 2 developers is subscribed to branch lp:dhis2.
>>>>> To unsubscribe from this branch go to
>>>>> https://code.launchpad.net/~dhis2-devs-core/dhis2/trunk/+edit-subscription
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java'
>>>>> ---
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>> 2015-06-03 15:00:19 +0000
>>>>> +++
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/DefaultGmlImportService.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -32,6 +32,8 @@
>>>>> import com.google.common.base.Strings;
>>>>> import com.google.common.collect.Iterators;
>>>>> import com.google.common.collect.Maps;
>>>>> +import org.apache.commons.io.IOUtils;
>>>>> +import org.hibernate.Hibernate;
>>>>> import org.hisp.dhis.common.IdentifiableObjectManager;
>>>>> import org.hisp.dhis.common.IdentifiableProperty;
>>>>> import org.hisp.dhis.common.MergeStrategy;
>>>>> @@ -88,13 +90,91 @@
>>>>> // GmlImportService implementation
>>>>> //
>>>>> -------------------------------------------------------------------------
>>>>>
>>>>> + @Transactional
>>>>> + @Override
>>>>> + public GmlPreProcessingResult preProcessGml( InputStream
>>>>> inputStream )
>>>>> + {
>>>>> + InputStream dxfStream = null;
>>>>> + MetaData metaData = null;
>>>>> +
>>>>> + try
>>>>> + {
>>>>> + dxfStream = transformGml( inputStream );
>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>> MetaData.class );
>>>>> + }
>>>>> + catch ( IOException | TransformerException e )
>>>>> + {
>>>>> + return GmlPreProcessingResult.failure( e );
>>>>> + }
>>>>> + finally
>>>>> + {
>>>>> + IOUtils.closeQuietly( dxfStream );
>>>>> + }
>>>>> +
>>>>> + Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>> +
>>>>> + matchAndFilterOnIdentifiers( metaData.getOrganisationUnits(),
>>>>> uidMap, codeMap, nameMap );
>>>>> +
>>>>> + Map<String, OrganisationUnit> persistedUidMap =
>>>>> getMatchingPersistedOrgUnits( uidMap.keySet(), IdentifiableProperty.UID );
>>>>> + Map<String, OrganisationUnit> persistedCodeMap =
>>>>> getMatchingPersistedOrgUnits( codeMap.keySet(), IdentifiableProperty.CODE );
>>>>> + Map<String, OrganisationUnit> persistedNameMap =
>>>>> getMatchingPersistedOrgUnits( nameMap.keySet(), IdentifiableProperty.NAME );
>>>>> +
>>>>> + Iterator<OrganisationUnit> persistedIterator =
>>>>> Iterators.concat( persistedUidMap.values().iterator(),
>>>>> + persistedCodeMap.values().iterator(),
>>>>> persistedNameMap.values().iterator() );
>>>>> +
>>>>> + while ( persistedIterator.hasNext() )
>>>>> + {
>>>>> + OrganisationUnit persisted = persistedIterator.next(),
>>>>> imported = null;
>>>>> +
>>>>> + if ( !Strings.isNullOrEmpty( persisted.getUid() ) &&
>>>>> uidMap.containsKey( persisted.getUid() ) )
>>>>> + {
>>>>> + imported = uidMap.get( persisted.getUid() );
>>>>> + }
>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getCode() )
>>>>> && codeMap.containsKey( persisted.getCode() ) )
>>>>> + {
>>>>> + imported = codeMap.get( persisted.getCode() );
>>>>> + }
>>>>> + else if ( !Strings.isNullOrEmpty( persisted.getName() )
>>>>> && nameMap.containsKey( persisted.getName() ) )
>>>>> + {
>>>>> + imported = nameMap.get( persisted.getName() );
>>>>> + }
>>>>> +
>>>>> + if ( imported == null || imported.getCoordinates() ==
>>>>> null || imported.getFeatureType() == null )
>>>>> + {
>>>>> + continue; // Failed to dereference a persisted entity
>>>>> for this org unit or geo data incomplete/missing, therefore ignore
>>>>> + }
>>>>> +
>>>>> + mergeNonGeoData( persisted, imported );
>>>>> + }
>>>>> +
>>>>> + return GmlPreProcessingResult.success( metaData );
>>>>> + }
>>>>> +
>>>>> @Override
>>>>> public MetaData fromGml( InputStream inputStream )
>>>>> throws IOException, TransformerException
>>>>> {
>>>>> - InputStream dxfStream = transformGml( inputStream );
>>>>> - MetaData metaData = renderService.fromXml( dxfStream,
>>>>> MetaData.class );
>>>>> - dxfStream.close();
>>>>> + InputStream dxfStream;
>>>>> + MetaData metaData;
>>>>> +
>>>>> + try
>>>>> + {
>>>>> + dxfStream = transformGml( inputStream );
>>>>> + }
>>>>> + catch (Exception e)
>>>>> + {
>>>>> + dxfStream = null;
>>>>> + }
>>>>> +
>>>>> + if(dxfStream != null)
>>>>> + {
>>>>> + metaData = renderService.fromXml( dxfStream,
>>>>> MetaData.class );
>>>>> + dxfStream.close();
>>>>> + }
>>>>> + else
>>>>> + {
>>>>> + return null;
>>>>> + }
>>>>>
>>>>> Map<String, OrganisationUnit> uidMap = Maps.newHashMap(),
>>>>> codeMap = Maps.newHashMap(), nameMap = Maps.newHashMap();
>>>>>
>>>>> @@ -143,6 +223,7 @@
>>>>> importService.importMetaData( userUid, fromGml( inputStream
>>>>> ), importOptions, taskId );
>>>>> }
>>>>>
>>>>> + @Transactional
>>>>> @Override
>>>>> public void importGml( MetaData metaData, String userUid,
>>>>> ImportOptions importOptions, TaskId taskId )
>>>>> {
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java'
>>>>> ---
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>> 2015-06-03 15:00:19 +0000
>>>>> +++
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlImportService.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -62,6 +62,24 @@
>>>>> throws IOException, TransformerException;
>>>>>
>>>>> /**
>>>>> + * Pre-process a GML document. The process, in short, entails the
>>>>> following:
>>>>> + * <ol>
>>>>> + * <li>Parse the GML payload and transform it into DXF2
>>>>> format</li>
>>>>> + * <li>Get the given identifiers (uid, code or name) from the
>>>>> parsed payload and fetch
>>>>> + * the corresponding entities from the DB</li>
>>>>> + * <li>Merge the geospatial data given in the input GML into
>>>>> DB entities and return</li>
>>>>> + * </ol>
>>>>> + *
>>>>> + * The result of this process in returned in a {@link
>>>>> GmlPreProcessingResult} which
>>>>> + * encapsulates the returned {@link MetaData} object or the
>>>>> exception in cause of parse
>>>>> + * failure due to IO errors or malformed input.
>>>>> + *
>>>>> + * @param gmlInputStream the InputStream providing the GML input.
>>>>> + * @return a GmlPreProcessingResult representing the end result
>>>>> of the process.
>>>>> + */
>>>>> + GmlPreProcessingResult preProcessGml( InputStream gmlInputStream
>>>>> );
>>>>> +
>>>>> + /**
>>>>> * Imports GML data and merges the geospatial data updates into
>>>>> the database.
>>>>> * See {@link #fromGml(InputStream)} for details on the
>>>>> underlying process.
>>>>> *
>>>>>
>>>>> === added file
>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java'
>>>>> ---
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>> 1970-01-01 00:00:00 +0000
>>>>> +++
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/main/java/org/hisp/dhis/dxf2/gml/GmlPreProcessingResult.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -0,0 +1,70 @@
>>>>> +package org.hisp.dhis.dxf2.gml;
>>>>> +
>>>>> +import org.hisp.dhis.dxf2.metadata.MetaData;
>>>>> +
>>>>> +import java.io.InputStream;
>>>>> +
>>>>> +/**
>>>>> + * Wraps the result of {@link
>>>>> GmlImportService#preProcessGml(InputStream)}.
>>>>> + * This is necessary when performing GML import on a context where
>>>>> exceptions
>>>>> + * due to malformed input cannot be caught, thus leaving the user
>>>>> uninformed of
>>>>> + * the error. This class will wrap the failure and provide the
>>>>> Throwable to the
>>>>> + * consuming class on error or the resulting MetaData object on
>>>>> success.
>>>>> + *
>>>>> + * @author Halvdan Hoem grelland <halvdanhg@xxxxxxxxx>
>>>>> + */
>>>>> +public final class GmlPreProcessingResult
>>>>> +{
>>>>> + private boolean isSuccess;
>>>>> + private MetaData resultMetaData;
>>>>> + private Throwable throwable;
>>>>> +
>>>>> + private GmlPreProcessingResult(){}
>>>>> +
>>>>> + public static GmlPreProcessingResult success( MetaData
>>>>> resultMetaData ) {
>>>>> + GmlPreProcessingResult result = new GmlPreProcessingResult();
>>>>> + result.setResultMetaData( resultMetaData );
>>>>> + result.setSuccess( true );
>>>>> +
>>>>> + return result;
>>>>> + }
>>>>> +
>>>>> + public static GmlPreProcessingResult failure( Throwable throwable
>>>>> )
>>>>> + {
>>>>> + GmlPreProcessingResult result = new GmlPreProcessingResult();
>>>>> + result.setThrowable( throwable );
>>>>> + result.setSuccess( false );
>>>>> +
>>>>> + return result;
>>>>> + }
>>>>> +
>>>>> + private void setSuccess( boolean isSuccess )
>>>>> + {
>>>>> + this.isSuccess = isSuccess;
>>>>> + }
>>>>> +
>>>>> + public boolean isSuccess()
>>>>> + {
>>>>> + return isSuccess;
>>>>> + }
>>>>> +
>>>>> + private void setResultMetaData( MetaData resultMetaData )
>>>>> + {
>>>>> + this.resultMetaData = resultMetaData;
>>>>> + }
>>>>> +
>>>>> + public MetaData getResultMetaData()
>>>>> + {
>>>>> + return resultMetaData;
>>>>> + }
>>>>> +
>>>>> + private void setThrowable( Throwable throwable )
>>>>> + {
>>>>> + this.throwable = throwable;
>>>>> + }
>>>>> +
>>>>> + public Throwable getThrowable()
>>>>> + {
>>>>> + return throwable;
>>>>> + }
>>>>> +}
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java'
>>>>> ---
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>> 2015-01-17 07:41:26 +0000
>>>>> +++
>>>>> dhis-2/dhis-services/dhis-service-dxf2/src/test/java/org/hisp/dhis/dxf2/gml/GmlImportServiceTest.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -117,4 +117,6 @@
>>>>> assertEquals( 1, units.get(
>>>>> "Blindern").getCoordinatesAsList().get( 0 ).getNumberOfCoordinates() );
>>>>> assertEquals( 76, units.get( "Forskningsparken"
>>>>> ).getCoordinatesAsList().get(0).getNumberOfCoordinates() );
>>>>> }
>>>>> +
>>>>> + // TODO Add test for GmlImportService#preProcessGml(InputStream)
>>>>> }
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java'
>>>>> ---
>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>> 2015-05-30 13:36:07 +0000
>>>>> +++
>>>>> dhis-2/dhis-support/dhis-support-system/src/main/java/org/hisp/dhis/system/scheduling/SpringScheduler.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -77,7 +77,7 @@
>>>>> {
>>>>> taskExecutor.execute( task );
>>>>> }
>>>>> -
>>>>> +
>>>>> @Override
>>>>> public boolean scheduleTask( String key, Runnable task, String
>>>>> cronExpr )
>>>>> {
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java'
>>>>> ---
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>> 2015-05-28 16:10:07 +0000
>>>>> +++
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/dxf2/MetaDataImportAction.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -187,7 +187,7 @@
>>>>> }
>>>>> else if ( "gml".equals( importFormat ) )
>>>>> {
>>>>> - scheduler.executeTask( new ImportMetaDataGmlTask( userId,
>>>>> gmlImportService, importOptions, in, taskId ) );
>>>>> + scheduler.executeTask( new ImportMetaDataGmlTask( userId,
>>>>> gmlImportService, notifier, importOptions, in, taskId ) );
>>>>> }
>>>>> else if ( "json".equals( importFormat ) || "xml".equals(
>>>>> importFormat ) )
>>>>> {
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java'
>>>>> ---
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>> 2015-04-11 14:06:51 +0000
>>>>> +++
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/java/org/hisp/dhis/importexport/action/util/ImportMetaDataGmlTask.java
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -28,14 +28,18 @@
>>>>> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
>>>>> */
>>>>>
>>>>> +import org.apache.commons.lang.exception.ExceptionUtils;
>>>>> import org.apache.commons.logging.Log;
>>>>> import org.apache.commons.logging.LogFactory;
>>>>> import org.hisp.dhis.dxf2.common.ImportOptions;
>>>>> import org.hisp.dhis.dxf2.gml.GmlImportService;
>>>>> +import org.hisp.dhis.dxf2.gml.GmlPreProcessingResult;
>>>>> import org.hisp.dhis.scheduling.TaskId;
>>>>> +import org.hisp.dhis.system.notification.NotificationLevel;
>>>>> +import org.hisp.dhis.system.notification.Notifier;
>>>>> +import org.springframework.web.util.HtmlUtils;
>>>>> +import org.xml.sax.SAXParseException;
>>>>>
>>>>> -import javax.xml.transform.TransformerException;
>>>>> -import java.io.IOException;
>>>>> import java.io.InputStream;
>>>>>
>>>>> /**
>>>>> @@ -50,12 +54,15 @@
>>>>>
>>>>> private String userUid;
>>>>>
>>>>> +
>>>>> //
>>>>> -------------------------------------------------------------------------
>>>>> // Dependencies
>>>>> //
>>>>> -------------------------------------------------------------------------
>>>>>
>>>>> private GmlImportService gmlImportService;
>>>>>
>>>>> + private Notifier notifier;
>>>>> +
>>>>> private ImportOptions importOptions;
>>>>>
>>>>> private InputStream inputStream;
>>>>> @@ -64,11 +71,12 @@
>>>>> // Constructors
>>>>> //
>>>>> -------------------------------------------------------------------------
>>>>>
>>>>> - public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>> gmlImportService,
>>>>> + public ImportMetaDataGmlTask( String userUid, GmlImportService
>>>>> gmlImportService, Notifier notifier,
>>>>> ImportOptions importOptions, InputStream inputStream, TaskId
>>>>> taskId )
>>>>> {
>>>>> this.userUid = userUid;
>>>>> this.gmlImportService = gmlImportService;
>>>>> + this.notifier = notifier;
>>>>> this.importOptions = importOptions;
>>>>> this.inputStream = inputStream;
>>>>> this.taskId = taskId;
>>>>> @@ -83,15 +91,37 @@
>>>>> {
>>>>> importOptions.setImportStrategy( "update" ); // Force update
>>>>> only for GML import
>>>>>
>>>>> - try
>>>>> - {
>>>>> - gmlImportService.importGml( inputStream, userUid,
>>>>> importOptions, taskId );
>>>>> - }
>>>>> - catch ( IOException | TransformerException e )
>>>>> - {
>>>>> - log.error( "Unable to read GML data from input stream", e
>>>>> );
>>>>> -
>>>>> - throw new RuntimeException( "Failed to parse GML input
>>>>> stream", e );
>>>>> - }
>>>>> + GmlPreProcessingResult gmlPreProcessingResult =
>>>>> gmlImportService.preProcessGml( inputStream );
>>>>> +
>>>>> + if ( !gmlPreProcessingResult.isSuccess() )
>>>>> + {
>>>>> + Throwable throwable =
>>>>> gmlPreProcessingResult.getThrowable();
>>>>> + String message = createErrorMessage( throwable );
>>>>> +
>>>>> + notifier.notify( taskId, NotificationLevel.ERROR,
>>>>> message, false );
>>>>> + log.error( "GML import failed: " + message, throwable );
>>>>> +
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + gmlImportService.importGml(
>>>>> gmlPreProcessingResult.getResultMetaData(), userUid, importOptions, taskId
>>>>> );
>>>>> + }
>>>>> +
>>>>> + private String createErrorMessage( Throwable throwable )
>>>>> + {
>>>>> + String message = "";
>>>>> + Throwable rootThrowable = ExceptionUtils.getRootCause(
>>>>> throwable );
>>>>> +
>>>>> + if ( rootThrowable instanceof SAXParseException )
>>>>> + {
>>>>> + SAXParseException e = (SAXParseException) rootThrowable;
>>>>> + message += "Syntax error on line " + e.getLineNumber() +
>>>>> ". " + e.getMessage();
>>>>> + }
>>>>> + else
>>>>> + {
>>>>> + message += rootThrowable.getMessage();
>>>>> + }
>>>>> +
>>>>> + return HtmlUtils.htmlEscape( message );
>>>>> }
>>>>> }
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties'
>>>>> ---
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>> 2015-02-20 11:17:31 +0000
>>>>> +++
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/resources/org/hisp/dhis/importexport/i18n_module.properties
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -300,6 +300,7 @@
>>>>> ignored=Ignored
>>>>> conflicts=Conflicts
>>>>> no_conflicts_found=No conflicts found
>>>>> +no_import_summary_available=No summary available
>>>>> type=Type
>>>>> count=Count
>>>>> export_as_xml=Export as XML
>>>>>
>>>>> === modified file
>>>>> 'dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm'
>>>>> ---
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>> 2013-03-21 09:15:24 +0000
>>>>> +++
>>>>> dhis-2/dhis-web/dhis-web-importexport/src/main/webapp/dhis-web-importexport/importMetaDataSummary.vm
>>>>> 2015-06-05 12:56:34 +0000
>>>>> @@ -1,4 +1,6 @@
>>>>> <h3>$i18n.getString( "import_summary" )</h3>
>>>>> +
>>>>> +#if( $summary ) ## ImportSummary can be null on failed GML import
>>>>> (pre-processing fails)
>>>>> <h4>$i18n.getString( "import_count" )</h4>
>>>>>
>>>>> $summary.importCount.imported Imported<br/>
>>>>> @@ -54,3 +56,6 @@
>>>>> #else
>>>>> <p>$i18n.getString( "no_conflicts_found" )</p>
>>>>> #end
>>>>> +#else
>>>>> + $i18n.getString( "no_summary_available" )
>>>>> +#end
>>>>> \ No newline at end of file
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *******************************************
>>>>
>>>> Calle Hedberg
>>>>
>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>
>>>> Tel/fax (home): +27-21-685-6472
>>>>
>>>> Cell: +27-82-853-5352
>>>>
>>>> Iridium SatPhone: +8816-315-19274
>>>>
>>>> Email: calle.hedberg@xxxxxxxxx
>>>>
>>>> Skype: calle_hedberg
>>>>
>>>> *******************************************
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help : https://help.launchpad.net/ListHelp
>
>
Follow ups
References
