dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

2012/2/14 Jason Pickering <jason.p.pickering@xxxxxxxxx>:
>>
>>
>> Depends really on the use case for import-export.  There should really
>> be very few cases where routine import/export should ever take close
>> to 30 seconds.  I recall the 300 000 data values we imported from the
>> dummy ihris data in Kenya took around 8 seconds.  Even so, I think the
>> 30 second timeout is related to inactivity on the thread
>> (maxIdleTime).  So long as user feedback messages are trickling back
>> at faster than every 30 seconds (which they should) then it shouldn't
>> ever timeout.
>>
> I am importing data now, around 800,00 data values, and it takes
> considerably longer than 30 seconds (more than 30 minutes actually) and this
> is on a very fast machine, i7,8 GB RAM, SSD.  Granted, this is a 1.4 import
> process, but I have seen similar things for DXF imports of these sizes, so I
> wish I knew how you got 300k values imported in 8 seconds.
>
>
> * INFO  17:02:53,105 Imported CompleteDataSetRegistrations
> (DefaultImportObjectManager.java [Thread-11])
> * INFO  17:38:04,768 867341 values were imported.
> (DefaultImportObjectManager.java [Thread-11])
> * INFO  17:38:05,398 Internal process completed successfully
> (AbstractStatementInternalProcess.java [Thread-11])

The problem here is that nothing happens on the thread between 17:02
and 17:38.  What I have done is to trickle some feedback to say 1000,
2000, 3000 , 4000 datavalues imported etc have been imported.

>
> Not sure this really matters, but obviously, I do not really care about if
> it is synchronous or not. I am going to go and have a cup of coffee and wait
> until it is finished, but would obviously like to know whether it is going
> to succeed or not before starting such a long process.

Ideally it should always succeed.  Though some values might be
rejected for reasons which are made explicit .. non-existent orgunits
etc.

>
> Regards,
> Jason
>