dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

2012/2/12 Lars Helge Øverland <larshelge@xxxxxxxxx>:
> Hi,
>
> I think the important thing here is to improve the user feedback
> mechanism, not the extra thread itself. First, in my opinion it is
> easier to provide user feedback this way - the current arrangement is
> to start a separate thread and then request/ping that thread
> continuously and return/display a status response through an ajax user
> interface. I don't see how running a synchronous process and then
> block the user interface will make it simpler.

It makes it considerably simpler in that you don't have to manage
communication between two threads.

>Second, long-running
> threads are at the mercy of the container timeout - as far as I can
> remember Jetty uses a default 30 second request timeout meaning large
> import processes are in danger of timing out.

Depends really on the use case for import-export.  There should really
be very few cases where routine import/export should ever take close
to 30 seconds.  I recall the 300 000 data values we imported from the
dummy ihris data in Kenya took around 8 seconds.  Even so, I think the
30 second timeout is related to inactivity on the thread
(maxIdleTime).  So long as user feedback messages are trickling back
at faster than every 30 seconds (which they should) then it shouldn't
ever timeout.

Either way, I can see there are some advantages and disadvantages with
both approaches.  But running a single synchronous import thread would
just be much simpler.  The alternative requires the creation of an
inter thread communication mechanism which is more sophisticated than
the cave process status message.  Of course that's doable (Jo and I
had a mechanism working about 2 years ago) but it really does seem
unnecessarily complicated for what we need to do.  And with relatively
large imports, by the time the last byte has arrived over the real
internet (rather than localhost on devs laptop) the data import
process would be close to complete.

>
> But yes, a total refactor of the import process is already on the plan
> and we also need to improve user feedback. It might be we should do it
> more like 1.4 - always do a preview of the import, then present a
> report with anomalies before importing into the db. Let me think a bit
> more about it.

Agree with regards metadata import.  We could simply enforce preview
on this always.

I think with data import it might be sufficient to import and return a
report of any failed values.  Partly because we want to support
pushing of datavaluesets as well as pulling through web UI.  If we did
use a separate thread for this then I guess that thread would persist
an identifying import uid of some sort which could be used later to
retrieve a report.  Though I still can't help thinking its much easier
to return this synchronously ...

I'll think some more about it when I've finished with some mydatamart
fixing.  I have some code which imports dxf1.0 datavalues without
importing metadata.  I want to implement that simply (with user
feedback) without rewriting the whole cave stuff ..

Bob

>
> Lars