dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

Knut,

Unless it was enhanced during the last 2-3 days, then it is still very slow
compared to other data imports (e.g importing 5.6 mill datavalue records
take around 10 minutes on my laptop - importing ~29,000 events with 250k
event values took about twice that)

regards
calle

On 12 March 2015 at 04:11, Knut Staring <knutst@xxxxxxxxx> wrote:

> Also I think the performance was enhanced recently.
> On 11 Mar 2015 23:09, "Pierre Dane" <pierre@xxxxxxxxx> wrote:
>
>> hi Calle
>>
>> FYI I have some database scripts that I used to import events into our
>> program - 300k records in 10 mins or so. if you need to bulk insert events
>> let me know and I can share. p
>>
>>
>>
>> On 11 Mar 2015, at 23:49, Calle Hedberg <calle.hedberg@xxxxxxxxx> wrote:
>>
>> Hi
>>
>> There seems to be at least two bugs related to the import of event-data
>> in CSV format introduced in 2.17.
>>
>> Firstly, a dry run importing around 29,000 events (~250,000 trackervalue
>> records) into 2.18 (version as of 09 Mar 2015) did not seem to work - no
>> feedback within reasonable time. NOTE that the import with dry run set to
>> NO worked fine, but it was quite slow - so there is a possibility the dry
>> run actually works BUT since it gives no indication of activity for a LOONG
>> time it is perceived as not working. If that is the case, it is important
>> to add a counter or similar to the user messages (e.g. "1000 events
>> imported" .... "2000 events imported" etc).
>>
>> Secondly, DHIS2 has a confusing set of standards for uppercase and
>> lowercase keywords. When you export some event data to CSV, it will export
>> the content of "ProvidedElsewhere" as it is stored: e.g. "FALSE" as an
>> uppercase word. If you try to import the same file, it is rejected and the
>> tomcat log indicates that only "false" and "true" are accepted (lowercase).
>>
>> Thirdly, from a more "cosmetic" perspective, I find it annoying that a
>> tomcat log entry is produced for EVERY event imported - it would be better
>> if only "abnormal" issues are logged otherwise any abnormalities or
>> warnings drowns in the "info about every event record" clutter.
>>
>> Regards
>> Calle
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472
>>
>> Cell: +27-82-853-5352
>>
>> Iridium SatPhone: +8816-315-19274
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>>  _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>


-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19274

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************