dhis2-devs team mailing list archive

[Calle Hedberg <calle.hedberg@xxxxxxxxx>]

David,

I'm importing into 2.26, postgresql is tuned as far as possible (the test
was on my laptop), and 12GB RAM. I was using CSV, put there should not be
any significant difference between JSON and CSV.

I will import the 2.5 mill events directly and see what time that takes,
comparatively.

Regards
Calle

On 30 June 2017 at 18:05, David Huser <dhuser@xxxxxxxxxxxxxx> wrote:

> Hi Calle
>
> When some conditions are met, the event import can be really speedy
> (imports of ~100 000 in <30 minutes).
>
> - Recent version (2.25+)
> - PostgreSQL properly tuned (this is important)
> - Enough RAM (8 GB+)
>
> I usually use JSON files and post them against /api/events.
>
> David
>
> On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
> wrote:
>
>> Ime,
>>
>> I was using the UI - have not looked at the API endpoint.
>>
>> For now I will use sql scripts as I've done before - I'm just trying to
>> find out if anybody is working on improving what is (now) an import
>> function that in reality only caters for small data sets. There is no
>> question about the core developers ability to speed this up - I saw that
>> with the import of OU meta-data - but not sure if anybody has done anything.
>>
>> If no answers from core developers, the I will write a JIRA issue for it.
>>
>> Regards
>> Calle
>>
>> On 29 June 2017 at 02:25, Ime Asangansi <imeasangansi@xxxxxxxxx> wrote:
>>
>>> Hi Calle,
>>>
>>> Thanks for your question.
>>> Just curious (I don't have answer). As we are planning to do this soon.
>>>
>>> Are you doing this through UI or API?
>>> If API, which endpoint enables this?
>>>
>>> Thanks
>>>
>>> Ime
>>>
>>>
>>>
>>> On Jun 28, 2017 19:15, "Calle Hedberg" <calle.hedberg@xxxxxxxxx> wrote:
>>>
>>>> Hi
>>>>
>>>> I started importing around 700,000 events 2 days ago (about 5 mill
>>>> individual values) - and the import is still running, 48 hours later.
>>>> Import is slowing - it seemed to be importing around 7-8 values per second,
>>>> now down to 2-3 per second. It looks like that import might take around 200
>>>> hours - which makes no sense from a usability point of view. Importing
>>>> OrgUnits used to be equally slow, but a revamp recently change a 10,000
>>>> OrgUnit import&update from hours to something like 5 seconds.
>>>>
>>>> It seems obvious that event imports have been designed to import values
>>>> one by one, which is equivalent to asking people to fetch water using a
>>>> leaking tea-spoon.
>>>>
>>>> I've searched JIRA but cannot find any issue related to this, thus this
>>>> post: Are there any plans underway to make import of Event data usable for
>>>> larger record sets?
>>>>
>>>> Regards
>>>> Calle
>>>>
>>>> *******************************************
>>>>
>>>> Calle Hedberg
>>>>
>>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>>
>>>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>>>
>>>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>>>
>>>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>>>
>>>> Email: calle.hedberg@xxxxxxxxx
>>>>
>>>> Skype: calle_hedberg
>>>>
>>>> *******************************************
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>
>>
>> --
>>
>> *******************************************
>>
>> Calle Hedberg
>>
>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>
>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>
>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>
>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>
>> Email: calle.hedberg@xxxxxxxxx
>>
>> Skype: calle_hedberg
>>
>> *******************************************
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
>
>
> *David Huser*
> DHIS2 Support Specialist
> dhuser@xxxxxxxxxxxxxx | https://baosystems.com | Skype: dafhus | 2900 K
> Street, Suite 406, Washington D.C. 20007
>



-- 

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@xxxxxxxxx

Skype: calle_hedberg

*******************************************