dhis2-devs team mailing list archive

[David Huser <dhuser@xxxxxxxxxxxxxx>]

Hi Calle

When some conditions are met, the event import can be really speedy
(imports of ~100 000 in <30 minutes).

- Recent version (2.25+)
- PostgreSQL properly tuned (this is important)
- Enough RAM (8 GB+)

I usually use JSON files and post them against /api/events.

David

On Fri, Jun 30, 2017 at 12:58 PM, Calle Hedberg <calle.hedberg@xxxxxxxxx>
wrote:

> Ime,
>
> I was using the UI - have not looked at the API endpoint.
>
> For now I will use sql scripts as I've done before - I'm just trying to
> find out if anybody is working on improving what is (now) an import
> function that in reality only caters for small data sets. There is no
> question about the core developers ability to speed this up - I saw that
> with the import of OU meta-data - but not sure if anybody has done anything.
>
> If no answers from core developers, the I will write a JIRA issue for it.
>
> Regards
> Calle
>
> On 29 June 2017 at 02:25, Ime Asangansi <imeasangansi@xxxxxxxxx> wrote:
>
>> Hi Calle,
>>
>> Thanks for your question.
>> Just curious (I don't have answer). As we are planning to do this soon.
>>
>> Are you doing this through UI or API?
>> If API, which endpoint enables this?
>>
>> Thanks
>>
>> Ime
>>
>>
>>
>> On Jun 28, 2017 19:15, "Calle Hedberg" <calle.hedberg@xxxxxxxxx> wrote:
>>
>>> Hi
>>>
>>> I started importing around 700,000 events 2 days ago (about 5 mill
>>> individual values) - and the import is still running, 48 hours later.
>>> Import is slowing - it seemed to be importing around 7-8 values per second,
>>> now down to 2-3 per second. It looks like that import might take around 200
>>> hours - which makes no sense from a usability point of view. Importing
>>> OrgUnits used to be equally slow, but a revamp recently change a 10,000
>>> OrgUnit import&update from hours to something like 5 seconds.
>>>
>>> It seems obvious that event imports have been designed to import values
>>> one by one, which is equivalent to asking people to fetch water using a
>>> leaking tea-spoon.
>>>
>>> I've searched JIRA but cannot find any issue related to this, thus this
>>> post: Are there any plans underway to make import of Event data usable for
>>> larger record sets?
>>>
>>> Regards
>>> Calle
>>>
>>> *******************************************
>>>
>>> Calle Hedberg
>>>
>>> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>>>
>>> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>>>
>>> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>>>
>>> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>>>
>>> Email: calle.hedberg@xxxxxxxxx
>>>
>>> Skype: calle_hedberg
>>>
>>> *******************************************
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>
>
> --
>
> *******************************************
>
> Calle Hedberg
>
> 46D Alma Road, 7700 Rosebank, SOUTH AFRICA
>
> Tel/fax (home): +27-21-685-6472 <+27%2021%20685%206472>
>
> Cell: +27-82-853-5352 <+27%2082%20853%205352>
>
> Iridium SatPhone: +8816-315-19119 <+881%206%20315%2019119>
>
> Email: calle.hedberg@xxxxxxxxx
>
> Skype: calle_hedberg
>
> *******************************************
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 


*David Huser*
DHIS2 Support Specialist
dhuser@xxxxxxxxxxxxxx | https://baosystems.com | Skype: dafhus | 2900 K
Street, Suite 406, Washington D.C. 20007