dhis2-devs team mailing list archive

[Morten Olav Hansen <mortenoh@xxxxxxxxx>]

It's also fine splitting it up in many small pieces. As you say, you must
ensure that the parents exists, so doing it one level at the time is the
approach. If not, you must do import multiple times, so it can update the
parents (not really a proper solution).

Does the server have sufficient memory? because 37k OUs (8mb) should not
really be a problem.. I have imported 50-60mb on my local machine before
without any issues (although that was more than just OUs)

--
Morten

On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
wrote:

> Jason,
>
> I guess I'm going on the XSLT solution and split it per level, this is the
> fastest line for me.
>
> Thanks a lot for the advices. Enjoy you mushrooms wanderings.
>
> See you
> Nicolas
>
>
>
> On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering <
> jason.p.pickering@xxxxxxxxx> wrote:
>
>> Hi Nicolas
>>
>> Not sure there is a tool per se, but there are approaches.  Not sure
>> where your data is coming from, but if you are transforming to JSON/XML, it
>> may be just as easy to transform to SQL insert statements.
>>
>> The trick is to do it level by level and be sure you get the parent ID
>> correct. Otherwise, set the parent level and be sure you reset the
>> hibernate sequence of the database to avoid collisions with future inserts.
>>
>> Otherwise, maybe try what you have but in batches of a few thousand at a
>> time since you seem to have it already in an importable format.
>>
>> Thanks. I need to do something other than  dhis2 and looking for
>> mushrooms helps to clear the mind.
>>
>> Regards,
>> Jason
>>
>> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>> wrote:
>>
>>> Hello Jason,
>>>
>>> Nope, no stacktrace. I suppose the transaction is too big and blow the
>>> whole thing up. Yep sure, I don't really care about the method as long as
>>> it doesn't take to much time to setup. I'm thinking about creating a quick
>>> XSLT to extract org unit per level but if you have any tool/idea which
>>> allow direct SQL I'm in.
>>>
>>> Nice mushroom picture by the way.
>>> Regards.
>>> Nicolas
>>>
>>>
>>>
>>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>
>>>> Hi Nicolas,
>>>>
>>>> Hard to say without a stack trace.  It just bombs with no explanation?
>>>>
>>>> Could you consider using direct SQL injection for such a large number
>>>> of orgunits?
>>>>
>>>> Regards,
>>>> Jason
>>>>
>>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to make a rather large import of orgunit. The size of the
>>>>> file is around 8megs zipped and it contains more than 37000 orgunits.
>>>>>
>>>>> When I'm launching the import it takes a little bit more than an hour.
>>>>> I can see in the log file (dhis.log) outputs telling me that org unit are
>>>>> being imported.  At the end I'm receiving the following message in
>>>>> catalina.out
>>>>>
>>>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category:
>>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done.
>>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>>>
>>>>> and then the server crash without any message on what when wrong.
>>>>> Nothing is imported.
>>>>>
>>>>> I'm wondering how the import works, can I split the file arbitrary in
>>>>> multiple smaller parts (will it handle the fk parent id correctly) ? Or do
>>>>> I have to split the file per level to ensure parent finding ?
>>>>>
>>>>> Thanks
>>>>> Regards
>>>>> Nicolas
>>>>>
>>>>>
>>>>> --
>>>>> *Nicolas Germeau*
>>>>> *Software Developer*
>>>>>
>>>>> GSM:  +32 487 129820
>>>>>
>>>>> Tel: +32 2 737 65 16
>>>>> Skype: nicolas.germeau
>>>>>
>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Nicolas Germeau*
>>> *Software Developer*
>>>
>>> GSM:  +32 487 129820
>>>
>>> Tel: +32 2 737 65 16
>>> Skype: nicolas.germeau
>>>
>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>> Visit our new website <http://www.bluesquare.org/>
>>>
>>
>
>
> --
> *Nicolas Germeau*
> *Software Developer*
>
> GSM:  +32 487 129820
>
> Tel: +32 2 737 65 16
> Skype: nicolas.germeau
>
> Follow us on Twitter <https://twitter.com/bluesquare_org>
> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
> Visit our new website <http://www.bluesquare.org/>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>