dhis2-devs team mailing list archive

[Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>]

Hello Morten,

Thanks for the advice. For what I know the server looks big enough (4gb
mem). I might try to scale it to 8gb change some vm parameters and see how
it works. But knowing that I have the backup solution to split it in level
is always good.

Regards,
Nicolas


On Fri, Oct 23, 2015 at 3:01 AM, Morten Olav Hansen <mortenoh@xxxxxxxxx>
wrote:

> It's also fine splitting it up in many small pieces. As you say, you must
> ensure that the parents exists, so doing it one level at the time is the
> approach. If not, you must do import multiple times, so it can update the
> parents (not really a proper solution).
>
> Does the server have sufficient memory? because 37k OUs (8mb) should not
> really be a problem.. I have imported 50-60mb on my local machine before
> without any issues (although that was more than just OUs)
>
> --
> Morten
>
> On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx
> > wrote:
>
>> Jason,
>>
>> I guess I'm going on the XSLT solution and split it per level, this is
>> the fastest line for me.
>>
>> Thanks a lot for the advices. Enjoy you mushrooms wanderings.
>>
>> See you
>> Nicolas
>>
>>
>>
>> On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering <
>> jason.p.pickering@xxxxxxxxx> wrote:
>>
>>> Hi Nicolas
>>>
>>> Not sure there is a tool per se, but there are approaches.  Not sure
>>> where your data is coming from, but if you are transforming to JSON/XML, it
>>> may be just as easy to transform to SQL insert statements.
>>>
>>> The trick is to do it level by level and be sure you get the parent ID
>>> correct. Otherwise, set the parent level and be sure you reset the
>>> hibernate sequence of the database to avoid collisions with future inserts.
>>>
>>> Otherwise, maybe try what you have but in batches of a few thousand at a
>>> time since you seem to have it already in an importable format.
>>>
>>> Thanks. I need to do something other than  dhis2 and looking for
>>> mushrooms helps to clear the mind.
>>>
>>> Regards,
>>> Jason
>>>
>>> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Hello Jason,
>>>>
>>>> Nope, no stacktrace. I suppose the transaction is too big and blow the
>>>> whole thing up. Yep sure, I don't really care about the method as long as
>>>> it doesn't take to much time to setup. I'm thinking about creating a quick
>>>> XSLT to extract org unit per level but if you have any tool/idea which
>>>> allow direct SQL I'm in.
>>>>
>>>> Nice mushroom picture by the way.
>>>> Regards.
>>>> Nicolas
>>>>
>>>>
>>>>
>>>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Nicolas,
>>>>>
>>>>> Hard to say without a stack trace.  It just bombs with no explanation?
>>>>>
>>>>> Could you consider using direct SQL injection for such a large number
>>>>> of orgunits?
>>>>>
>>>>> Regards,
>>>>> Jason
>>>>>
>>>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to make a rather large import of orgunit. The size of the
>>>>>> file is around 8megs zipped and it contains more than 37000 orgunits.
>>>>>>
>>>>>> When I'm launching the import it takes a little bit more than an
>>>>>> hour. I can see in the log file (dhis.log) outputs telling me that org unit
>>>>>> are being imported.  At the end I'm receiving the following message in
>>>>>> catalina.out
>>>>>>
>>>>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category:
>>>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done.
>>>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>>>>
>>>>>> and then the server crash without any message on what when wrong.
>>>>>> Nothing is imported.
>>>>>>
>>>>>> I'm wondering how the import works, can I split the file arbitrary in
>>>>>> multiple smaller parts (will it handle the fk parent id correctly) ? Or do
>>>>>> I have to split the file per level to ensure parent finding ?
>>>>>>
>>>>>> Thanks
>>>>>> Regards
>>>>>> Nicolas
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Nicolas Germeau*
>>>>>> *Software Developer*
>>>>>>
>>>>>> GSM:  +32 487 129820
>>>>>>
>>>>>> Tel: +32 2 737 65 16
>>>>>> Skype: nicolas.germeau
>>>>>>
>>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>>>
>>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Nicolas Germeau*
>>>> *Software Developer*
>>>>
>>>> GSM:  +32 487 129820
>>>>
>>>> Tel: +32 2 737 65 16
>>>> Skype: nicolas.germeau
>>>>
>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>> Visit our new website <http://www.bluesquare.org/>
>>>>
>>>
>>
>>
>> --
>> *Nicolas Germeau*
>> *Software Developer*
>>
>> GSM:  +32 487 129820
>>
>> Tel: +32 2 737 65 16
>> Skype: nicolas.germeau
>>
>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>> Visit our new website <http://www.bluesquare.org/>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>


-- 
*Nicolas Germeau*
*Software Developer*

GSM:  +32 487 129820

Tel: +32 2 737 65 16
Skype: nicolas.germeau

Follow us on Twitter <https://twitter.com/bluesquare_org>
Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
Visit our new website <http://www.bluesquare.org/>