dhis2-devs team mailing list archive

[Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>]

Thanks a lot.

Nicolas

On Fri, Oct 23, 2015 at 9:55 AM, Morten Olav Hansen <mortenoh@xxxxxxxxx>
wrote:

> Ok, if you are still having issues (and can share the org units) feel free
> to email me the file and I will have a look.
>
> --
> Morten
>
> On Fri, Oct 23, 2015 at 3:53 PM, Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
> wrote:
>
>> Hello Morten,
>>
>> Thanks for the advice. For what I know the server looks big enough (4gb
>> mem). I might try to scale it to 8gb change some vm parameters and see how
>> it works. But knowing that I have the backup solution to split it in level
>> is always good.
>>
>> Regards,
>> Nicolas
>>
>>
>> On Fri, Oct 23, 2015 at 3:01 AM, Morten Olav Hansen <mortenoh@xxxxxxxxx>
>> wrote:
>>
>>> It's also fine splitting it up in many small pieces. As you say, you
>>> must ensure that the parents exists, so doing it one level at the time is
>>> the approach. If not, you must do import multiple times, so it can update
>>> the parents (not really a proper solution).
>>>
>>> Does the server have sufficient memory? because 37k OUs (8mb) should not
>>> really be a problem.. I have imported 50-60mb on my local machine before
>>> without any issues (although that was more than just OUs)
>>>
>>> --
>>> Morten
>>>
>>> On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau <
>>> ngermeau@xxxxxxxxxxxxxx> wrote:
>>>
>>>> Jason,
>>>>
>>>> I guess I'm going on the XSLT solution and split it per level, this is
>>>> the fastest line for me.
>>>>
>>>> Thanks a lot for the advices. Enjoy you mushrooms wanderings.
>>>>
>>>> See you
>>>> Nicolas
>>>>
>>>>
>>>>
>>>> On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering <
>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>
>>>>> Hi Nicolas
>>>>>
>>>>> Not sure there is a tool per se, but there are approaches.  Not sure
>>>>> where your data is coming from, but if you are transforming to JSON/XML, it
>>>>> may be just as easy to transform to SQL insert statements.
>>>>>
>>>>> The trick is to do it level by level and be sure you get the parent ID
>>>>> correct. Otherwise, set the parent level and be sure you reset the
>>>>> hibernate sequence of the database to avoid collisions with future inserts.
>>>>>
>>>>> Otherwise, maybe try what you have but in batches of a few thousand at
>>>>> a time since you seem to have it already in an importable format.
>>>>>
>>>>> Thanks. I need to do something other than  dhis2 and looking for
>>>>> mushrooms helps to clear the mind.
>>>>>
>>>>> Regards,
>>>>> Jason
>>>>>
>>>>> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hello Jason,
>>>>>>
>>>>>> Nope, no stacktrace. I suppose the transaction is too big and blow
>>>>>> the whole thing up. Yep sure, I don't really care about the method as long
>>>>>> as it doesn't take to much time to setup. I'm thinking about creating a
>>>>>> quick XSLT to extract org unit per level but if you have any tool/idea
>>>>>> which allow direct SQL I'm in.
>>>>>>
>>>>>> Nice mushroom picture by the way.
>>>>>> Regards.
>>>>>> Nicolas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
>>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>>
>>>>>>> Hi Nicolas,
>>>>>>>
>>>>>>> Hard to say without a stack trace.  It just bombs with no
>>>>>>> explanation?
>>>>>>>
>>>>>>> Could you consider using direct SQL injection for such a large
>>>>>>> number of orgunits?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Jason
>>>>>>>
>>>>>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm trying to make a rather large import of orgunit. The size of
>>>>>>>> the file is around 8megs zipped and it contains more than 37000 orgunits.
>>>>>>>>
>>>>>>>> When I'm launching the import it takes a little bit more than an
>>>>>>>> hour. I can see in the log file (dhis.log) outputs telling me that org unit
>>>>>>>> are being imported.  At the end I'm receiving the following message in
>>>>>>>> catalina.out
>>>>>>>>
>>>>>>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category:
>>>>>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done.
>>>>>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>>>>>>
>>>>>>>> and then the server crash without any message on what when wrong.
>>>>>>>> Nothing is imported.
>>>>>>>>
>>>>>>>> I'm wondering how the import works, can I split the file arbitrary
>>>>>>>> in multiple smaller parts (will it handle the fk parent id correctly) ? Or
>>>>>>>> do I have to split the file per level to ensure parent finding ?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Regards
>>>>>>>> Nicolas
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Nicolas Germeau*
>>>>>>>> *Software Developer*
>>>>>>>>
>>>>>>>> GSM:  +32 487 129820
>>>>>>>>
>>>>>>>> Tel: +32 2 737 65 16
>>>>>>>> Skype: nicolas.germeau
>>>>>>>>
>>>>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>>>>> Join us on Facebook
>>>>>>>> <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Nicolas Germeau*
>>>>>> *Software Developer*
>>>>>>
>>>>>> GSM:  +32 487 129820
>>>>>>
>>>>>> Tel: +32 2 737 65 16
>>>>>> Skype: nicolas.germeau
>>>>>>
>>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>>>
>>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Nicolas Germeau*
>>>> *Software Developer*
>>>>
>>>> GSM:  +32 487 129820
>>>>
>>>> Tel: +32 2 737 65 16
>>>> Skype: nicolas.germeau
>>>>
>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>> Visit our new website <http://www.bluesquare.org/>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>
>>
>> --
>> *Nicolas Germeau*
>> *Software Developer*
>>
>> GSM:  +32 487 129820
>>
>> Tel: +32 2 737 65 16
>> Skype: nicolas.germeau
>>
>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>> Visit our new website <http://www.bluesquare.org/>
>>
>
>


-- 
*Nicolas Germeau*
*Software Developer*

GSM:  +32 487 129820

Tel: +32 2 737 65 16
Skype: nicolas.germeau

Follow us on Twitter <https://twitter.com/bluesquare_org>
Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
Visit our new website <http://www.bluesquare.org/>