dhis2-devs team mailing list archive

[Morten Olav Hansen <mortenoh@xxxxxxxxx>]

Ok, if you are still having issues (and can share the org units) feel free
to email me the file and I will have a look.

--
Morten

On Fri, Oct 23, 2015 at 3:53 PM, Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
wrote:

> Hello Morten,
>
> Thanks for the advice. For what I know the server looks big enough (4gb
> mem). I might try to scale it to 8gb change some vm parameters and see how
> it works. But knowing that I have the backup solution to split it in level
> is always good.
>
> Regards,
> Nicolas
>
>
> On Fri, Oct 23, 2015 at 3:01 AM, Morten Olav Hansen <mortenoh@xxxxxxxxx>
> wrote:
>
>> It's also fine splitting it up in many small pieces. As you say, you must
>> ensure that the parents exists, so doing it one level at the time is the
>> approach. If not, you must do import multiple times, so it can update the
>> parents (not really a proper solution).
>>
>> Does the server have sufficient memory? because 37k OUs (8mb) should not
>> really be a problem.. I have imported 50-60mb on my local machine before
>> without any issues (although that was more than just OUs)
>>
>> --
>> Morten
>>
>> On Fri, Oct 23, 2015 at 12:21 AM, Nicolas Germeau <
>> ngermeau@xxxxxxxxxxxxxx> wrote:
>>
>>> Jason,
>>>
>>> I guess I'm going on the XSLT solution and split it per level, this is
>>> the fastest line for me.
>>>
>>> Thanks a lot for the advices. Enjoy you mushrooms wanderings.
>>>
>>> See you
>>> Nicolas
>>>
>>>
>>>
>>> On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering <
>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>
>>>> Hi Nicolas
>>>>
>>>> Not sure there is a tool per se, but there are approaches.  Not sure
>>>> where your data is coming from, but if you are transforming to JSON/XML, it
>>>> may be just as easy to transform to SQL insert statements.
>>>>
>>>> The trick is to do it level by level and be sure you get the parent ID
>>>> correct. Otherwise, set the parent level and be sure you reset the
>>>> hibernate sequence of the database to avoid collisions with future inserts.
>>>>
>>>> Otherwise, maybe try what you have but in batches of a few thousand at
>>>> a time since you seem to have it already in an importable format.
>>>>
>>>> Thanks. I need to do something other than  dhis2 and looking for
>>>> mushrooms helps to clear the mind.
>>>>
>>>> Regards,
>>>> Jason
>>>>
>>>> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hello Jason,
>>>>>
>>>>> Nope, no stacktrace. I suppose the transaction is too big and blow the
>>>>> whole thing up. Yep sure, I don't really care about the method as long as
>>>>> it doesn't take to much time to setup. I'm thinking about creating a quick
>>>>> XSLT to extract org unit per level but if you have any tool/idea which
>>>>> allow direct SQL I'm in.
>>>>>
>>>>> Nice mushroom picture by the way.
>>>>> Regards.
>>>>> Nicolas
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>
>>>>>> Hi Nicolas,
>>>>>>
>>>>>> Hard to say without a stack trace.  It just bombs with no
>>>>>> explanation?
>>>>>>
>>>>>> Could you consider using direct SQL injection for such a large number
>>>>>> of orgunits?
>>>>>>
>>>>>> Regards,
>>>>>> Jason
>>>>>>
>>>>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm trying to make a rather large import of orgunit. The size of the
>>>>>>> file is around 8megs zipped and it contains more than 37000 orgunits.
>>>>>>>
>>>>>>> When I'm launching the import it takes a little bit more than an
>>>>>>> hour. I can see in the log file (dhis.log) outputs telling me that org unit
>>>>>>> are being imported.  At the end I'm receiving the following message in
>>>>>>> catalina.out
>>>>>>>
>>>>>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category:
>>>>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done.
>>>>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>>>>>
>>>>>>> and then the server crash without any message on what when wrong.
>>>>>>> Nothing is imported.
>>>>>>>
>>>>>>> I'm wondering how the import works, can I split the file arbitrary
>>>>>>> in multiple smaller parts (will it handle the fk parent id correctly) ? Or
>>>>>>> do I have to split the file per level to ensure parent finding ?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Regards
>>>>>>> Nicolas
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Nicolas Germeau*
>>>>>>> *Software Developer*
>>>>>>>
>>>>>>> GSM:  +32 487 129820
>>>>>>>
>>>>>>> Tel: +32 2 737 65 16
>>>>>>> Skype: nicolas.germeau
>>>>>>>
>>>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>>>> Join us on Facebook
>>>>>>> <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Nicolas Germeau*
>>>>> *Software Developer*
>>>>>
>>>>> GSM:  +32 487 129820
>>>>>
>>>>> Tel: +32 2 737 65 16
>>>>> Skype: nicolas.germeau
>>>>>
>>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>>> Visit our new website <http://www.bluesquare.org/>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Nicolas Germeau*
>>> *Software Developer*
>>>
>>> GSM:  +32 487 129820
>>>
>>> Tel: +32 2 737 65 16
>>> Skype: nicolas.germeau
>>>
>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>> Visit our new website <http://www.bluesquare.org/>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
> --
> *Nicolas Germeau*
> *Software Developer*
>
> GSM:  +32 487 129820
>
> Tel: +32 2 737 65 16
> Skype: nicolas.germeau
>
> Follow us on Twitter <https://twitter.com/bluesquare_org>
> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
> Visit our new website <http://www.bluesquare.org/>
>