dhis2-devs team mailing list archive

[Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>]

Jason,

I guess I'm going on the XSLT solution and split it per level, this is the
fastest line for me.

Thanks a lot for the advices. Enjoy you mushrooms wanderings.

See you
Nicolas



On Thu, Oct 22, 2015 at 6:09 PM, Jason Pickering <
jason.p.pickering@xxxxxxxxx> wrote:

> Hi Nicolas
>
> Not sure there is a tool per se, but there are approaches.  Not sure where
> your data is coming from, but if you are transforming to JSON/XML, it may
> be just as easy to transform to SQL insert statements.
>
> The trick is to do it level by level and be sure you get the parent ID
> correct. Otherwise, set the parent level and be sure you reset the
> hibernate sequence of the database to avoid collisions with future inserts.
>
> Otherwise, maybe try what you have but in batches of a few thousand at a
> time since you seem to have it already in an importable format.
>
> Thanks. I need to do something other than  dhis2 and looking for mushrooms
> helps to clear the mind.
>
> Regards,
> Jason
>
> On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
> wrote:
>
>> Hello Jason,
>>
>> Nope, no stacktrace. I suppose the transaction is too big and blow the
>> whole thing up. Yep sure, I don't really care about the method as long as
>> it doesn't take to much time to setup. I'm thinking about creating a quick
>> XSLT to extract org unit per level but if you have any tool/idea which
>> allow direct SQL I'm in.
>>
>> Nice mushroom picture by the way.
>> Regards.
>> Nicolas
>>
>>
>>
>> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
>> jason.p.pickering@xxxxxxxxx> wrote:
>>
>>> Hi Nicolas,
>>>
>>> Hard to say without a stack trace.  It just bombs with no explanation?
>>>
>>> Could you consider using direct SQL injection for such a large number of
>>> orgunits?
>>>
>>> Regards,
>>> Jason
>>>
>>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm trying to make a rather large import of orgunit. The size of the
>>>> file is around 8megs zipped and it contains more than 37000 orgunits.
>>>>
>>>> When I'm launching the import it takes a little bit more than an hour.
>>>> I can see in the log file (dhis.log) outputs telling me that org unit are
>>>> being imported.  At the end I'm receiving the following message in
>>>> catalina.out
>>>>
>>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category:
>>>> METADATA_IMPORT, time: Thu Oct 22 15:26:50 GMT 2015, message: Import done.
>>>> Completed in 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>>
>>>> and then the server crash without any message on what when wrong.
>>>> Nothing is imported.
>>>>
>>>> I'm wondering how the import works, can I split the file arbitrary in
>>>> multiple smaller parts (will it handle the fk parent id correctly) ? Or do
>>>> I have to split the file per level to ensure parent finding ?
>>>>
>>>> Thanks
>>>> Regards
>>>> Nicolas
>>>>
>>>>
>>>> --
>>>> *Nicolas Germeau*
>>>> *Software Developer*
>>>>
>>>> GSM:  +32 487 129820
>>>>
>>>> Tel: +32 2 737 65 16
>>>> Skype: nicolas.germeau
>>>>
>>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>>> Visit our new website <http://www.bluesquare.org/>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>
>>
>> --
>> *Nicolas Germeau*
>> *Software Developer*
>>
>> GSM:  +32 487 129820
>>
>> Tel: +32 2 737 65 16
>> Skype: nicolas.germeau
>>
>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>> Visit our new website <http://www.bluesquare.org/>
>>
>


-- 
*Nicolas Germeau*
*Software Developer*

GSM:  +32 487 129820

Tel: +32 2 737 65 16
Skype: nicolas.germeau

Follow us on Twitter <https://twitter.com/bluesquare_org>
Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
Visit our new website <http://www.bluesquare.org/>