dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Nicolas

Not sure there is a tool per se, but there are approaches.  Not sure where
your data is coming from, but if you are transforming to JSON/XML, it may
be just as easy to transform to SQL insert statements.

The trick is to do it level by level and be sure you get the parent ID
correct. Otherwise, set the parent level and be sure you reset the
hibernate sequence of the database to avoid collisions with future inserts.

Otherwise, maybe try what you have but in batches of a few thousand at a
time since you seem to have it already in an importable format.

Thanks. I need to do something other than  dhis2 and looking for mushrooms
helps to clear the mind.

Regards,
Jason

On Thu, Oct 22, 2015, 18:57 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx> wrote:

> Hello Jason,
>
> Nope, no stacktrace. I suppose the transaction is too big and blow the
> whole thing up. Yep sure, I don't really care about the method as long as
> it doesn't take to much time to setup. I'm thinking about creating a quick
> XSLT to extract org unit per level but if you have any tool/idea which
> allow direct SQL I'm in.
>
> Nice mushroom picture by the way.
> Regards.
> Nicolas
>
>
>
> On Thu, Oct 22, 2015 at 5:34 PM, Jason Pickering <
> jason.p.pickering@xxxxxxxxx> wrote:
>
>> Hi Nicolas,
>>
>> Hard to say without a stack trace.  It just bombs with no explanation?
>>
>> Could you consider using direct SQL injection for such a large number of
>> orgunits?
>>
>> Regards,
>> Jason
>>
>> On Thu, Oct 22, 2015, 18:14 Nicolas Germeau <ngermeau@xxxxxxxxxxxxxx>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to make a rather large import of orgunit. The size of the
>>> file is around 8megs zipped and it contains more than 37000 orgunits.
>>>
>>> When I'm launching the import it takes a little bit more than an hour. I
>>> can see in the log file (dhis.log) outputs telling me that org unit are
>>> being imported.  At the end I'm receiving the following message in
>>> catalina.out
>>>
>>> * INFO  2015-10-22 15:26:50,878 [Level: INFO, category: METADATA_IMPORT,
>>> time: Thu Oct 22 15:26:50 GMT 2015, message: Import done. Completed in
>>> 4171.72 seconds.] (InMemoryNotifier.java [taskScheduler-1])
>>>
>>> and then the server crash without any message on what when wrong.
>>> Nothing is imported.
>>>
>>> I'm wondering how the import works, can I split the file arbitrary in
>>> multiple smaller parts (will it handle the fk parent id correctly) ? Or do
>>> I have to split the file per level to ensure parent finding ?
>>>
>>> Thanks
>>> Regards
>>> Nicolas
>>>
>>>
>>> --
>>> *Nicolas Germeau*
>>> *Software Developer*
>>>
>>> GSM:  +32 487 129820
>>>
>>> Tel: +32 2 737 65 16
>>> Skype: nicolas.germeau
>>>
>>> Follow us on Twitter <https://twitter.com/bluesquare_org>
>>> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
>>> Visit our new website <http://www.bluesquare.org/>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>
>
> --
> *Nicolas Germeau*
> *Software Developer*
>
> GSM:  +32 487 129820
>
> Tel: +32 2 737 65 16
> Skype: nicolas.germeau
>
> Follow us on Twitter <https://twitter.com/bluesquare_org>
> Join us on Facebook <https://www.facebook.com/BlueSquare.org?fref=ts>
> Visit our new website <http://www.bluesquare.org/>
>