dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

I think Lars is probably the definitive source here, but let me offer my
views.

I think this has nothing to do with the database (MySQL or Postgresql) but
rather how DHIS2 does the aggregation. A big chunk of data will get pulled
into memory. If the data is too big and the memory is not enough, this
error will result. I have seen it many times.  We have had similar issues
with our database, when aggregating historical data in one chunk. So,
instead, I simply do it year by year, piece by piece. This is fairly easy
to automate with a few CURL calls. We are using Postgres, and have had
similar issues, which have been solved by
 1) Increasing the PermSpace parameter
 2) Aggregate less data at a time.

Switching to Postgres may have some other advantage, but I do not think it
will solve this issue. Properly tuned, it should work fine.

Regards,
Jason


On Tue, Sep 11, 2012 at 4:06 PM, Hannan Khan <hannank@xxxxxxxxx> wrote:

> Dear Knut
>
> Actually I never use PostgreSQL. So I asked  few of friends and they
> suggests me that if the application are tuned and used in MySQL then stay
> with that.
>
> As you are suggesting to use PostgreSQL can you suggest how I could I
> shift database from MySQL to PostgreSQL?
>
> Regards
>
> Hannan
>
>
> On Tue, Sep 11, 2012 at 4:29 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
>
>> Perhaps this is a case where it would make sense to switch to
>> PostgreSQL and possibly also make use of more of the internal
>> crunching capabilities of the database?
>>
>> Knut
>>
>> On Tue, Sep 11, 2012 at 10:09 AM, Hannan Khan <hannank@xxxxxxxxx> wrote:
>> > I might found an alternative solution. Yesterday we run data mart for 9
>> > month and it was completed successfully but when we run for 5 years, the
>> > errors arrived.
>> >
>> > Today and trying with again and get back to you all with the result. But
>> > when we run datamart for 9 moth it takes 34 minutes, which is long is
>> not
>> > it?
>> >
>> > Regards
>> >
>> > Hannan
>> >
>> >
>> > On Tue, Sep 11, 2012 at 2:01 PM, Morten Olav Hansen <mortenoh@xxxxxxxxx
>> >
>> > wrote:
>> >>
>> >> That's a really big database... I think maybe Lars should answer if the
>> >> memory of the server is large enough (I'm sure he can compare it with
>> what
>> >> he has been doing to tune the Kenya server).
>> >>
>> >> --
>> >> Morten
>> >>
>> >>
>> >>
>> >> On Tue, Sep 11, 2012 at 2:59 PM, Hannan Khan <hannank@xxxxxxxxx>
>> wrote:
>> >>>
>> >>> Hi Morten
>> >>>
>> >>> Server have 8GB memory and database is 3159MB. We are using MySQL.
>> >>>
>> >>> Regards
>> >>>
>> >>> Hannan
>> >>>
>> >>>
>> >>> On Tue, Sep 11, 2012 at 1:53 PM, Morten Olav Hansen <
>> mortenoh@xxxxxxxxx>
>> >>> wrote:
>> >>>>
>> >>>> Hi
>> >>>>
>> >>>> How much memory does your server have? and what is the size of your
>> >>>> database?
>> >>>>
>> >>>> --
>> >>>> Morten
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, Sep 11, 2012 at 2:51 PM, Hannan Khan <hannank@xxxxxxxxx>
>> wrote:
>> >>>>>
>> >>>>> -XX:MaxPermSize=2048m and -XX:PermSize=512m
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Sep 11, 2012 at 11:27 AM, Jason Pickering
>> >>>>> <jason.p.pickering@xxxxxxxxx> wrote:
>> >>>>>>
>> >>>>>> The PergGen space is the important option.  What are you
>> JAVA_OPTS.?
>> >>>>>>
>> >>>>>> --
>> >>>>>> Sent from my mobile
>> >>>>>>
>> >>>>>> On Sep 11, 2012 7:25 AM, "Hannan Khan" <hannank@xxxxxxxxx> wrote:
>> >>>>>>>
>> >>>>>>> Thanks Jason.
>> >>>>>>>
>> >>>>>>> I put the Java heap space 1GB and Maximum 6GB (Xmx6144m and
>> >>>>>>> Xms1024m). But still getting 'java.lang.OutOfMemoryError: Java
>> heap space'
>> >>>>>>> error. That why I asked that whether that is a bug or not.
>> >>>>>>>
>> >>>>>>> I am trying to other options and contact you soon.
>> >>>>>>>
>> >>>>>>> Regards
>> >>>>>>>
>> >>>>>>> Hannan
>> >>>>>>>
>> >>>>>>> On Mon, Sep 10, 2012 at 11:40 AM, Jason Pickering
>> >>>>>>> <jason.p.pickering@xxxxxxxxx> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi Hannan,
>> >>>>>>>>
>> >>>>>>>> You configuration may have worked previously, but some things
>> were
>> >>>>>>>> changed in the datamart and you will need to allocate more
>> memory to
>> >>>>>>>> it after 2.7.
>> >>>>>>>>
>> >>>>>>>> The details are here of the server setup are here
>> >>>>>>>>
>> >>>>>>>>
>> http://dhis2.org/doc/snapshot/en/implementer/html/ch08.html#d5e413
>> >>>>>>>>
>> >>>>>>>> and the important part is the JAVA_OPTS environment variable, for
>> >>>>>>>> instance...
>> >>>>>>>>
>> >>>>>>>> export JAVA_OPTS='-Xmx6000m -Xms3000m -XX:MaxPermSize=800m
>> >>>>>>>> -XX:PermSize=400m'
>> >>>>>>>>
>> >>>>>>>> I am not really sure what your settings are, but you should at
>> least
>> >>>>>>>> specify these options, of course adjusting them for the amount of
>> >>>>>>>> memory which can be allocated to DHIS2 .
>> >>>>>>>>
>> >>>>>>>> Best regards,
>> >>>>>>>> Jason
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Mon, Sep 10, 2012 at 10:49 AM, Hannan Khan <hannank@xxxxxxxxx
>> >
>> >>>>>>>> wrote:
>> >>>>>>>> > Dear Jason
>> >>>>>>>> >
>> >>>>>>>> > Thanks for your advice. We are running dhis2 from 2010. In
>> earlier
>> >>>>>>>> > version
>> >>>>>>>> > we dont have any problem creating datamart. After updating to
>> >>>>>>>> > version 2.7 we
>> >>>>>>>> > imported a big chuck of data from two data souce and after
>> that we
>> >>>>>>>> > have
>> >>>>>>>> > problem creating datamart.
>> >>>>>>>> >
>> >>>>>>>> > So far I know the configuration is ok, however I will check
>> again
>> >>>>>>>> > and get
>> >>>>>>>> > back to you.
>> >>>>>>>> >
>> >>>>>>>> > Regards
>> >>>>>>>> >
>> >>>>>>>> > Hannan
>> >>>>>>>> > from Samsung Galaxy
>> >>>>>>>> >
>> >>>>>>>> > On Sep 9, 2012 5:39 PM, "Jason Pickering"
>> >>>>>>>> > <jason.p.pickering@xxxxxxxxx>
>> >>>>>>>> > wrote:
>> >>>>>>>> >>
>> >>>>>>>> >> This is not a bug. You must increase the heap size. Please
>> >>>>>>>> >> consult the
>> >>>>>>>> >> implementation manual for details.
>> >>>>>>>> >>
>> >>>>>>>> >> Regards,
>> >>>>>>>> >> Jason
>> >>>>>>>> >>
>> >>>>>>>> >> --
>> >>>>>>>> >> Sent from my mobile
>> >>>>>>>> >>
>> >>>>>>>> >> On Sep 9, 2012 1:12 PM, "Hannan Khan" <hannank@xxxxxxxxx>
>> wrote:
>> >>>>>>>> >>>
>> >>>>>>>> >>> We are using dhis2 version 2.7. Whenever we generating
>> datamart
>> >>>>>>>> >>> are it
>> >>>>>>>> >>> taking long time and it never ends.The error message is (in
>> the
>> >>>>>>>> >>> tomcat log
>> >>>>>>>> >>> file)
>> >>>>>>>> >>>
>> >>>>>>>> >>>
>> 'dhis2-org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
>> >>>>>>>> >>> processChildren SEVERE: Exception invoking periodic
>> operation:
>> >>>>>>>> >>> java.lang.OutOfMemoryError: Java heap space'.
>> >>>>>>>> >>>
>> >>>>>>>> >>> I attached the log file.
>> >>>>>>>> >>>
>> >>>>>>>> >>> Regards
>> >>>>>>>> >>>
>> >>>>>>>> >>> Hannan Khan
>> >>>>>>>> >>>
>> >>>>>>>> >>> National Data Warehouse
>> >>>>>>>> >>>
>> >>>>>>>> >>>
>> >>>>>>>> >>>
>> >>>>>>>> >>>
>> >>>>>>>> >>> _______________________________________________
>> >>>>>>>> >>> Mailing list: https://launchpad.net/~dhis2-devs
>> >>>>>>>> >>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> >>>>>>>> >>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> >>>>>>>> >>> More help   : https://help.launchpad.net/ListHelp
>> >>>>>>>> >>>
>> >>>>>>>> >
>> >>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Mailing list: https://launchpad.net/~dhis2-devs
>> >>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> >>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> >>>>> More help   : https://help.launchpad.net/ListHelp
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Mailing list: https://launchpad.net/~dhis2-devs
>> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> > Unsubscribe : https://launchpad.net/~dhis2-devs
>> > More help   : https://help.launchpad.net/ListHelp
>> >
>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> +4791880522
>> http://dhis2.org
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>