dhis2-devs team mailing list archive

[Saptarshi Purkayastha <sunbiz@xxxxxxxxx>]

>From all the experiments on the state servers and our office server, we have
learnt a few lessons, some of which may be useful to other implementers and
developers:

We have started different tomcats on different ports and each tomcat uses
1GB max memory. The advantage of such a setup is that different state
applications can be started/stopped/different memory
allocations/threadpools, etc and one state application is not affecting the
other state application... The disadvantage is that the port number in the
URL can be discomforting to the user. So we created a domain and then each
state has a subdomain mapped to the application. i.e. hp.nhsrc-hmis.org or
bihar.nhsrc-hmis.org... We can play and move the state apps off
servers/tomcats and no one realizes that we moved. Can be used to move apps
when we know a load may be coming.

In our office I was able to deploy 14 state apps on 6GB max memory to one
tomcat ... but only when one app was deployed at a time from the manager
application. or else the c3p0 bean would suggest a memory leak and the JDBC
connection to MySQL would just get lost. This means whenever after 14 apps
were deployed and tomcat was restarted, the apps would fail to deploy
because all would deploy together.

To find what caused the error, I tried to do a profiler run and found out
some points of optimization. Memory areas where hibernate sessions were
active went through frequent paging... something of a bottleneck of sorts.
The c3p0 bean was creating Datasource pools even when we were not using them
and probably would only use later after the application was fully running
and doing something useful. Being able to remove hibernate and its mates
altogether can make the application very much lighter...

That brings me to an empirical question: Can JPA solve some of these memory
issues?? Since it has learnt lessons from all the ORMs including hibernate
and improvised a lot on EJB... I talked to abyot that we may want to try JPA
in the CBS project and try to find an answer to that question.

BTW, thanks bob for the suggestion. I tried to put the libs out to tomcat,
but they only work when all are put... I couldn't make tomcat selective.
That means we may have translations lost or all apps using the same
language, since they are in the jars. Other than that, one loaded app can
steal other apps resources... and when there is load on multiple apps
running in this configuration, it still gives the JDBC connection lost
exception... only difference is not during deployment, but when using the
apps, which is more dangerous.
---
Regards,
Saptarshi PURKAYASTHA
Director R & D, HISP India
Health Information Systems Programme

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE


2009/4/10 Bob Jolliffe <bobjolliffe@xxxxxxxxx>

> On tomcat I put ALL jars in shared/lib.  And I have this at the bottom
> of my server.xml:
>
>        <Context path= "/state1"
> docBase="/home/bobj/dhis2/dhis.war"></Context>
> <!--      <Context path= "/state2"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--       <Context path= "/state3"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state4"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state5"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state6"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state7"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state8"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state9"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state10"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state11"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state12"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state13"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state14"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state15"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state16"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state17"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state18"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state19"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
> <!--    <Context path= "/state20"
> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>      </Host>
>    </Engine>
>  </Service>
> </Server>
>
> And yes, if I uncomment, I can start all 20 of these instances on my
> 2M machine without any particular stress on my available Perm space.
> They are all sharing the same jar so the difference between 1 and 20
> is pretty marginal.  It does take a LONG time to load them all up
> though.  Caveats:
> 1.  I'm not saying this is a good idea!  Just a proof of concept.  One
> badly behaved app could bring down all the others.
> 2.  The big issue now becomes heap usage.  One of the areas we need to
> think about optimizing some of the DHIS2 routines is in the use of
> heap.  I haven't looked in detail yet, but for example, the reason why
> we have sometimes had memory issues with import is that we seem to be
> effectively pulling all the data into the heap before flushing the
> hibernate cache.  This can be kind of a waste seeing that we are using
> a SAX parser for the very reason that we might scale to very large
> input streams.  Probably we can look at ways of chunking these sort of
> operations to increase scalability.  In fact I'd be very surprised if
> the Hibernate cache manager isn't smart enough to do this chunking
> already.  Either way this kind of issue is solvable.
> 3.  Obviously, with the above described arrangement, my database
> connections are all messed up.  Still have to sort out the dhis2_home
> context init_parameter.
>
> Cheers
> Bob
>
> 2009/4/10 Knut Staring <knutst@xxxxxxxxx>:
> > On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>
> wrote:
> >>
> >> If you pull out all the WEB-INF/lib/*.jar files from the war and place
> >> those into tomcat's lib directory you are left with a dhis.war which
> >> is only 1.8M.  (I did it on jetty, but same should apply to tomcat).
> >
> > For some reason, it seems it is not as straightforward as that with
> tomcat -
> > if I move some .jar files from WEB-INF/lib Tomcat doesn't seem to find
> them
> > anymore.
> > Knut
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>