dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Hi Saptarshi

2009/4/11 Saptarshi Purkayastha <sunbiz@xxxxxxxxx>:
> From all the experiments on the state servers and our office server, we have
> learnt a few lessons, some of which may be useful to other implementers and
> developers:
>
> We have started different tomcats on different ports and each tomcat uses
> 1GB max memory. The advantage of such a setup is that different state
> applications can be started/stopped/different memory
> allocations/threadpools, etc and one state application is not affecting the
> other state application... The disadvantage is that the port number in the
> URL can be discomforting to the user. So we created a domain and then each
> state has a subdomain mapped to the application. i.e. hp.nhsrc-hmis.org or
> bihar.nhsrc-hmis.org... We can play and move the state apps off
> servers/tomcats and no one realizes that we moved. Can be used to move apps
> when we know a load may be coming.
>
> In our office I was able to deploy 14 state apps on 6GB max memory to one
> tomcat ... but only when one app was deployed at a time from the manager
> application. or else the c3p0 bean would suggest a memory leak and the JDBC
> connection to MySQL would just get lost. This means whenever after 14 apps
> were deployed and tomcat was restarted, the apps would fail to deploy
> because all would deploy together.
>
> To find what caused the error, I tried to do a profiler run and found out
> some points of optimization. Memory areas where hibernate sessions were
> active went through frequent paging... something of a bottleneck of sorts.
> The c3p0 bean was creating Datasource pools even when we were not using them
> and probably would only use later after the application was fully running
> and doing something useful. Being able to remove hibernate and its mates
> altogether can make the application very much lighter...
>
> That brings me to an empirical question: Can JPA solve some of these memory
> issues?? Since it has learnt lessons from all the ORMs including hibernate
> and improvised a lot on EJB... I talked to abyot that we may want to try JPA
> in the CBS project and try to find an answer to that question.

I think finding where the optimizations are with hibernate and fixing
them is a more realistic short term strategy than migrating away from
hibernate.  That's a pretty major move.  Having said that I see we do
have some ibatis code in dhis as well.  Probably there is a historical
story here.

Besides optimizing the existing hibernate code, I think the correct
way to go is to start getting more rigorous about defining and using
functionality through the services/API/model layer.  There are a
couple of places where we may be bypassing altogether, and still
others where we have code in the dhis-web modules which needs to be
pushed down into the model.  Then we should certainly look at
optimizing the model - at some stage it might even include swapping
out the hibernate layer, but ideally only when we know this can be
done transparent to the layers above.  I've been looking quite a bit
at the hibernate related stuff and there is quite a bit of invested
work in here (I think Lars would know best).  I doubt if folk would
easily walk away from that.

> BTW, thanks bob for the suggestion. I tried to put the libs out to tomcat,
> but they only work when all are put... I couldn't make tomcat selective.
> That means we may have translations lost or all apps using the same
> language, since they are in the jars. Other than that, one loaded app can
> steal other apps resources... and when there is load on multiple apps
> running in this configuration, it still gives the JDBC connection lost
> exception... only difference is not during deployment, but when using the
> apps, which is more dangerous.

In the short term I agree you cannot do it this way.  There are too
many issues to resolve.  Longer term I think it would be good to
resolve all of the above.  Your working arrangement with the multiple
tomcat instances has some serious benefits regarding isolation, but is
less than optimal regarding resource allocation.  I guess there is a
balance to be struck - you can get even better isolation with
virtualisation but the costs keep rising ...

Are you running https?

Cheers
Bob

> ---
> Regards,
> Saptarshi PURKAYASTHA
> Director R & D, HISP India
> Health Information Systems Programme
>
> My Tech Blog:  http://sunnytalkstech.blogspot.com
> You Live by CHOICE, Not by CHANCE
>
>
> 2009/4/10 Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>>
>> On tomcat I put ALL jars in shared/lib.  And I have this at the bottom
>> of my server.xml:
>>
>>        <Context path= "/state1"
>> docBase="/home/bobj/dhis2/dhis.war"></Context>
>> <!--      <Context path= "/state2"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--       <Context path= "/state3"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state4"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state5"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state6"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state7"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state8"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state9"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state10"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state11"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state12"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state13"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state14"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state15"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state16"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state17"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state18"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state19"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>> <!--    <Context path= "/state20"
>> docBase="/home/bobj/dhis2/dhis.war"></Context> -->
>>      </Host>
>>    </Engine>
>>  </Service>
>> </Server>
>>
>> And yes, if I uncomment, I can start all 20 of these instances on my
>> 2M machine without any particular stress on my available Perm space.
>> They are all sharing the same jar so the difference between 1 and 20
>> is pretty marginal.  It does take a LONG time to load them all up
>> though.  Caveats:
>> 1.  I'm not saying this is a good idea!  Just a proof of concept.  One
>> badly behaved app could bring down all the others.
>> 2.  The big issue now becomes heap usage.  One of the areas we need to
>> think about optimizing some of the DHIS2 routines is in the use of
>> heap.  I haven't looked in detail yet, but for example, the reason why
>> we have sometimes had memory issues with import is that we seem to be
>> effectively pulling all the data into the heap before flushing the
>> hibernate cache.  This can be kind of a waste seeing that we are using
>> a SAX parser for the very reason that we might scale to very large
>> input streams.  Probably we can look at ways of chunking these sort of
>> operations to increase scalability.  In fact I'd be very surprised if
>> the Hibernate cache manager isn't smart enough to do this chunking
>> already.  Either way this kind of issue is solvable.
>> 3.  Obviously, with the above described arrangement, my database
>> connections are all messed up.  Still have to sort out the dhis2_home
>> context init_parameter.
>>
>> Cheers
>> Bob
>>
>> 2009/4/10 Knut Staring <knutst@xxxxxxxxx>:
>> > On Tue, Apr 7, 2009 at 9:47 AM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>> > wrote:
>> >>
>> >> If you pull out all the WEB-INF/lib/*.jar files from the war and place
>> >> those into tomcat's lib directory you are left with a dhis.war which
>> >> is only 1.8M.  (I did it on jetty, but same should apply to tomcat).
>> >
>> > For some reason, it seems it is not as straightforward as that with
>> > tomcat -
>> > if I move some .jar files from WEB-INF/lib Tomcat doesn't seem to find
>> > them
>> > anymore.
>> > Knut
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>
>