dhis2-users team mailing list archive

[Knut Staring <knutst@xxxxxxxxx>]

This discussion can be of interest to DHIS2 administrators as well.

---------- Forwarded message ----------
From: Saptarshi Purkayastha <sunbiz@xxxxxxxxx>
Date: Fri, Apr 12, 2013 at 11:25 PM
Subject: Re: Improving Tomcat memory parameters
To: Mark Goodrich <mgoodrich@xxxxxxx>
Cc: implementers@xxxxxxxxxxx, Rowan Seymour <rowanseymour@xxxxxxxxx>,
augustine12th thomas <augustine12th@xxxxxxxxx>, Gilbert Tuwei <
giltuwei2004@xxxxxxxxx>


If you are running dual processor system, definitely use CMS. If its a
single processor with quad-core still use it. I've seen people set one core
affinity to tomcat/java process, but I would recommend that in experimental
cases. If its a dual-core or worse, one of those fake-extra thread Intel
processors (I forget their name), don't enable CMS.

yes, I meant by default... It is also not meant to be compacting GC and
hence it's time to GC is unpredictable

The problem with jdk7 at the moment in OpenMRS is only compiler related.
The runtime issues with the infamous PorterStemmer bug or the Lucene/Solr
bugs were with the first couple of releases. Since, then I haven't seen any
in-the-face stability problems. Performance gains are quite good. If the
benchmarksgame@alioth wouldn't have removed java6, I could have shown some
evidence :-/


---
Regards,
Saptarshi PURKAYASTHA

My Tech Blog:  http://sunnytalkstech.blogspot.com
You Live by CHOICE, Not by CHANCE


On 12 April 2013 23:53, Mark Goodrich <mgoodrich@xxxxxxx> wrote:

>  A couple follow-on questions... (forgive my ignorance! :)
>
> How many cores would you recommend for using CMS?
>
> You said that CMS does not do class unloading... but yet the parameter I
> saw was "CMSClassUnloadingEnabled".  Do you mean that CMS doesn't do
> class unloading by default?
>
> I'll talk with others about moving up to jdk7, but I'm not sure if we want
> to risk the potential instability at this point...
>
> Take care,
> Mark
>
>
> On 04/12/2013 05:30 PM, Saptarshi Purkayastha wrote:
>
> Hi Mark,
>
> If you are using CMS for the JVM, you should ensure that your processor
> has enough cores, which I expect most modern servers would.
> But please check... or else your system would hang which GC goes on.
>
> I'll be a little scared to enable PermGenSweeping on OpenMRS or for that
> matter any system that uses Spring, lazy loading and localization.
> From my use of profiling OpenMRS core, PermGen creates a sawtooth graph if
> GC is enabled on PermGen. Unless you are really constrained on RAM, please
> do not enable it.
>
> On the other hand, class unloading might work much better. This is
> especially true for scripting languages that run on JVM. Rhino for sure
> speeds up quite a bit with CMS class unloading enabled. I've not tried
> JRuby or JPython, but I've read somewhere that these languages are the
> biggest gainers from G1 style compacting. CMS is not a compacting garbage
> collector and thus, doesn't do class unloading.
>
> So I'd recommend that you try enabling it and running YourKit or another
> profiler.
> Simplest I'd suggest using jdk7_update 4 and higher, which enables G1 by
> default
>
> ---
> Regards,
> Saptarshi PURKAYASTHA
>
> My Tech Blog:  http://sunnytalkstech.blogspot.com
> You Live by CHOICE, Not by CHANCE
>
>
> On 12 April 2013 16:57, Jeremy Keiper <jeremy@xxxxxxxxxxx> wrote:
>
>> AMPATH's production machine uses these values, and we have fought with
>> the PermGen errors before:
>>
>>  *-Xmx3584M -Xms512M  -XX:PermSize=512M -XX:MaxPermSize=512M
>> -XX:NewSize=256M*
>>
>>  We also turned on *-XX:+UseConcMarkSweepGC *
>>
>>  Right now we are fighting with CPU load, and have not explored the
>> options Mark just mentioned.
>>
>>
>>
>> Jeremy Keiper
>> OpenMRS Core Developer
>> AMPATH / IU-Kenya Support
>>
>>
>> On Fri, Apr 12, 2013 at 10:10 AM, Mark Goodrich <mgoodrich@xxxxxxx>wrote:
>>
>>>  Saptarshi,
>>>
>>> Thanks so much!  This is very helpful...
>>>
>>> +1 to Rowan's suggestion of updating the wiki... if you don't have time,
>>> let me know, and I can at least paste in what you've written here.
>>>
>>> This is a good page as well:
>>>
>>> https://wiki.openmrs.org/display/docs/Performance+Tuning
>>>
>>> Do you have any experience with these parameters?
>>>
>>> -XX:+CMSPermGenSweepingEnabled
>>> -XX:+CMSClassUnloadingEnabled**
>>>
>>>
>>> It looks like we are experiences a classloading leak due to the user of
>>> Groovy in the UI framework... I checked yesterday and there were 4,000+
>>> dead classloaders on our production most of which were Groovy
>>> classloaders.  Darius put a tweak into the UI Framework last night, but I
>>> don't think it has resolved it.
>>>
>>> Mark
>>>
>>>
>>>
>>> On 04/12/2013 04:09 AM, Rowan Seymour wrote:
>>>
>>> @Saptarshi maybe you could update
>>> https://wiki.openmrs.org/display/docs/Troubleshooting+Memory+Errors
>>>
>>>  I think every implementation runs into these kind of problems
>>> eventually and just googling for answers gives all sorts of conflicting
>>> advice.
>>>
>>>
>>> On 12 April 2013 00:21, Saptarshi Purkayastha <sunbiz@xxxxxxxxx> wrote:
>>>
>>>> To add to what I wrote earlier, so that this is not considered Voodoo
>>>> knowledge, lemme quote references:
>>>> http://www.oracle.com/technetwork/java/javase/6u18-142093.html
>>>> *Server JVM heap configuration ergonomics are now the same as the
>>>> Client, except that the default maximum heap size for 32-bit JVMs is 1
>>>> gigabyte, corresponding to a physical memory size of 4 gigabytes, and for
>>>> 64-bit JVMs is 32 gigabytes, corresponding to a physical memory size of 128
>>>> gigabytes*.
>>>>
>>>> So, by a calculation that the default JVM memory allocations have some
>>>> science behind it, they allocate 1/4 total physical memory size. On your
>>>> 32GB server, lets consider MySQL has been given 16GB, so 1/4th of the
>>>> remaining I'd recommend 4G for heap.
>>>>
>>>> Also one should move to JDK7 for the G1 garbage collector's efficiency
>>>> (once we figure out why those 2 tests randomly end in errors, but not fail)
>>>>
>>>> http://www.oracle.com/technetwork/java/javase/tech/g1-intro-jsp-135488.html
>>>>
>>>>
>>>> ---
>>>> Regards,
>>>> Saptarshi PURKAYASTHA
>>>>
>>>> My Tech Blog:  http://sunnytalkstech.blogspot.com
>>>> You Live by CHOICE, Not by CHANCE
>>>>
>>>>
>>>>   On 11 April 2013 23:59, Saptarshi Purkayastha <sunbiz@xxxxxxxxx>wrote:
>>>>
>>>>> Firstly expecting that this is a 64-bit OS with 64-bit JVM,
>>>>>
>>>>> I'd have PermSize and MaxPermSize different, probably double for a
>>>>> server with 32GB RAM
>>>>>    -XX:PermSize=256m -XX:MaxPermSize=512m
>>>>>
>>>>> If Java activities and MySQL and equally prioritized on the server.
>>>>> I'd still give 4Gb max heap size for Java.
>>>>>    -Xmx4G -Xms1024m
>>>>>
>>>>> If you have enough RAM for everything else running on the server and
>>>>> its not paging, 4Gb heap size shouldn't be a problem.
>>>>> Heap thrashing is a misnomer coming from early IBM implementations.
>>>>> Sun/Oracle JVM is really well managed
>>>>>
>>>>> You are also probably using sun java6 (hotspot jvm), I'd enable the
>>>>> CMS (Concurrent Mark & Sweep Garbage collector)
>>>>>    -XX:+UseConcMarkSweepGC
>>>>>
>>>>> ---
>>>>> Regards,
>>>>> Saptarshi PURKAYASTHA
>>>>>
>>>>> My Tech Blog:  http://sunnytalkstech.blogspot.com
>>>>> You Live by CHOICE, Not by CHANCE
>>>>>
>>>>>
>>>>> On 11 April 2013 23:33, Ellen Ball <ellnball@xxxxxxxxx> wrote:
>>>>>
>>>>>> We've experienced the pain of tomcat crashing on an OpenMRS 1.9.3
>>>>>> production server with this memory error:  "Java.lang.OutOfMemoryError:
>>>>>>  PermGen"
>>>>>>
>>>>>>    - The server has 32GB of RAM.
>>>>>>    - tomcat is  using JAVA_OPTS = "-Xmx512m -Xms512m
>>>>>>    -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=128m"
>>>>>>
>>>>>> Has anyone else seen this error?  Has anyone experimented with
>>>>>> increased memory parameter values?  It seems that the Xmx and Xms values
>>>>>> should be the same for a server ("In a server environment, you
>>>>>> normally want Xms and Xmx set to the same value to avoid heap thrashing.").
>>>>>>  The heap space shouldn't be too high, but if anyone know the optimal size
>>>>>> that would be helpful.
>>>>>>
>>>>>>  Thanks,
>>>>>>
>>>>>>  Ellen Ball
>>>>>> Partners In Health
>>>>>>
>>>>>>
>>>>>>   --
>>>>>> OpenMRS Implementers: http://go.openmrs.org/implementers
>>>>>> Post: implementers@xxxxxxxxxxx
>>>>>> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
>>>>>> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> --
>>>> OpenMRS Implementers: http://go.openmrs.org/implementers
>>>> Post: implementers@xxxxxxxxxxx
>>>> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
>>>> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>  --
>>> *Rowan Seymour*
>>> tel: +250 783835665 <%2B250%20783835665>
>>>  --
>>> OpenMRS Implementers: http://go.openmrs.org/implementers
>>> Post: implementers@xxxxxxxxxxx
>>> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
>>> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>>>
>>>
>>>
>>>
>>>    --
>>> OpenMRS Implementers: http://go.openmrs.org/implementers
>>> Post: implementers@xxxxxxxxxxx
>>> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
>>> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>>>
>>>
>>>
>>
>>    --
>> OpenMRS Implementers: http://go.openmrs.org/implementers
>> Post: implementers@xxxxxxxxxxx
>> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
>> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>>
>>
>>
>
> --
> OpenMRS Implementers: http://go.openmrs.org/implementers
> Post: implementers@xxxxxxxxxxx
> Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
> Manage your OpenMRS subscriptions at https://id.openmrs.org/
>
>
>
>
>
 --
OpenMRS Implementers: http://go.openmrs.org/implementers
Post: implementers@xxxxxxxxxxx
Unsubscribe: implementers+unsubscribe@xxxxxxxxxxx
Manage your OpenMRS subscriptions at https://id.openmrs.org/





-- 
Knut Staring
Dept. of Informatics, University of Oslo
+4791880522
http://dhis2.org