dhis2-devs team mailing list archive

[Lars Helge Øverland <lars@xxxxxxxxx>]

On Mon, Mar 21, 2016 at 8:14 AM, David Siang Fong Oh <doh@xxxxxxxxxxxxxxxx>
wrote:

> Thanks Lars.
>
> I've re-run the performance benchmark and the issue seems to be resolved.
> The metadata export is taking approximately 15 seconds to download a 50MB
> file.
>

Sounds good. Thanks for testing.


> The file size seems to have increased which I'm assuming is related to the
> refactoring of the export service by Morten?
>
>
Okay - will need Morten's input on that.

regards,

Lars


> Cheers,
>
> -doh
>
> On Mon, Mar 14, 2016 at 9:26 PM, Lars Helge Øverland <lars@xxxxxxxxx>
> wrote:
>
>> Hi David,
>>
>> that seems very reasonable. I suspect the problem is that
>> SystemSettingManager is transactional, meaning that you get one new
>> transaction per call to the get method. I have done a fix now in trunk (rev
>> 22301) and 2.22 for SystemSettingManager where a transaction is only
>> initiated whenever there is a cache miss in the get method. Since calendar
>> hardly changes the cache hit rate should be close to 100%.
>>
>> Are you able to re-run your performance benchmark and see if this did the
>> trick?
>>
>> regards,
>>
>> Lars
>>
>>
>>
>>
>> On Sun, Jan 10, 2016 at 2:34 PM, David Siang Fong Oh <
>> doh@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi team,
>>>
>>> We've noticed a performance issue when exporting metadata through the
>>> API. The issue appears to have been introduced in revision 20839
>>> <http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/2.21/revision/20839> on
>>> the 2.21 branch (revision 20854
>>> <http://bazaar.launchpad.net/~dhis2-devs-core/dhis2/trunk/revision/20854> on
>>> trunk) where local caching of systemSettings in the DefaultCalendarService
>>> was removed in favour of leveraging the Guava caching in the
>>> DefaultSystemSettingManager.
>>>
>>> Performance impact when downloading full metadata export through API on
>>> a development machine (16GB RAM, 2.6 GHz quad-core CPU):
>>>
>>>    - Before (revision 20838 on 2.21 branch): ~11 seconds for a ~20MB
>>>    file
>>>    - After (revision 20839 on 2.21 branch): ~180 seconds for a ~20MB
>>>    file
>>>
>>> The caching performance issue seems to manifest itself when mapping the
>>> period dates for charts and reportTables which requires the system calendar
>>> setting (stacktrace below). Excluding charts and reportTables from the
>>> metadata export resolves the issue (download takes ~10 seconds for a ~13 MB
>>> file).
>>>
>>> Have you experienced any similar issues with the Guava cache implemented
>>> in the DefaultSystemSettingManager (or anywhere else)? I wanted to check
>>> before investigating any further.
>>>
>>> Cheers,
>>>
>>> -doh
>>>
>>> Stacktrace when mapping period dates for charts:
>>>
>>> *at
>>> org.hisp.dhis.setting.DefaultSystemSettingManager.getSystemSetting(DefaultSystemSettingManager.java:157)*
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>> at
>>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>>> at
>>> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
>>> at
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
>>> at
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>>> at
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
>>> at com.sun.proxy.$Proxy58.getSystemSetting(Unknown Source)
>>> *at
>>> org.hisp.dhis.calendar.DefaultCalendarService.getSystemCalendar(DefaultCalendarService.java:102)*
>>> *at org.hisp.dhis.period.PeriodType.getCalendar(PeriodType.java:101)*
>>> *at
>>> org.hisp.dhis.period.PeriodType.createLocalDateUnitInstance(PeriodType.java:364)*
>>> *at org.hisp.dhis.period.PeriodType.getIsoDate(PeriodType.java:501)*
>>> *at org.hisp.dhis.period.Period.getIsoDate(Period.java:158)*
>>> at
>>> org.hisp.dhis.common.adapter.JacksonPeriodSerializer.serialize(JacksonPeriodSerializer.java:48)
>>> at
>>> org.hisp.dhis.common.adapter.JacksonPeriodSerializer.serialize(JacksonPeriodSerializer.java:42)
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Lars Helge Øverland
>> Lead developer, DHIS 2
>> University of Oslo
>> Skype: larshelgeoverland
>> http://www.dhis2.org <https://www.dhis2.org/>
>>
>>
>


-- 
Lars Helge Øverland
Lead developer, DHIS 2
University of Oslo
Skype: larshelgeoverland
http://www.dhis2.org <https://www.dhis2.org/>