dhis2-devs team mailing list archive

Thread
Date
Re: [Bug 1370486] Re: orgunit-merge-problem-report-tables

To: Rodolfo Melia <rmelia@xxxxxxxxxxxx>
From: Jason Pickering <jason.p.pickering@xxxxxxxxx>
Date: Fri, 25 Sep 2015 16:27:07 +0200
Cc: Bug 1370486 <1370486@xxxxxxxxxxxxxxxxxx>, "dhis2-devs@xxxxxxxxxxxxxxxxxxx" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <CAOShsgvCGcnE7pG8W4AboUz-CJ0pnaETe-+V4una3K-uJvgQkA@mail.gmail.com>
Hi Rodolfo,
Yes, it would be nice, but let me explain a bit our thinking.

There were numerous issues with the implementation, and after some
discussion among the dev team, it was decided it was safer to remove it
entirely for now, until a further review and possible re-implementation
could be done.

The code was quite old, and did not handle properly things like the
"attributeOptionCombo". If this was in use, the functionality could have
caused really major problems when data overlapped. Also, it did not handle
events, among other things. Additionally, the default logic was to "SUM"
numeric data when the data was overlapping, although this was not really
totally apparent unless you looked very carefully at the code, and may not
have really been intended. In the end, it was too risky to keep it in, and
I would strongly advise everyone not to use it.

As Lars says, we are in the process of developing some SQL scripts to
handle this operation, which is something which usually does not need to be
done frequently, and since it is destructive, needs to be done with great
care. The dev team is currently reviewing my initial scripts for this, and
we will release it very soon. Even these scripts will not handle all cases
for what to do when merging data, for instance, how to handle non-numeric
data, like text or options sets. It gets complicated when you want to try
and figure out how to merge this stuff, and may be very implementation
specific.

Its never nice to lose functionality, but this was deemed to be too risky
to keep in.

Maybe we can put it back in the future with some more code, but since it
again is something which can be handled by SQL, and which may need to be
reconfigured per implementation, it may be difficult to have a general
functionality which will handle every situation.

Regards,
Jason



On Fri, Sep 25, 2015 at 4:07 PM, Rodolfo Melia <rmelia@xxxxxxxxxxxx> wrote:

> Hi Jason - losing this functionality is a shame. Wouldn't be better trying
> to get it fixed?
>
> *Rodolfo Meliá*
> *Principal  |  *rmelia@xxxxxxxxxxxx
> Skype: rod.melia  |  +44 777 576 4090  |  +1 708 872 7636
> www.knowming.com
>
> On Thu, Sep 24, 2015 at 5:47 PM, jason.p.pickering <
> 1370486@xxxxxxxxxxxxxxxxxx> wrote:
>
>> OU merge functionality has been removed and will be replaced with
>> administrative SQL scripts due to various implementation scenarios.
>>
>> ** Changed in: dhis2
>>        Status: New => Fix Committed
>>
>> --
>> You received this bug notification because you are a member of DHIS 2
>> developers, which is subscribed to DHIS.
>> https://bugs.launchpad.net/bugs/1370486
>>
>> Title:
>>   orgunit-merge-problem-report-tables
>>
>> Status in DHIS:
>>   Fix Committed
>>
>> Bug description:
>>   Version:
>>   2.15
>>   Build revision:
>>   15196
>>
>>   When orgunits which are part of a report table are attempted to be
>>   merged with another orgunit, the following error results.
>>
>>   DELETE FROM reporttable_organisationunits where organisationunitid =
>>   76018;
>>
>>   followed by clearing the cache is a workaround.
>>
>>   * INFO  2014-09-17 08:15:05,583 DELETE FROM datavalue_audit WHERE
>> sourceid=76018; (JdbcDataMergeStore.java [http-bio-8080-exec-81])
>>   * INFO  2014-09-17 08:15:05,584 DELETE FROM datavalue WHERE
>> sourceid=76018; (JdbcDataMergeStore.java [http-bio-8080-exec-81])
>>   * INFO  2014-09-17 08:15:05,815 DELETE FROM completedatasetregistration
>> WHERE sourceid=76018; (JdbcDataMergeStore.java [http-bio-8080-exec-81])
>>   * ERROR 2014-09-17 08:15:05,848 Error while executing action
>> (ExceptionInterceptor.java [http-bio-8080-exec-81])
>>   org.hisp.dhis.common.DeleteNotAllowedException: ReportTable
>>           at
>> org.hisp.dhis.system.deletion.DefaultDeletionManager.execute(DefaultDeletionManager.java:102)
>>           at
>> org.hisp.dhis.system.deletion.DeletionInterceptor.intercept(DeletionInterceptor.java:54)
>>           at sun.reflect.GeneratedMethodAccessor2549.invoke(Unknown
>> Source)
>>           at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>           at
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>>           at
>> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:603)
>>           at
>> org.springframework.aop.aspectj.AspectJMethodBeforeAdvice.before(AspectJMethodBeforeAdvice.java:39)
>>           at
>> org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
>>           at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>           at
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>           at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>           at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>           at com.sun.proxy.$Proxy54.deleteOrganisationUnit(Unknown Source)
>>           at
>> org.hisp.dhis.datamerge.DefaultDataMergeService.mergeOrganisationUnits(DefaultDataMergeService.java:129)
>>           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>           at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>           at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>           at
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>           at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>           at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>           at
>> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96)
>>           at
>> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260)
>>           at
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
>>           at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>           at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>           at com.sun.proxy.$Proxy118.mergeOrganisationUnits(Unknown
>> Source)
>>           at
>> org.hisp.dhis.dataadmin.action.organisationunitmerge.MergeOrganisationUnitsAction.execute(MergeOrganisationUnitsAction.java:95)
>>           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>           at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>           at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>           at java.lang.reflect.Method.invoke(Method.java:606)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:450)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:289)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:252)
>>           at
>> org.hisp.dhis.webportal.interceptor.XWorkPortalUserInterceptor.intercept(XWorkPortalUserInterceptor.java:88)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.webportal.interceptor.XWorkPortalModuleInterceptor.intercept(XWorkPortalModuleInterceptor.java:89)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.webportal.interceptor.XWorkPortalMenuInterceptor.intercept(XWorkPortalMenuInterceptor.java:97)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.webportal.interceptor.XWorkPortalParamsInterceptor.intercept(XWorkPortalParamsInterceptor.java:147)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.interceptor.ContextInterceptor.intercept(ContextInterceptor.java:83)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.interceptor.UserSettingInterceptor.intercept(UserSettingInterceptor.java:81)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.security.intercept.LoginInterceptor.intercept(LoginInterceptor.java:85)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.security.intercept.XWorkSecurityInterceptor.intercept(XWorkSecurityInterceptor.java:113)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.interceptor.SystemSettingInterceptor.intercept(SystemSettingInterceptor.java:112)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:136)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:249)
>>           at
>> com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.hisp.dhis.interceptor.ExceptionInterceptor.intercept(ExceptionInterceptor.java:110)
>>           at
>> com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:246)
>>           at
>> org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
>>           at
>> org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:562)
>>           at
>> org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:77)
>>           at
>> org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
>>           at
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
>>           at
>> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.hisp.dhis.security.filter.CustomAuthenticationFilter.doFilter(CustomAuthenticationFilter.java:64)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.hisp.dhis.security.filter.AutomaticAccessFilter.doFilter(AutomaticAccessFilter.java:113)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
>>           at
>> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
>>           at
>> org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
>>           at
>> org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
>>           at
>> org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
>>           at
>> org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>           at
>> org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:149)
>>           at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>           at
>> org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
>>           at
>> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>           at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>           at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>           at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>           at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>           at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>           at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>           at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
>>           at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>           at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>           at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
>>           at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>           at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>           at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>           at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>           at java.lang.Thread.run(Thread.java:744)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/dhis2/+bug/1370486/+subscriptions
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+46764147049
References

[Bug 1370486] [NEW] orgunit-merge-problem-report-tables
From: jason.p.pickering, 2014-09-17
[Bug 1370486] Re: orgunit-merge-problem-report-tables
From: jason.p.pickering, 2015-09-24
Re: [Bug 1370486] Re: orgunit-merge-problem-report-tables
From: Rodolfo Melia, 2015-09-25