dhis2-devs team mailing list archive

[Sam Johnson <samuel.johnson@xxxxxxxxxx>]

Hi Markus,

Many thanks for looking into this, I know the code for these program indicators is complex!  Will this be fixed in a future release?  (I realise it’s probably not a high priority for 2.25 if there’s a known workaround.)

And many thanks for clarifying that I now need to specify a custom aggregation for V{event_count} – I’ll put that in now, so the eventual upgrade to 2.25 goes smoothly.  ☺

Cheers, Sam.


From: Markus Bekken <markus.bekken@xxxxxxxxx>
Date: Friday, 7 October 2016 at 08:44
To: Sam Johnson <samuel.johnson@xxxxxxxxxx>
Cc: DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx>, Lars Helge Øverland <lars@xxxxxxxxx>
Subject: Re: [Dhis2-devs] [Bug 1612375] [NEW] Program indicators: error in percentage calculations when using d2:condition (inconsistent results at different levels of aggregation)

Hi there Sam,
thank you for the detailed report.

The division is done with integers when both the inputs are integers. So the result of the divison is rounded down from f.ex. 0,66 to 0 before the multiplication with 100 happens. You found a workaround yourself, another option would be to do the multiplication before the division:
•         sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0)) * 100 / V{event_count}

P.S. To get the given example to work on the latest 2.24 and 2.25 snapshot with custom aggregation it was also necessary to specify aggregation operator for the event count variable:
•         sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0)) * 100 / count(V{event_count})

Best regards,
Markus

25. aug. 2016 kl. 15.13 skrev Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>:

Hi all,

Just following up on this bug report again – is there any chance I could get it looked at/prioritized?

I’ve dug a little deeper, and it looks like this bug could be as simple as an error in variable handling.

Specifically, although in the example I originally gave, the following DOESN’T work:
•         sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0)) / V{event_count} * 100
the following DOES seem to work:
•         sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',100,0)) / V{event_count}

Is it possible that when the aggregation step of the indicator calculation is undertaken, any decimal results are at some point forced to an integer, generating incorrect values wherever the result is <1?

Cheers,

Sam.


From: Dhis2-devs <dhis2-devs-bounces+samuel.johnson=qebo.co.uk@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs-bounces+samuel.johnson=qebo.co.uk@xxxxxxxxxxxxxxxxxxx>> on behalf of Samuel Scott <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>
Date: Tuesday, 23 August 2016 at 15:08
To: DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>>, Lars Helge Øverland <lars@xxxxxxxxx<mailto:lars@xxxxxxxxx>>
Subject: Re: [Dhis2-devs] [Bug 1612375] [NEW] Program indicators: error in percentage calculations when using d2:condition (inconsistent results at different levels of aggregation)

Hi all,

I reported a bug in the web program indicator calculations a couple of weeks ago, and included in the report everything that’s needed to quickly replicate the bug, but I don’t think anyone has had a chance to look at this report yet.

Have I followed the correct process for reporting this?  Is it at all possible to get an idea of when a fix for this might be available?  (It’s quite a serious bug, as ‘valid’ expressions are returning incorrect/inconsistent results at different levels of aggregation.)

Many thanks,

Sam.


From: Dhis2-devs <dhis2-devs-bounces+samuel.johnson=qebo.co.uk@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs-bounces+samuel.johnson=qebo.co.uk@xxxxxxxxxxxxxxxxxxx>> on behalf of Samuel Scott <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>
Reply-To: Bug 1612375 <1612375@xxxxxxxxxxxxxxxxxx<mailto:1612375@xxxxxxxxxxxxxxxxxx>>
Date: Thursday, 11 August 2016 at 19:21
To: DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>>
Subject: [Dhis2-devs] [Bug 1612375] [NEW] Program indicators: error in percentage calculations when using d2:condition (inconsistent results at different levels of aggregation)

Public bug reported:

Bug found in 2.22, still present in 2.24 online demo (2.24 rev.23557)
Online demo accessed using Chrome 52.0.2743.116.

There appears to be a bug in the program indicator calculations when:
a sum() custom aggregation is used twice, to enable the calculation of a percentage AND
an element of the percentage calculation includes d2:condition.

Note that either of these situations works properly on its own - ie the
numerator and denominator work perfectly when separated into two
different indicators - it is only when combined that the bug occurs.

TO REPRODUCE THIS BUG

In the online demo instance, using the Antenatal Care Visit program:

1. Create a program indicator called TestNumerator, with aggregation type 'custom' and expression:
sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0))

2. Create a program indicator called TestDenominator, with aggregation type 'custom' and expression:
V{event_count}

3. Create a program indicator called TestPercentage, with aggregation type 'custom' and expression, which combines these two expressions as a percentage:
sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0)) / V{event_count} * 100

4. Create a pivot table based on the Antenatal program, add all three indicators to the pivot, and run it:
Numerator and denominator calculate correctly, but percentage returns 2/3 = 0 (instead of 66.7)

5. Now change the Period from Last 12 months to Last 52 weeks, and re-run:
All indicators now calculate correctly, with percentage correctly showing 0/1 = 0 and 2/2 = 100.


Note that this is not a bug in the event_count variable, as the same bug consistently appears even when V{event_count} is replaced by sum(1) in the indicator expressions.

I have a project whose viability depends on being able to correctly
calculate these percentages at higher levels of aggregation (regions
etc), so if confirmed as a bug, I would be hugely grateful if it were
possible to fix this in the current release, rather than in 2.25.

Many thanks, Sam.

** Affects: dhis2
     Importance: Undecided
         Status: New

--
You received this bug notification because you are a member of DHIS 2
developers, which is subscribed to DHIS.
https://bugs.launchpad.net/bugs/1612375

Title:
  Program indicators: error in percentage calculations when using
  d2:condition (inconsistent results at different levels of aggregation)

Status in DHIS:
  New

Bug description:
  Bug found in 2.22, still present in 2.24 online demo (2.24 rev.23557)
  Online demo accessed using Chrome 52.0.2743.116.

  There appears to be a bug in the program indicator calculations when:
  a sum() custom aggregation is used twice, to enable the calculation of a percentage AND
  an element of the percentage calculation includes d2:condition.

  Note that either of these situations works properly on its own - ie
  the numerator and denominator work perfectly when separated into two
  different indicators - it is only when combined that the bug occurs.

  TO REPRODUCE THIS BUG

  In the online demo instance, using the Antenatal Care Visit program:

  1. Create a program indicator called TestNumerator, with aggregation type 'custom' and expression:
  sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0))

  2. Create a program indicator called TestDenominator, with aggregation type 'custom' and expression:
  V{event_count}

  3. Create a program indicator called TestPercentage, with aggregation type 'custom' and expression, which combines these two expressions as a percentage:
  sum(d2:condition('#{dBwrot7S420.vANAXwtLwcT}>=10',1,0)) / V{event_count} * 100

  4. Create a pivot table based on the Antenatal program, add all three indicators to the pivot, and run it:
  Numerator and denominator calculate correctly, but percentage returns 2/3 = 0 (instead of 66.7)

  5. Now change the Period from Last 12 months to Last 52 weeks, and re-run:
  All indicators now calculate correctly, with percentage correctly showing 0/1 = 0 and 2/2 = 100.


  Note that this is not a bug in the event_count variable, as the same bug consistently appears even when V{event_count} is replaced by sum(1) in the indicator expressions.

  I have a project whose viability depends on being able to correctly
  calculate these percentages at higher levels of aggregation (regions
  etc), so if confirmed as a bug, I would be hugely grateful if it were
  possible to fix this in the current release, rather than in 2.25.

  Many thanks, Sam.

To manage notifications about this bug go to:
https://bugs.launchpad.net/dhis2/+bug/1612375/+subscriptions

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp