dhis2-devs team mailing list archive

[Sam Johnson <samuel.johnson@xxxxxxxxxx>]

Hi Calle,

> BTW, also keep an eye on spelling - I see in your source fields that you have "chronich ypertension" and "current Progranancy outcome".

That RMNCH Tracker isn’t our program, I’ve just used it to replicate the bugs I found in our own programs.  (We always try to replicate any bugs we find in Oslo’s own Play demo site, so the developers can easily see what we’re talking about.)

The programs I was talking about are the EquityTool surveys (http://www.equitytool.org/dhis2/), and all of the codes and names are automatically-generated, to ensure they’re consistent across countries – plus we rigorously test and check everything before release.  I’ll buy you lunch if you find a spelling mistake in one of our programs.  ;-)

Cheers, Sam.


From: Calle Hedberg <calle.hedberg@xxxxxxxxx>
Reply-To: "calle.hedberg@xxxxxxxxx" <calle.hedberg@xxxxxxxxx>
Date: Sunday, 30 April 2017 at 09:38
To: Sam Johnson <samuel.johnson@xxxxxxxxxx>
Cc: DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx>, Markus Bekken <markus@xxxxxxxxx>, Kenzo Fry <kenzo.fry.consultant@xxxxxxxxx>, Andrea Sprockett <andrea@xxxxxxxxxx>
Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam

My main issue with hyphens are usually related the source of variables or data elements or indicators etc being in a MS Word document, where Word tend to automatically replace hyphens with em-dash (the "long" hyphen). Underscore on the other hand has been common as a delimiter for time immemorial, and I've never seen it being used as a special character in any type of syntax.

BTW, also keep an eye on spelling - I see in your source fields that you have "chronich ypertension" and "current Progranancy outcome". Spelling mistakes often results in people adding multiple variables for the same thing but with different spelling. You cannot really set up check algorithms for spelling (beyond what's in the system by default) the same way you can set them up to block special characters, so it requires a keen human eye.

My 2c worth
Calle

On 28 April 2017 at 12:47, Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>> wrote:
Hi Calle,

I heartily agree, and I do actually use camelCase or similar (without spaces) for all codes.  However, the one departure from strict Camel Case that I quite frequently use is to partition codes with dashes (hyphens), because we use very structured naming conventions, and they can quickly become illegible without proper partitioning – eg in ‘EQT-CM11-Q06a’ it’s very easy to immediately pick out the country, year and question number, whereas it’s much more difficult in ‘EQTCM11Q06a’.  (It sounds like you add underscores for the same purpose.)

Markus has noted in his ticket that both dashes and underscores should be supported in DHIS2 codes, and I’ve never had problems with dashes before, but given your very long history with DHIS2, I might draw from your experience, and shift to using underscores for future projects…  ;-)

In the meantime, at Markus’ suggestion I’ve just created a new bug ticket for this, linked to the longer-term validation ‘improvement’ ticket that he created:
https://jira.dhis2.org/browse/DHIS2-1503

Cheers, Sam.



From: Calle Hedberg <calle.hedberg@xxxxxxxxx<mailto:calle.hedberg@xxxxxxxxx>>
Reply-To: "calle.hedberg@xxxxxxxxx<mailto:calle.hedberg@xxxxxxxxx>" <calle.hedberg@xxxxxxxxx<mailto:calle.hedberg@xxxxxxxxx>>
Date: Friday, 28 April 2017 at 06:37
To: Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>
Cc: Markus Bekken <markus@xxxxxxxxx<mailto:markus@xxxxxxxxx>>, Kenzo Fry <kenzo.fry.consultant@xxxxxxxxx<mailto:kenzo.fry.consultant@xxxxxxxxx>>, Andrea Sprockett <andrea@xxxxxxxxxx<mailto:andrea@xxxxxxxxxx>>, DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>>

Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Sam,

I understand your frustration with already published and deployed programs, and that you hope for some software fix that will make them work.

For everybody else, I would strongly recommend the naming practice that I've been following (and beating into all my colleagues in HISP-South Africa) since DHIS version 1.0 beta was released in May 1997: NEVER, repeat NEVER, use any non-alphanumeric character except underscore for any meta-data or variable name if you can avoid it. The problem is that ampersands, asterisks, forward and backslashes, equal signs, hashes - almost every special character like that is a restricted character in SOME context. Yes, we have escape sequences and all that - but look through the dev list over the last few years, and you see these kind of issues coming up again and again.

In 1.4 we blocked the most common problems (double spaces, leading spaces, all these special characters, em-dash instead of hyphen, etc etc) in the database itself, so users could not add them in the first place, but that method was dropped off when DHIS 1.4 was "ported" to DHIS2.

Regards
Calle

On 28 April 2017 at 00:32, Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>> wrote:
Hi Markus,

Just following up my question below – is there any chance we could get this treated as a bug, and fixed in 2.26?  This issue has crippled 30-odd programs that Metrics for Management (M4M) has already published and deployed, but I’m guessing that the fix is probably just be a very simple string-handling change…

Many thanks for any help you might be able to give!

Cheers, Sam.


From: Dhis2-devs <dhis2-devs-bounces+samuel.johnson=qebo.co.uk@xxxxxxxxxxxxxxxxxxx<mailto:qebo.co.uk@xxxxxxxxxxxxxxxxxxx>> on behalf of Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>
Date: Friday, 21 April 2017 at 17:58
To: Markus Bekken <markus@xxxxxxxxx<mailto:markus@xxxxxxxxx>>
Cc: Kenzo Fry <kenzo.fry.consultant@xxxxxxxxx<mailto:kenzo.fry.consultant@xxxxxxxxx>>, Andrea Sprockett <andrea@xxxxxxxxxx<mailto:andrea@xxxxxxxxxx>>, DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>>
Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26


This sender failed our fraud detection checks and may not be who they appear to be. Learn about spoofing<http://aka.ms/LearnAboutSpoofing>

Feedback<http://aka.ms/SafetyTipsFeedback>

Hi Markus (and other devs),

Many thanks for diagnosing this so quickly.  You’re spot-on, it’s the dash in the source name that’s causing the rules to fail – if I remove dashes from all source names, the rules fire without problems.  In fact, this seems wider than just dashes – program rules that were failing in another instance seem to now work if I similarly remove brackets and spaces from source names.

Would it be at all possible to get this treated as a bug and fixed in 2.26, rather than treated as an improvement for a future version of DHIS2?  These rules were all working perfectly when we did intensive testing on our programs in 2.25 (eg they work properly in 2.25 da6a2df / 2017-02-02 06:43), so this limitation seems to have been recently introduced (very likely by accident, since other element names allow dashes), and it’s breaking configurations that have worked well up until now.

The reason we’re so keen on this is that Metrics for Management has published XMLs for 29 EquityTool programs, and these have now been downloaded and installed in a wide range of DHIS2 instances.   It will be a huge job to not only update the source names in all of these 29 programs and re-publish them, but also contact all existing users and ask them to reinstall the corrected programs.  If there’s any chance we could get the program rules in 2.26 working as they did before, this would save us an enormous amount of work…

Many thanks for any help you might be able to give us with this!

Regards, Sam.


From: Markus Bekken <markus@xxxxxxxxx<mailto:markus@xxxxxxxxx>>
Date: Wednesday, 19 April 2017 at 14:55
To: Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>
Cc: DHIS2 Developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>>, Kenzo Fry <kenzo.fry.consultant@xxxxxxxxx<mailto:kenzo.fry.consultant@xxxxxxxxx>>
Subject: Re: [Dhis2-devs] Many program rules not firing in 2.26

Hey Sam!
Thanks for reporting.

I tested WHO RMNCH and 'Hide program stage' on Play, and found that it the problem is in fact what is indicated by the console message in point 4 below. The source field #{currentProganancyOutcome} did not exist. Seems someone has renamed it to
#{Current Progranancy Outcome} - see below to the left:
[cid:image001.png@01D2C1A1.BC9CE520]

Such a name change should ideally clean up the expressions using the source field, or at least flag the ones that has become illegal as a result of the name change. At the moment however you would have to manually update the expressions after a name change. Updated the expression now on Play, and the expression runs fine:
Expression with id rule:xOm49QX4Nsc was successfully run. Original condition was: #{current Progranancy Outcome}  != 'Live birth' - Evaluation ended up as:''  != 'Live birth' - Result of evaluation was:true
[cid:image001.png@01D2C1A1.BC9CE520]

In general your rules hiding program stages should work fine, let me know if you find any other problems here.

Looking into the second part of the mail regarding Event Capture - this is a different problem: There is a dash in the source field name. Registered an improvement in Jira based on this: https://jira.dhis2.org/browse/DHIS2-1449 - but avoiding dashes is a possible workaround.

Best regards,
Markus

19. apr. 2017 kl. 12.43 skrev Sam Johnson <samuel.johnson@xxxxxxxxxx<mailto:samuel.johnson@xxxxxxxxxx>>:

Hi Devs,

I’m finding that a number of program rules aren’t firing in 2.26 – this is happening with both Event Capture and Tracker Capture.

This includes the sample ‘Hide Program Stage’ rule which has been added to the WHO RMNCH Tracker for the 2.26 release.  To reproduce in Play demo:
1.       add a new registration to the WHO RMNCH Tracker with date 2017-01-01
2.       complete the initial visit, and click ‘cancel’ to prevent creation of a second antenatal care visit
3.       click ‘+’ to create a ‘Care at birth’ event on 2017-02-01
4.       for WHOMCH pregnancy outcome, select ‘Stillbirth’, and complete the event.
The rule which should hide the postnatal program stage has not fired.  In the browser console, the following two errors are recorded are:
•         Expression #{currentProgranancyOutcome}  != 'Live birth' contains variable currentProgranancyOutcome - but this variable is not defined.
•         Expression with id rule:xOm49QX4Nsc could not be run. Original condition was: #{currentProgranancyOutcome}  != 'Live birth' - Evaluation ended up as:#{currentProgranancyOutcome}  != 'Live birth' - error message:SyntaxError: Invalid or unexpected token

The important error seems to be the second one, as in many cases this is appearing in isolation (without the first one), and is still breaking program rules.  For example, I’ve loaded the Cameroon EquityTool into the Play Demo instance, and any attempt to enter new Event Capture events for this program keeps generating similar error messages in the browser console, eg:
•         Expression with id rule:gXwzJFfRn5m could not be run. Original condition was: #{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - Evaluation ended up as:#{EQT-CM11Q06a} + #{EQT-CM11Q06b} >1 - error message:SyntaxError: Invalid or unexpected token
These program rules have been extensively tested on earlier releases, and worked well – where users used to be prevented from entering invalid surveys, they can now save invalid surveys.

This is very serious, as it means that in 2.26, users can now bypass many program rules, and save incomplete or incorrect data.  But I’m also aware that there have recently been build issues with 2.25 and 2.26 releases – before I log a bug on Jira, could this be related to those build issues?

Regards, Sam.

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp


_______________________________________________
Mailing list: https://launchpad.net/~dhis2-devs
Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx<mailto:dhis2-devs@xxxxxxxxxxxxxxxxxxx>
Unsubscribe : https://launchpad.net/~dhis2-devs
More help   : https://help.launchpad.net/ListHelp



--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@xxxxxxxxx<mailto:calle.hedberg@xxxxxxxxx>

Skype: calle_hedberg

*******************************************




--

*******************************************

Calle Hedberg

46D Alma Road, 7700 Rosebank, SOUTH AFRICA

Tel/fax (home): +27-21-685-6472

Cell: +27-82-853-5352

Iridium SatPhone: +8816-315-19119

Email: calle.hedberg@xxxxxxxxx<mailto:calle.hedberg@xxxxxxxxx>

Skype: calle_hedberg

*******************************************