dhis2-devs team mailing list archive

[Ola Hodne Titlestad <olatitle@xxxxxxxxx>]

Hi,

After some more testing of data set, data set section and data entry forms
and Bob's more relaxed attitude on sdmx vs DHIS models I am a bit more
positive and have some suggestions (from an implementer's perspective) on
short and long term improvements to data sets and forms.

*Some general thought on this:
** For the time being the data set sections seem like a suitable object were
we can group data elements that belong to the same catcombo, for use in data
entry forms, and for data export purposes. They play the role of form
sections.

* I managed to create a dataset with mutliple catcombos, put these in
separate sections, and then designed a custom form for data entry that
included multiple tables/catcombos. I can even combine multiple combos into
one monster table if I like, it's all flexible down to the "data element +
catoptioncombo" level. So for entering in custom forms and storing
dimensional data the current model seem to work ok (not thinking of sdmx).
*
Some short term improvements (by early January latest to support new forms
for next year):
*
* Right now the data set sections are not supported in the default data
entry forms, which means you have to design custom forms to make use of
them. This should be fixed as it should be possible to have default forms
with more than 1 catcombo/table. A simple version would be to list these
tables 1 by 1, under each other with a table heading taken from the data set
section label property (which already exists for some reason).

* (Bob's) Sections on a form should have the constraint that they only allow
dataelements of similar dimensionality. (Ola's): Right now there is no such
constraint. If not, it will all be chaos in default tabular forms and messy
for sdmx I assume.

* When creating a new dataset with data elements from different catcombos
the data set sections should automatically be created, one per catcombo and
data elements allocated to these sections.

* Same as above for data set reports (forms with data for any level and
period), so that these static reports look exactly like the forms.

*Long term (3-4 months maybe?):
** Find a better name for data set section, possibly remodel a dataset,
form, data value set model (to cater for data entry in default and custom
mode, import/export dhis2dhis, and reporting via sdmx)

* It should be possible to make minor adjustments to what the default data
entry forms look like (not completely custom form). Default data entry form
view properties should include things like adding headings to each table
(catcombo), selecting which fields to grey out (make inactive), freely
select whether to put data element or catoptions on columns/rows (pivot the
table). In a more advanced version (>3 months) even be able to position
these tables on a screen grid e.g. 3x3 or 2x2, like the drag and drop portal
stuff that's around, e.g iGoogle.

* Data set reports should make use of the view properties to default forms
(the previous point) so that they look exactly like the forms

* Even more flexible data entry forms with multiple orgunits or periods and
possibility to pivot these around in a table

* It should be possible to create forms that combine data for a facility, a
household and a patient/client. Saptarshi had some ideas on how Xforms might
support that.

There's probably much more, but that's what I've got from an implementer's
perspective at least. I'm sure you guys have comments and additions. Let's
get this discussion done quickly and into a blueprint and into action so
that all the people working on creating new data sets and forms for 2010 can
get their work done more easily.


Ola
-----------

2009/12/10 Bob Jolliffe <bobjolliffe@xxxxxxxxx>

> Ola,
>
> I am sorry I have set you off on this path of datasets with common
> categorycombos but I am now having doubts .... see below.
>
>  2009/12/10 Ola Hodne Titlestad <olatitle@xxxxxxxxx>
>
>  Hi,
>>
>> I have brought up this discussion a couple of times before and good to see
>> it moving now.
>>
>> I like the idea of allowing multiple catcombos in one data entry form, and
>> I think this should also go for default non-custom forms, as in many cases
>> it would be a matter of listing multiple tables on a page, one table per
>> catcombo.
>>
>> We are facing exactly this challenge now that we are working on revising
>> the Sierra Leone database for 2010. We want to use multidimensional data
>> elements as much as possible and that typically means having 1-5 catcombos
>> per paper form. At the same time we want this database to be compatible with
>> the new SDMX-HD reporting so that it can easily work together with OpenMRS
>> and other software. Then we need to make sure that our datasets only have 1
>> catcombo, if not the sdmx export will have trouble finding and structuring
>> the data. Our problem then becomes how to create data entry forms that look
>> exactly like the paper forms when such a form typically will have more then
>> 1 catcombo? So we are under a bit of pressure here to be able to come up
>> with such forms before they start entering data for 2010 in beginning of
>> February next year.
>>
>> I agree with Bob's view that a data entry form is the combined thing, what
>> you see and where you enter data, typically the same as the paper form. In
>> order to support multiple catcombos (which to me means multiple datasets),
>> then I think it is natural to be able to link a data entry form to n data
>> sets. And a dataset can be used in multiple forms, that still needs to be
>> supported. The datasets become the crucial database design part, where we
>> define how data is modelled in the DHIS. The forms as I see it are simply
>> views on top of that which can make data entry easier and be more like the
>> paper experience.
>>
>
> After considerable reflection I have softened in some respects on this on
> hardened in others.  I do think we need to have a notion of a set of
> reported data - whether that data has arrived through a data entry
> screen/form or sucked in from the ether.  That is a particular set of
> datavalues rather than a set of dataelements.  The Dataset has always been
> approximately (but not quite) this abstraction - I guess I am talking about
> a Dataset instance - a ReportedDataSet.  I think this is in some ways
> similar to our completedDataSetRegistration.
>
> But I have also come to understand the fact that SDMX-HD datasets are
> obliged to be of the same dimensionality as being a limitation of SDMX and
> not necessarily the shining example of best practice in our field.  When we
> read in data (say an sdmx package from openmrs which might be a group of
> SDMX datasets) its not really that important that we have to match 1-1 those
> SDMX datasets onto DHIS datasets.  In fact there is no loss of meaningful
> information if they all go into the same DHIS dataset.  As long as we can
> pull back out values byCategoryCombo() then that is fine.
>
> And I suspect the same thing is true in reverse - when we are reporting
> data rather than receiving reported data.  That is a more complicated
> problem which I haven't fully got my head around yet (its different because
> we would be dealing mostly with aggregated indicator values with no
> categorycombos) but again DHIS just needs to produce the reported data set.
> If that data has to be subdivided further to parcel into SDMX messages that
> is a downstream issue which shouldn't necessarily affect constraints on our
> model.
>
> So what about the forms/screens?  Well I think we must cleanly separate
> model-view issues.  The requirement to group dataentry elements in different
> sized tables is primarily a UI issue which should be taken care of at the UI
> layer.  The UI might have screens with forms or screens with sections or
> what have you.  And those sections might have a common dimensionality
> requirement but I don't think this has necessarily got to do with how we
> model datasets.  Currently I think sections are a property of the DataSet -
> the SectionManager has getSectionsByDataSet() for example.  This I am now
> suspecting is the wrong approach.  One data set for reporting instrument
> (whether we call that a screen or a form is immaterial as long as we agree
> on something) makes sense.  And each section on that form should have a
> requirement that only dataelements from the dataset which share the same
> dimensionality are eligle for inclusion in that section.  But these are
> sections on the form not anything structurally meaningful to the DataSet.  I
> think this can be enforced without fragmenting our current model.
>
> So in summary (my list):
> 1.  discount my earlier concerns about sdmx datasets - they are not the
> same as dhis datasets and we can workaround the difference.
> 2.  think about something to represent an actual set of reported
> (electronic or screen-entered) values.  There is metadata regarding this
> from sdmx which we might currently have to discard.  It might be a
> refinement of completedDataSetRegistartion.
> 3.  sections on a form should have the constraint that they only allow
> dataelements of similar dimensionality.
> 4.  If the UI (the web forms) requires a breakdown of datasets by
> categorycombo (or dimensionality) for the purpose of layout then keep that
> mechanism in the view layer.  Sometimes the view needs its own micro-model
> ...
>
> Bob.
>
>
>
>
>> It seems we are thinking along the same lines, just use different names
>> for the various parts.
>>
>> I think a group of data elements sharing the same catcombo should be a
>> dataset, as this also best matches the terminology of SDMX-HD where one
>> dataset will contain data that share the same dimensions.
>>
>> *Whatever the names we go for my main requirements are:
>> *1) need for an artefact that groups together data elements that share
>> the same category combo and also define the allowed periodicity (periodType)
>> for these data elements
>> 2) when entering data it should be possible to see multiple tables with
>> different dimensions in the same form
>> - these tables should be generated by default (like today for 1 catcombo)
>> from catcombos without a need to design them manually
>> 3) it should be able to make minor adjustments to what the default data
>> entry form looks (not completely custom form) by e.g. adding headings for
>> each table (catcombo), selecting which fields to grey out (make inactive),
>> freely select whether to put data element or catoptions on columns/rows
>> (pivot the table)
>> 4) Like today offer a custom form designer with full freedom to link to
>> data elements+catoptioncombos from multiple catcombos
>> 5) When exporting data we again need easily to be able to pick data
>> elements that share the same category combo ("sdmx datasets")
>> 6) In report tables we need to filter by category combos and get options
>> from these (already implemented)
>>
>> It also seems like this improvement would be equally important to both
>> routine and patient data.
>>
>> Ola
>> ----------
>>
>>
>>
>> I like the idea of separating forms (or screens) from data sets and to
>> allow multiple data sets (or groups of data elements with the same catcombo)
>> in one form.
>>
>>
>> 2009/12/10 Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>>
>>> Hi Abyot
>>>
>>> I like your suggestion very much but want (again I know!) to suggest a
>>> slightly different slant. I tend to think of a form as being the composite
>>> thing (like a paper form which might even have a reference number - form
>>> ANC-B3).  And that form might have different screens and/or sections.
>>> Rather than the screen having multiple forms.  Just a matter of naming I
>>> know, but if we are going to refactor like this, we do need to think about
>>> what is the least confusing to users.
>>>
>>> Otherwise I am fully behind the suggestion of subsetting by common
>>> categorycombos - it does make sense.  Possibly we need to implement a
>>> getDataElementsByCategoryCombo() method on DataSet to make it easier to
>>> select appropriate dataelements.
>>>
>>> Regards
>>> Bob.
>>>
>>>
>>> 2009/12/10 Abyot Gizaw <abyota@xxxxxxxxx>
>>>
>>>>  Hi Bharath,
>>>>
>>>> What I wanted to say is - think of separating dataentry screens from
>>>> dataentry forms they shouldn't be the same. A dataentry screen contains
>>>> dataentryforms. And there shouldn't be any object called dataentryScreen but
>>>> only dataentryForm. Dataentry screen is more of a presentation and something
>>>> to be implemented using action classes at the web layer.
>>>>
>>>> When you have a dataset having dataelements with different category
>>>> combos then difficult to display them using a single table. So what I am
>>>> suggesting is to group these dataelements (those having the same catcombo)
>>>> and create them dataentryform.... like this a dataset can have have multiple
>>>> dataentry forms. DataEntry form is to have a group of dataelements all
>>>> sharing same categorycombo. Finally the dataset is free to have a set of
>>>> dataforms, but only one dataentryscreen. I have attached you a sample form
>>>> ... we need to handle such a requirement.
>>>>
>>>> Thank you
>>>> Abyot.
>>>>
>>>> On Thu, Dec 10, 2009 at 6:11 AM, bharath kumar <chbharathk@xxxxxxxxx>wrote:
>>>>
>>>>>
>>>>> Hi Abyot,
>>>>>
>>>>> Sorry I didn't understand clearly.
>>>>>
>>>>> You are saying one dataset say dataset1 can have multiple dataentry
>>>>> screens like dataentryscreen1,dataentryscreen2,dataentryscreen3.
>>>>>
>>>>> When we select dataset1 in the dataentry module we should display these
>>>>> 3 screens for dataentry?
>>>>>
>>>>> These 3 screens can have same dataelements or different dataelements?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 9, 2009 at 9:53 PM, Abyot Gizaw <abyota@xxxxxxxxx> wrote:
>>>>>
>>>>>> and also assigning multiple forms for datasets or programstages. For
>>>>>> example a dataset might have dataelements having different
>>>>>> categorycombos.... for such cases it makes sense to group those datalements
>>>>>> having the same categorycombo together then create a form.... in the end
>>>>>> such datasets will have a set of forms. So when displaying the dataentry
>>>>>> screen we can display a list of forms and come up a nice looking dataentry
>>>>>> screen.
>>>>>>
>>>>>> abyot.
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 9, 2009 at 5:59 AM, bharath kumar <chbharathk@xxxxxxxxx>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>> Regarding DataEntryScreen, as we (Abyot & Me) discussed on his last
>>>>>>> visit we agreed to make CustomDataEntryScreen as Genric.
>>>>>>>
>>>>>>> For that what we started is:
>>>>>>>
>>>>>>> We changed DataEntryForm object to contain its id, name and html code
>>>>>>> only. (before it was bound with datasetid).
>>>>>>> All the CustomScreen associations we kept in another table *
>>>>>>> dataentryformassociation* which contains associationtablename(
>>>>>>> represents which object DataSet/ProgramStage etc), associationid (
>>>>>>> represents datasetid/programstage id), dataentryformid.
>>>>>>>
>>>>>>> *From the GUI:*
>>>>>>> If you goto dataset screen there will be icon to design custom screen
>>>>>>> for dataset like before. only change is before it was saving in
>>>>>>> dataentryform table. now it will save the design in dataentryform table and
>>>>>>> association will be saved in dataentryformassociation table.
>>>>>>>
>>>>>>> Similarly for ProgramState also we can design screens.
>>>>>>>
>>>>>>> Breifly that is what we are doing regarding CustomScreens. Is it OK
>>>>>>> or any changes?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/12/9 Lars Helge Øverland <larshelge@xxxxxxxxx>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> 2009/12/8 Viet Nguyen <phamquocviet@xxxxxxxxx>
>>>>>>>>
>>>>>>>> Hi Lars  and  Abyot,
>>>>>>>>>
>>>>>>>>> I am sorry, the message of that commit was missing some
>>>>>>>>> information.
>>>>>>>>>
>>>>>>>>> I had some problem while committing the source. It was locked, and
>>>>>>>>> I tried to use the break-lock command many times and finally it worked. That
>>>>>>>>> is why I did not write the full commit message.
>>>>>>>>>
>>>>>>>>> So, that commit include
>>>>>>>>>
>>>>>>>>>    - Custom DataEntryForm design form for ProgramStage ( this is
>>>>>>>>>    what the FCK editor is for ).
>>>>>>>>>    - I Added class DataEntryFormAssociation. And yes Abyot, when I
>>>>>>>>>    was creating that class, I also wanted to create another package for all the
>>>>>>>>>    dataentryform things. But that was like just a few days I touch the dhis
>>>>>>>>>    code and I did not want to change so much. I just tried to follow the old
>>>>>>>>>    work flow. Please do not worry, after read your email, I changed it
>>>>>>>>>    immediately (  really happy to do that :)  ) . You will see it in the next
>>>>>>>>>    commit.
>>>>>>>>>    -  Custom data entry screen for case ( patient ) .  This is not
>>>>>>>>>    completely done, I am still working on it.
>>>>>>>>>
>>>>>>>>> The reason for this commit is to merge my local changes with your
>>>>>>>>> trunk. Because I have been coding for about two weeks on this module with my
>>>>>>>>> local source code, not on any branch...so it really need to be merged as
>>>>>>>>> soon as possible.
>>>>>>>>>
>>>>>>>>> And actually , I have not been  in the
>>>>>>>>> dhis2-devs@xxxxxxxxxxxxxxxxxxx util Abyot send me and Bharath the
>>>>>>>>> email. I have just subscribed to the list yesterday. I did not know about
>>>>>>>>> that mailing list. When you added me to the core-dev team, I looked for
>>>>>>>>> mailing list of this team, but I could not.
>>>>>>>>>
>>>>>>>>> About Abyot 's question :
>>>>>>>>>
>>>>>>>>> The other thing, if you remember we even talked about having a set
>>>>>>>>> of forms for datasets/programstages. Because sometimes you might have
>>>>>>>>> multiple tables/sections. Do you think you can include these things?
>>>>>>>>>
>>>>>>>>> Sorry, I do not know about this... Please explain it again if you
>>>>>>>>> have time.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks Viet for your reply.. Don't misunderstand, we're happy to see
>>>>>>>> u contributing and I'm exited to see the new code/functionality. Its just
>>>>>>>> important that Abyot understands whats going on and gets the change to
>>>>>>>> review things as he is leading the patient system work... Now that you have
>>>>>>>> enlisted for the dev list that's will be easier:-) Tell me if you there is
>>>>>>>> other stuff u are wondering about.
>>>>>>>>
>>>>>>>> cheers Lars
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Regards,
>>>>>>> Bharath Kumar. Ch
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Regards,
>>>>> Bharath Kumar. Ch
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs>
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs>
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs>
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs<https://launchpad.net/%7Edhis2-devs>
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>