dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Lars,
This has never been an issue before. It has only recently ( I think in
2.11) been noticed. Perhaps it has just gone undetected until now.

I guess what is really puzzling is that it "seems" work fine for the new
PivotTables. Anything which depends on the datamart does not work, so the
logic does not seem to be consistent. What I mean my "seems", is that I
have not confirmed in the values which appear in the analytics actually sum
to what they "should", in this case that the weekly and monthly values
should be summed. There of course could be other situations (similar to the
"Aggregation levels") in which weekly data (higher frequency) and monthly
data (lower frequency) data should not be summed at all (which does not
seem to be accounted for).

I guess what I am missing really is the relationship between a dataset and
a data value, and perhaps  a data element. One could think of a data
warehousing situation when in fact there is not a dataset for a data
element, unless this is implicitly required (as is perhaps indicated with
the Data Integrity checks?) Perhaps it is never entered through the data
entry screen at all, but simply imported from an external system. One could
also think of situations where data elements are recorded at different
frequencies. This is exactly what is happening here in Zambia, where
certain data elements are reported weekly on some datasets, and monthly in
others. In this case, they are for different orgunits, whereby certainly
orgunits report weekly and others report monthly.

Regards,
Jason







On Sat, May 11, 2013 at 7:40 PM, Lars Helge Øverland <larshelge@xxxxxxxxx>wrote:

> Hi Jason,
>
> the issue here is that "dis-aggregation" of data elements is not allowed.
> So for a "monthly" data element, weekly or daily data is ignored (data with
> higher frequency).
>
> The frequency/period type of a data element is derived from the period
> type of the data sets which it is assigned to. When a data element is
> member of data sets with different period types, the behavior is undefined,
> ie. the "first" data set is just selected randomly. If there are no data
> sets, a yearly period type is assumed. We could of course change this
> behavior.
>
> regards,
>
> Lars
>
>
>
> On Sat, May 11, 2013 at 2:15 PM, Jason Pickering <
> jason.p.pickering@xxxxxxxxx> wrote:
>
>> OK, so let me get this straight. Lets say I have a data element which
>> belongs to a weekly and to a monthly dataset. Some orgunits report it
>> weekly, others monthly. This is what is happening here in Zambia.
>>
>> Which one wins? The weekly or the monthly?
>>
>> It would seem that we would need to aggregate all of the data which was
>> reported with a weekly period, to weekly, and to monthly, and all of the
>> data which was reported monthly to monthly?
>>
>> I will need to confirm whether or not this data element is part of
>> another dataset.
>>
>> What happens if the data element is not part of a data set at all?
>>
>> Regards,
>> Jason
>>
>>
>>
>> On Sat, May 11, 2013 at 9:17 AM, Ola Hodne Titlestad <olati@xxxxxxxxxx>wrote:
>>
>>> Just to clarify:
>>> You'll get many aggregated period types in the datamart of course, but
>>> only one period type per data element can be used when looking/searching
>>> for raw data values to put into aggregation calculations during datamart
>>> export.
>>>
>>> So your monthly data values might be ignored in the datamart export
>>> process if quarterly data values exist for the same data elements. And we
>>> use the dataset period type to know what data values to look for. At least
>>> that's how I've understood this.
>>>
>>> Lars, correct me if this is wrong.
>>>
>>> Ola
>>> ------
>>> On 11 May 2013 09:10, "Ola Hodne Titlestad" <olati@xxxxxxxxxx> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> Are these data elements (with missing monthly vslues in datamart) also
>>>> part of another dataset with e.g. quarterly collection frequency (period
>>>> type)?
>>>>
>>>> Only one period type per data element is supported in datamart and this
>>>> is derived from the dataset's period type. So if the data element is member
>>>> of two data sets with different period types you will only get one of these
>>>> period types in the datamart.
>>>>
>>>> Ola
>>>> ------
>>>> On 9 May 2013 18:02, "Jason Pickering" <jason.p.pickering@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Hi devs,
>>>>> I am having some issues with some data elements not appearing in the
>>>>> data visualizer (and other modules which depend on datamart). Using the
>>>>> latest 2.11 file.
>>>>>
>>>>> I get expected results from the analytics pivot tables, but nothing
>>>>> for the "Past twelve months" as a relative period in the DV.
>>>>>
>>>>> Looked at the aggregateddata value table where this should appear, and
>>>>> see a some periods, but no monthly values (this is how the data is
>>>>> collected, by month).
>>>>>
>>>>> SELECT startdate from period where periodid in (SELECT DISTINCT
>>>>> periodid from aggregateddatavalue where dataelementid = 80);
>>>>>  startdate  |  enddate
>>>>> ------------+------------
>>>>>  2012-01-01 | 2012-12-31
>>>>>  2012-01-01 | 2012-03-31
>>>>>  2012-01-01 | 2012-06-30
>>>>>  2012-04-01 | 2012-06-30
>>>>>  2012-07-01 | 2012-09-30
>>>>>  2012-07-01 | 2012-12-31
>>>>>  2012-10-01 | 2012-12-31
>>>>>  2013-01-01 | 2013-06-30
>>>>>  2013-01-01 | 2013-03-31
>>>>>  2013-01-01 | 2013-12-31
>>>>> (10 rows)
>>>>>
>>>>> Looking at the data values table, just to be sure.
>>>>>
>>>>> SELECT startdate,enddate from period where periodid in (SELECT
>>>>> DISTINCT periodid from datavalue where dataelementid = 80) ORDER BY
>>>>> startdate DESC;
>>>>>
>>>>> startdate  |  enddate
>>>>> ------------+------------
>>>>>  2013-03-01 | 2013-03-31
>>>>>  2013-02-01 | 2013-02-28
>>>>>  2013-01-01 | 2013-01-31
>>>>>  2012-12-01 | 2012-12-31
>>>>>  2012-11-01 | 2012-11-30
>>>>>  2012-10-01 | 2012-10-31
>>>>>  2012-09-01 | 2012-09-30
>>>>>  2012-08-01 | 2012-08-31
>>>>>  2012-07-01 | 2012-07-31
>>>>>  2012-06-01 | 2012-06-30
>>>>>  2012-05-01 | 2012-05-31
>>>>>  2012-04-01 | 2012-04-30
>>>>>  2012-03-01 | 2012-03-31
>>>>>  2012-02-01 | 2012-02-29
>>>>>  2012-01-01 | 2012-01-31
>>>>>  2011-12-01 | 2011-12-31
>>>>>  2011-11-01 | 2011-11-30
>>>>> :
>>>>> So, the raw data appears to be there.
>>>>>
>>>>> I see no obvious errors in the logs. I did check the "Monthly" option
>>>>> for the data mart.
>>>>>
>>>>> I find this really strange. Any ideas what may be going on? For other
>>>>> data elements collected in the same form, I do get monthly values.
>>>>>
>>>>> Regards,
>>>>> Jason
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>