dhis2-users team mailing list archive

[Alex Tumwesigye <atumwesigye@xxxxxxxxx>]

Hi Knut,

We are not talking of hours here, dashboards needs to be updated in almost
real time and outbreaks needed to be detected as data is entered (once
thresholds are met). So 10 minutes as an option in the dropdown for
analytics is ok for me and I would consider it ok. Adding it would make the
system almost real time and this would cater for the IDSR requirement. A
lag of 10 minutes for an update of dashboard on screen is ok.
I wish, it could be added to the system than doing it using the API given
the reasons above.

Alex

On Fri, Apr 22, 2016 at 3:27 PM, Knut Staring <knutst@xxxxxxxxx> wrote:

> Hi Alex - I agree that IDSR requires "immediate" response - but I we are
> usually talking hours, not minutes, right?
>
> Knut
>
> On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
> wrote:
>
>> Jason/Morten,
>>
>> I agree that real time analytics would not be easily implemented. May be
>> it is time to separate Aggregate and Tracker analytics as a start but it
>> also depends on what the tracker is being used for e.g surveys may require
>> no immediate analytics but IDSR/outbreaks may require instant update of
>> analytics. We are looking at IDSR features and as I see, real time
>> analytics will be a requirement. The IDSR requires real time analytics
>> since people need to respond to outbreaks and handle outbreak responses and
>> management in real time so that you can intervene and stop the outbreak.
>>
>> Here is what I propose (just a thought from my discussion with Calle)
>>   We set a configurable (checked) variable/attribute that indicates that
>> if this variable is changed, the analytics process is started for example
>> changing population data has a very big impact on indicators that depend on
>> population,entering lab results or requests or patient updates for IDSR
>> requires immediate analytics, etc. If we had this attribute/variable, then
>> we would use it to identify the corresponding meta data that might need to
>> be / may have changed and update (through temp tables) the only affected
>> analytics tables. This way we can control the load required to run
>> analytics in realtime since the system  will only be updating affected
>> changes.
>> Using the API to trigger analytics every after X minutes may be feasible
>> but not sustainable as we do not control how many threads may be running as
>> the API call through curl does not easily get feedback/update if the
>> previous analytics process has been completed for the new one to start
>> otherwise we can end up in a forever loop if the server resources are not
>> enough.
>>
>> Alex
>>
>>
>>
>> On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> 1 ns is fine... but i depends on what the ramifications are.. what if
>>> you start a new job every 1 ms? is that ok? what happens during table swap?
>>> is analytic blocked?
>>>
>>> Maybe this is already documented.. just curious :)
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering <
>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>
>>>> Why not ever 1 nanosecond? Its always going to take some amount of
>>>> time, question is what is reasonable.
>>>>
>>>> One could argue that real-time analytics in an aggregate data system is
>>>> not needed. This is not real time data. Its not even close to it.
>>>> Look at Google Analytics. They provide reports once a day, and you do
>>>> not see a whole lot of people complaning. Yes, you can get some limited
>>>> real time information from this as well, but its limited. The data must be
>>>> processed first, and that takes computational time. Same with DHIS2.
>>>>
>>>> From my experience, people think they need "real time analytics" when
>>>> they really are just in a rush.  Data takes time to review and analayze and
>>>> whether its available now, 1 nanosecond from now, or 10 minutes from now,
>>>> makes no difference in the end, as the amount of time which is required to
>>>> digest that information is on a totally different time scale (hours days or
>>>> weeks). Once an hour is probably easily achievable depending on the scale
>>>> of the system however.
>>>>
>>>> As for the call to the API, just create a Bash script and call it as
>>>> frequently as you like with a cron task.
>>>>
>>>> This is a very simple one, but you should really check for things like
>>>> "Is analytics already running and should I trigger another run?"
>>>>
>>>> #!/bin/sh
>>>>
>>>> /usr/bin/curl
>>>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2"
>>>> -X POST -u admin:district >/dev/null 2>&1
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> Well, it sounds like a bad solution for me ;) if every 10 min works
>>>>> fine.. why not every 5 min.. why not every 1 min..
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh <
>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>
>>>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins
>>>>>> sounds like a good place to start. but am kind of not sure how to implement
>>>>>> this with the API calls and am sure this is not possible through the user
>>>>>> interface. If you guys can point me in the right direction with an example
>>>>>> of some sort that will greatly helpful and highly appreciated.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering <
>>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>>
>>>>>>> No they are swapped now, so first they are built as temp tables,
>>>>>>> then swapped. So, this is the brief point in time in which things may not
>>>>>>> be available.
>>>>>>>
>>>>>>> Every 10 minutes might be OK, depending on your server, loading, and
>>>>>>> amount of data. It just requires some experimentation.
>>>>>>>
>>>>>>> Point is, "real time" analytics is not possible. Near-real time may
>>>>>>> be.
>>>>>>>
>>>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <
>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> But I thought all analytic tables was cleared out during
>>>>>>>> re-generation? is this not true?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Morten Olav Hansen
>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>> University of Oslo
>>>>>>>> http://www.dhis2.org
>>>>>>>>
>>>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knutst@xxxxxxxxx>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Depends on the data. I trigger analytics every 10 min.
>>>>>>>>> Hi All,
>>>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box  realtime
>>>>>>>>> analytics and Dashboards. But after further inspection i realized that
>>>>>>>>> current data entry will not available for analysis or dashboards until the
>>>>>>>>> next day or by manually doing export tables.  This is a real bottle neck to
>>>>>>>>> the intended implementation am working on. I have looked at Scheduling, but
>>>>>>>>> the options for analytics table is not ideal in my case. *Is
>>>>>>>>> there way this can be reduced or removed to gain real-time dashboards and
>>>>>>>>> analytics?*
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Bayoh.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Ibrahim Rashid Bayoh
>>>>>>>>> Information Systems Coordinator,
>>>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>>>> Mobile: +232 88-765-638
>>>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>>>> http://ehealthafrica.org/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jason P. Pickering
>>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>>> tel:+46764147049
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ibrahim Rashid Bayoh
>>>>>> Information Systems Coordinator,
>>>>>> eHealth Africa(Sierra Leone)
>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>> Mobile: +232 88-765-638
>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>> http://ehealthafrica.org/
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Jason P. Pickering
>>>> email: jason.p.pickering@xxxxxxxxx
>>>> tel:+46764147049
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-users
>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Alex Tumwesigye
>>
>> Technical Advisor - DHIS2 (Consultant),
>> Ministry of Health/AFENET
>> Kampala
>> Uganda
>> +256 774149 775, + 256 759 800161
>> Skype ID: talexie
>>
>> IT Consultant (Servers, Networks and Security, Health Information Systems
>> - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>>
>>
>> "I don't want to be anything other than what I have been - one tree hill "
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Knut Staring
> Dept. of Informatics, University of Oslo
> Norway: +4791880522
> Skype: knutstar
> http://dhis2.org
>



-- 
Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET
Kampala
Uganda
+256 774149 775, + 256 759 800161
Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems -
DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant


"I don't want to be anything other than what I have been - one tree hill "