dhis2-users team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Thing is, how long does it take for the event to actually get reported?

If it shows up on a person's dashboard in ten minutes, but they are out
having lunch, or it has taken 3 days for the event to be reported, what's
the use? Its no longer real time. Not even close to it.

I agree there are use cases where things needs to be sped up, and it can be
very simply with a very small curl script. You can even check and see if
analytics is running first just to be sure you do not trigger it again.

But lets not over-complicate things, and think about how real-time the data
is which is actually being aggregated.

Regards,
Jason


On Fri, Apr 22, 2016 at 2:43 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
wrote:

> Hi Knut,
>
> We are not talking of hours here, dashboards needs to be updated in almost
> real time and outbreaks needed to be detected as data is entered (once
> thresholds are met). So 10 minutes as an option in the dropdown for
> analytics is ok for me and I would consider it ok. Adding it would make the
> system almost real time and this would cater for the IDSR requirement. A
> lag of 10 minutes for an update of dashboard on screen is ok.
> I wish, it could be added to the system than doing it using the API given
> the reasons above.
>
> Alex
>
> On Fri, Apr 22, 2016 at 3:27 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
>
>> Hi Alex - I agree that IDSR requires "immediate" response - but I we are
>> usually talking hours, not minutes, right?
>>
>> Knut
>>
>> On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
>> wrote:
>>
>>> Jason/Morten,
>>>
>>> I agree that real time analytics would not be easily implemented. May be
>>> it is time to separate Aggregate and Tracker analytics as a start but it
>>> also depends on what the tracker is being used for e.g surveys may require
>>> no immediate analytics but IDSR/outbreaks may require instant update of
>>> analytics. We are looking at IDSR features and as I see, real time
>>> analytics will be a requirement. The IDSR requires real time analytics
>>> since people need to respond to outbreaks and handle outbreak responses and
>>> management in real time so that you can intervene and stop the outbreak.
>>>
>>> Here is what I propose (just a thought from my discussion with Calle)
>>>   We set a configurable (checked) variable/attribute that indicates that
>>> if this variable is changed, the analytics process is started for example
>>> changing population data has a very big impact on indicators that depend on
>>> population,entering lab results or requests or patient updates for IDSR
>>> requires immediate analytics, etc. If we had this attribute/variable, then
>>> we would use it to identify the corresponding meta data that might need to
>>> be / may have changed and update (through temp tables) the only affected
>>> analytics tables. This way we can control the load required to run
>>> analytics in realtime since the system  will only be updating affected
>>> changes.
>>> Using the API to trigger analytics every after X minutes may be feasible
>>> but not sustainable as we do not control how many threads may be running as
>>> the API call through curl does not easily get feedback/update if the
>>> previous analytics process has been completed for the new one to start
>>> otherwise we can end up in a forever loop if the server resources are not
>>> enough.
>>>
>>> Alex
>>>
>>>
>>>
>>> On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>> wrote:
>>>
>>>> 1 ns is fine... but i depends on what the ramifications are.. what if
>>>> you start a new job every 1 ms? is that ok? what happens during table swap?
>>>> is analytic blocked?
>>>>
>>>> Maybe this is already documented.. just curious :)
>>>>
>>>> --
>>>> Morten Olav Hansen
>>>> Senior Engineer, DHIS 2
>>>> University of Oslo
>>>> http://www.dhis2.org
>>>>
>>>> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering <
>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>
>>>>> Why not ever 1 nanosecond? Its always going to take some amount of
>>>>> time, question is what is reasonable.
>>>>>
>>>>> One could argue that real-time analytics in an aggregate data system
>>>>> is not needed. This is not real time data. Its not even close to it.
>>>>> Look at Google Analytics. They provide reports once a day, and you do
>>>>> not see a whole lot of people complaning. Yes, you can get some limited
>>>>> real time information from this as well, but its limited. The data must be
>>>>> processed first, and that takes computational time. Same with DHIS2.
>>>>>
>>>>> From my experience, people think they need "real time analytics" when
>>>>> they really are just in a rush.  Data takes time to review and analayze and
>>>>> whether its available now, 1 nanosecond from now, or 10 minutes from now,
>>>>> makes no difference in the end, as the amount of time which is required to
>>>>> digest that information is on a totally different time scale (hours days or
>>>>> weeks). Once an hour is probably easily achievable depending on the scale
>>>>> of the system however.
>>>>>
>>>>> As for the call to the API, just create a Bash script and call it as
>>>>> frequently as you like with a cron task.
>>>>>
>>>>> This is a very simple one, but you should really check for things like
>>>>> "Is analytics already running and should I trigger another run?"
>>>>>
>>>>> #!/bin/sh
>>>>>
>>>>> /usr/bin/curl
>>>>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2"
>>>>> -X POST -u admin:district >/dev/null 2>&1
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Well, it sounds like a bad solution for me ;) if every 10 min works
>>>>>> fine.. why not every 5 min.. why not every 1 min..
>>>>>>
>>>>>> --
>>>>>> Morten Olav Hansen
>>>>>> Senior Engineer, DHIS 2
>>>>>> University of Oslo
>>>>>> http://www.dhis2.org
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh <
>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>>
>>>>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins
>>>>>>> sounds like a good place to start. but am kind of not sure how to implement
>>>>>>> this with the API calls and am sure this is not possible through the user
>>>>>>> interface. If you guys can point me in the right direction with an example
>>>>>>> of some sort that will greatly helpful and highly appreciated.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering <
>>>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> No they are swapped now, so first they are built as temp tables,
>>>>>>>> then swapped. So, this is the brief point in time in which things may not
>>>>>>>> be available.
>>>>>>>>
>>>>>>>> Every 10 minutes might be OK, depending on your server, loading,
>>>>>>>> and amount of data. It just requires some experimentation.
>>>>>>>>
>>>>>>>> Point is, "real time" analytics is not possible. Near-real time may
>>>>>>>> be.
>>>>>>>>
>>>>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <
>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> But I thought all analytic tables was cleared out during
>>>>>>>>> re-generation? is this not true?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Morten Olav Hansen
>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>> University of Oslo
>>>>>>>>> http://www.dhis2.org
>>>>>>>>>
>>>>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knutst@xxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Depends on the data. I trigger analytics every 10 min.
>>>>>>>>>> Hi All,
>>>>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box  realtime
>>>>>>>>>> analytics and Dashboards. But after further inspection i realized that
>>>>>>>>>> current data entry will not available for analysis or dashboards until the
>>>>>>>>>> next day or by manually doing export tables.  This is a real bottle neck to
>>>>>>>>>> the intended implementation am working on. I have looked at Scheduling, but
>>>>>>>>>> the options for analytics table is not ideal in my case. *Is
>>>>>>>>>> there way this can be reduced or removed to gain real-time dashboards and
>>>>>>>>>> analytics?*
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Bayoh.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Ibrahim Rashid Bayoh
>>>>>>>>>> Information Systems Coordinator,
>>>>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>>>>> Mobile: +232 88-765-638
>>>>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>>>>> http://ehealthafrica.org/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jason P. Pickering
>>>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>>>> tel:+46764147049
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ibrahim Rashid Bayoh
>>>>>>> Information Systems Coordinator,
>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>> Mobile: +232 88-765-638
>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>> http://ehealthafrica.org/
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jason P. Pickering
>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>> tel:+46764147049
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>> --
>>> Alex Tumwesigye
>>>
>>> Technical Advisor - DHIS2 (Consultant),
>>> Ministry of Health/AFENET
>>> Kampala
>>> Uganda
>>> +256 774149 775, + 256 759 800161
>>> Skype ID: talexie
>>>
>>> IT Consultant (Servers, Networks and Security, Health Information
>>> Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>>>
>>>
>>> "I don't want to be anything other than what I have been - one tree hill
>>> "
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-users
>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Knut Staring
>> Dept. of Informatics, University of Oslo
>> Norway: +4791880522
>> Skype: knutstar
>> http://dhis2.org
>>
>
>
>
> --
> Alex Tumwesigye
>
> Technical Advisor - DHIS2 (Consultant),
> Ministry of Health/AFENET
> Kampala
> Uganda
> +256 774149 775, + 256 759 800161
> Skype ID: talexie
>
> IT Consultant (Servers, Networks and Security, Health Information Systems
> - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>
>
> "I don't want to be anything other than what I have been - one tree hill "
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+46764147049