dhis2-users team mailing list archive

[Knut Staring <knutst@xxxxxxxxx>]

Hi Alex - I agree that IDSR requires "immediate" response - but I we are
usually talking hours, not minutes, right?

Knut

On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
wrote:

> Jason/Morten,
>
> I agree that real time analytics would not be easily implemented. May be
> it is time to separate Aggregate and Tracker analytics as a start but it
> also depends on what the tracker is being used for e.g surveys may require
> no immediate analytics but IDSR/outbreaks may require instant update of
> analytics. We are looking at IDSR features and as I see, real time
> analytics will be a requirement. The IDSR requires real time analytics
> since people need to respond to outbreaks and handle outbreak responses and
> management in real time so that you can intervene and stop the outbreak.
>
> Here is what I propose (just a thought from my discussion with Calle)
>   We set a configurable (checked) variable/attribute that indicates that
> if this variable is changed, the analytics process is started for example
> changing population data has a very big impact on indicators that depend on
> population,entering lab results or requests or patient updates for IDSR
> requires immediate analytics, etc. If we had this attribute/variable, then
> we would use it to identify the corresponding meta data that might need to
> be / may have changed and update (through temp tables) the only affected
> analytics tables. This way we can control the load required to run
> analytics in realtime since the system  will only be updating affected
> changes.
> Using the API to trigger analytics every after X minutes may be feasible
> but not sustainable as we do not control how many threads may be running as
> the API call through curl does not easily get feedback/update if the
> previous analytics process has been completed for the new one to start
> otherwise we can end up in a forever loop if the server resources are not
> enough.
>
> Alex
>
>
>
> On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <morten@xxxxxxxxx>
> wrote:
>
>> 1 ns is fine... but i depends on what the ramifications are.. what if you
>> start a new job every 1 ms? is that ok? what happens during table swap? is
>> analytic blocked?
>>
>> Maybe this is already documented.. just curious :)
>>
>> --
>> Morten Olav Hansen
>> Senior Engineer, DHIS 2
>> University of Oslo
>> http://www.dhis2.org
>>
>> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering <
>> jason.p.pickering@xxxxxxxxx> wrote:
>>
>>> Why not ever 1 nanosecond? Its always going to take some amount of time,
>>> question is what is reasonable.
>>>
>>> One could argue that real-time analytics in an aggregate data system is
>>> not needed. This is not real time data. Its not even close to it.
>>> Look at Google Analytics. They provide reports once a day, and you do
>>> not see a whole lot of people complaning. Yes, you can get some limited
>>> real time information from this as well, but its limited. The data must be
>>> processed first, and that takes computational time. Same with DHIS2.
>>>
>>> From my experience, people think they need "real time analytics" when
>>> they really are just in a rush.  Data takes time to review and analayze and
>>> whether its available now, 1 nanosecond from now, or 10 minutes from now,
>>> makes no difference in the end, as the amount of time which is required to
>>> digest that information is on a totally different time scale (hours days or
>>> weeks). Once an hour is probably easily achievable depending on the scale
>>> of the system however.
>>>
>>> As for the call to the API, just create a Bash script and call it as
>>> frequently as you like with a cron task.
>>>
>>> This is a very simple one, but you should really check for things like
>>> "Is analytics already running and should I trigger another run?"
>>>
>>> #!/bin/sh
>>>
>>> /usr/bin/curl
>>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2"
>>> -X POST -u admin:district >/dev/null 2>&1
>>>
>>>
>>>
>>>
>>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>> wrote:
>>>
>>>> Well, it sounds like a bad solution for me ;) if every 10 min works
>>>> fine.. why not every 5 min.. why not every 1 min..
>>>>
>>>> --
>>>> Morten Olav Hansen
>>>> Senior Engineer, DHIS 2
>>>> University of Oslo
>>>> http://www.dhis2.org
>>>>
>>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh <
>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins
>>>>> sounds like a good place to start. but am kind of not sure how to implement
>>>>> this with the API calls and am sure this is not possible through the user
>>>>> interface. If you guys can point me in the right direction with an example
>>>>> of some sort that will greatly helpful and highly appreciated.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering <
>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>
>>>>>> No they are swapped now, so first they are built as temp tables, then
>>>>>> swapped. So, this is the brief point in time in which things may not be
>>>>>> available.
>>>>>>
>>>>>> Every 10 minutes might be OK, depending on your server, loading, and
>>>>>> amount of data. It just requires some experimentation.
>>>>>>
>>>>>> Point is, "real time" analytics is not possible. Near-real time may
>>>>>> be.
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <morten@xxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>> But I thought all analytic tables was cleared out during
>>>>>>> re-generation? is this not true?
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knutst@xxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Depends on the data. I trigger analytics every 10 min.
>>>>>>>> Hi All,
>>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box  realtime
>>>>>>>> analytics and Dashboards. But after further inspection i realized that
>>>>>>>> current data entry will not available for analysis or dashboards until the
>>>>>>>> next day or by manually doing export tables.  This is a real bottle neck to
>>>>>>>> the intended implementation am working on. I have looked at Scheduling, but
>>>>>>>> the options for analytics table is not ideal in my case. *Is there
>>>>>>>> way this can be reduced or removed to gain real-time dashboards and
>>>>>>>> analytics?*
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bayoh.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ibrahim Rashid Bayoh
>>>>>>>> Information Systems Coordinator,
>>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>>> Mobile: +232 88-765-638
>>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>>> http://ehealthafrica.org/
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason P. Pickering
>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>> tel:+46764147049
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ibrahim Rashid Bayoh
>>>>> Information Systems Coordinator,
>>>>> eHealth Africa(Sierra Leone)
>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>> Mobile: +232 88-765-638
>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>> http://ehealthafrica.org/
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Jason P. Pickering
>>> email: jason.p.pickering@xxxxxxxxx
>>> tel:+46764147049
>>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Alex Tumwesigye
>
> Technical Advisor - DHIS2 (Consultant),
> Ministry of Health/AFENET
> Kampala
> Uganda
> +256 774149 775, + 256 759 800161
> Skype ID: talexie
>
> IT Consultant (Servers, Networks and Security, Health Information Systems
> - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>
>
> "I don't want to be anything other than what I have been - one tree hill "
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Knut Staring
Dept. of Informatics, University of Oslo
Norway: +4791880522
Skype: knutstar
http://dhis2.org