dhis2-users team mailing list archive

[Alex Tumwesigye <atumwesigye@xxxxxxxxx>]

Jason/Morten,

I agree that real time analytics would not be easily implemented. May be it
is time to separate Aggregate and Tracker analytics as a start but it also
depends on what the tracker is being used for e.g surveys may require no
immediate analytics but IDSR/outbreaks may require instant update of
analytics. We are looking at IDSR features and as I see, real time
analytics will be a requirement. The IDSR requires real time analytics
since people need to respond to outbreaks and handle outbreak responses and
management in real time so that you can intervene and stop the outbreak.

Here is what I propose (just a thought from my discussion with Calle)
  We set a configurable (checked) variable/attribute that indicates that if
this variable is changed, the analytics process is started for example
changing population data has a very big impact on indicators that depend on
population,entering lab results or requests or patient updates for IDSR
requires immediate analytics, etc. If we had this attribute/variable, then
we would use it to identify the corresponding meta data that might need to
be / may have changed and update (through temp tables) the only affected
analytics tables. This way we can control the load required to run
analytics in realtime since the system  will only be updating affected
changes.
Using the API to trigger analytics every after X minutes may be feasible
but not sustainable as we do not control how many threads may be running as
the API call through curl does not easily get feedback/update if the
previous analytics process has been completed for the new one to start
otherwise we can end up in a forever loop if the server resources are not
enough.

Alex



On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <morten@xxxxxxxxx>
wrote:

> 1 ns is fine... but i depends on what the ramifications are.. what if you
> start a new job every 1 ms? is that ok? what happens during table swap? is
> analytic blocked?
>
> Maybe this is already documented.. just curious :)
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering <
> jason.p.pickering@xxxxxxxxx> wrote:
>
>> Why not ever 1 nanosecond? Its always going to take some amount of time,
>> question is what is reasonable.
>>
>> One could argue that real-time analytics in an aggregate data system is
>> not needed. This is not real time data. Its not even close to it.
>> Look at Google Analytics. They provide reports once a day, and you do not
>> see a whole lot of people complaning. Yes, you can get some limited real
>> time information from this as well, but its limited. The data must be
>> processed first, and that takes computational time. Same with DHIS2.
>>
>> From my experience, people think they need "real time analytics" when
>> they really are just in a rush.  Data takes time to review and analayze and
>> whether its available now, 1 nanosecond from now, or 10 minutes from now,
>> makes no difference in the end, as the amount of time which is required to
>> digest that information is on a totally different time scale (hours days or
>> weeks). Once an hour is probably easily achievable depending on the scale
>> of the system however.
>>
>> As for the call to the API, just create a Bash script and call it as
>> frequently as you like with a cron task.
>>
>> This is a very simple one, but you should really check for things like
>> "Is analytics already running and should I trigger another run?"
>>
>> #!/bin/sh
>>
>> /usr/bin/curl
>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2"
>> -X POST -u admin:district >/dev/null 2>&1
>>
>>
>>
>>
>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>> wrote:
>>
>>> Well, it sounds like a bad solution for me ;) if every 10 min works
>>> fine.. why not every 5 min.. why not every 1 min..
>>>
>>> --
>>> Morten Olav Hansen
>>> Senior Engineer, DHIS 2
>>> University of Oslo
>>> http://www.dhis2.org
>>>
>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh <
>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>
>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins sounds
>>>> like a good place to start. but am kind of not sure how to implement this
>>>> with the API calls and am sure this is not possible through the user
>>>> interface. If you guys can point me in the right direction with an example
>>>> of some sort that will greatly helpful and highly appreciated.
>>>>
>>>> Thanks.
>>>>
>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering <
>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>
>>>>> No they are swapped now, so first they are built as temp tables, then
>>>>> swapped. So, this is the brief point in time in which things may not be
>>>>> available.
>>>>>
>>>>> Every 10 minutes might be OK, depending on your server, loading, and
>>>>> amount of data. It just requires some experimentation.
>>>>>
>>>>> Point is, "real time" analytics is not possible. Near-real time may
>>>>> be.
>>>>>
>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> But I thought all analytic tables was cleared out during
>>>>>> re-generation? is this not true?
>>>>>>
>>>>>> --
>>>>>> Morten Olav Hansen
>>>>>> Senior Engineer, DHIS 2
>>>>>> University of Oslo
>>>>>> http://www.dhis2.org
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knutst@xxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> Depends on the data. I trigger analytics every 10 min.
>>>>>>> Hi All,
>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box  realtime
>>>>>>> analytics and Dashboards. But after further inspection i realized that
>>>>>>> current data entry will not available for analysis or dashboards until the
>>>>>>> next day or by manually doing export tables.  This is a real bottle neck to
>>>>>>> the intended implementation am working on. I have looked at Scheduling, but
>>>>>>> the options for analytics table is not ideal in my case. *Is there
>>>>>>> way this can be reduced or removed to gain real-time dashboards and
>>>>>>> analytics?*
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Bayoh.
>>>>>>>
>>>>>>> --
>>>>>>> Ibrahim Rashid Bayoh
>>>>>>> Information Systems Coordinator,
>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>> Mobile: +232 88-765-638
>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>> http://ehealthafrica.org/
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jason P. Pickering
>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>> tel:+46764147049
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Ibrahim Rashid Bayoh
>>>> Information Systems Coordinator,
>>>> eHealth Africa(Sierra Leone)
>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>> Mobile: +232 88-765-638
>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>> http://ehealthafrica.org/
>>>>
>>>>
>>>
>>
>>
>> --
>> Jason P. Pickering
>> email: jason.p.pickering@xxxxxxxxx
>> tel:+46764147049
>>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET
Kampala
Uganda
+256 774149 775, + 256 759 800161
Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems -
DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant


"I don't want to be anything other than what I have been - one tree hill "