dhis2-users team mailing list archive

[Alex Tumwesigye <atumwesigye@xxxxxxxxx>]

Dear Jason,

I agree to not complicating things but as we look at IDSR module as a
generic module with DHIS2. We may need to think twice about the whole
process unless we want to keep it as a separate piece managed outside the
main DHIS2.
Scripts can be written, I have no problem with that by the way but how many
(of those using DHIS2) can write or manage simple scripts? If we make this
complicated, we run a risk of people not using some of the modules.

Alex

On Fri, Apr 22, 2016 at 3:49 PM, Jason Pickering <
jason.p.pickering@xxxxxxxxx> wrote:

> Thing is, how long does it take for the event to actually get reported?
>
> If it shows up on a person's dashboard in ten minutes, but they are out
> having lunch, or it has taken 3 days for the event to be reported, what's
> the use? Its no longer real time. Not even close to it.
>
> I agree there are use cases where things needs to be sped up, and it can
> be very simply with a very small curl script. You can even check and see if
> analytics is running first just to be sure you do not trigger it again.
>
> But lets not over-complicate things, and think about how real-time the
> data is which is actually being aggregated.
>
> Regards,
> Jason
>
>
> On Fri, Apr 22, 2016 at 2:43 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
> wrote:
>
>> Hi Knut,
>>
>> We are not talking of hours here, dashboards needs to be updated in
>> almost real time and outbreaks needed to be detected as data is entered
>> (once thresholds are met). So 10 minutes as an option in the dropdown for
>> analytics is ok for me and I would consider it ok. Adding it would make the
>> system almost real time and this would cater for the IDSR requirement. A
>> lag of 10 minutes for an update of dashboard on screen is ok.
>> I wish, it could be added to the system than doing it using the API given
>> the reasons above.
>>
>> Alex
>>
>> On Fri, Apr 22, 2016 at 3:27 PM, Knut Staring <knutst@xxxxxxxxx> wrote:
>>
>>> Hi Alex - I agree that IDSR requires "immediate" response - but I we are
>>> usually talking hours, not minutes, right?
>>>
>>> Knut
>>>
>>> On Fri, Apr 22, 2016 at 2:23 PM, Alex Tumwesigye <atumwesigye@xxxxxxxxx>
>>> wrote:
>>>
>>>> Jason/Morten,
>>>>
>>>> I agree that real time analytics would not be easily implemented. May
>>>> be it is time to separate Aggregate and Tracker analytics as a start but it
>>>> also depends on what the tracker is being used for e.g surveys may require
>>>> no immediate analytics but IDSR/outbreaks may require instant update of
>>>> analytics. We are looking at IDSR features and as I see, real time
>>>> analytics will be a requirement. The IDSR requires real time analytics
>>>> since people need to respond to outbreaks and handle outbreak responses and
>>>> management in real time so that you can intervene and stop the outbreak.
>>>>
>>>> Here is what I propose (just a thought from my discussion with Calle)
>>>>   We set a configurable (checked) variable/attribute that indicates
>>>> that if this variable is changed, the analytics process is started for
>>>> example changing population data has a very big impact on indicators that
>>>> depend on population,entering lab results or requests or patient updates
>>>> for IDSR requires immediate analytics, etc. If we had this
>>>> attribute/variable, then we would use it to identify the corresponding meta
>>>> data that might need to be / may have changed and update (through temp
>>>> tables) the only affected analytics tables. This way we can control the
>>>> load required to run analytics in realtime since the system  will only be
>>>> updating affected changes.
>>>> Using the API to trigger analytics every after X minutes may be
>>>> feasible but not sustainable as we do not control how many threads may be
>>>> running as the API call through curl does not easily get feedback/update if
>>>> the previous analytics process has been completed for the new one to start
>>>> otherwise we can end up in a forever loop if the server resources are not
>>>> enough.
>>>>
>>>> Alex
>>>>
>>>>
>>>>
>>>> On Fri, Apr 22, 2016 at 2:50 PM, Morten Olav Hansen <morten@xxxxxxxxx>
>>>> wrote:
>>>>
>>>>> 1 ns is fine... but i depends on what the ramifications are.. what if
>>>>> you start a new job every 1 ms? is that ok? what happens during table swap?
>>>>> is analytic blocked?
>>>>>
>>>>> Maybe this is already documented.. just curious :)
>>>>>
>>>>> --
>>>>> Morten Olav Hansen
>>>>> Senior Engineer, DHIS 2
>>>>> University of Oslo
>>>>> http://www.dhis2.org
>>>>>
>>>>> On Fri, Apr 22, 2016 at 6:48 PM, Jason Pickering <
>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>
>>>>>> Why not ever 1 nanosecond? Its always going to take some amount of
>>>>>> time, question is what is reasonable.
>>>>>>
>>>>>> One could argue that real-time analytics in an aggregate data system
>>>>>> is not needed. This is not real time data. Its not even close to it.
>>>>>> Look at Google Analytics. They provide reports once a day, and you do
>>>>>> not see a whole lot of people complaning. Yes, you can get some limited
>>>>>> real time information from this as well, but its limited. The data must be
>>>>>> processed first, and that takes computational time. Same with DHIS2.
>>>>>>
>>>>>> From my experience, people think they need "real time analytics" when
>>>>>> they really are just in a rush.  Data takes time to review and analayze and
>>>>>> whether its available now, 1 nanosecond from now, or 10 minutes from now,
>>>>>> makes no difference in the end, as the amount of time which is required to
>>>>>> digest that information is on a totally different time scale (hours days or
>>>>>> weeks). Once an hour is probably easily achievable depending on the scale
>>>>>> of the system however.
>>>>>>
>>>>>> As for the call to the API, just create a Bash script and call it as
>>>>>> frequently as you like with a cron task.
>>>>>>
>>>>>> This is a very simple one, but you should really check for things
>>>>>> like "Is analytics already running and should I trigger another run?"
>>>>>>
>>>>>> #!/bin/sh
>>>>>>
>>>>>> /usr/bin/curl
>>>>>> "localhost:8080/api/resourceTables/analytics?skipResourceTables=true&lastYears=2"
>>>>>> -X POST -u admin:district >/dev/null 2>&1
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 22, 2016 at 1:34 PM, Morten Olav Hansen <morten@xxxxxxxxx
>>>>>> > wrote:
>>>>>>
>>>>>>> Well, it sounds like a bad solution for me ;) if every 10 min works
>>>>>>> fine.. why not every 5 min.. why not every 1 min..
>>>>>>>
>>>>>>> --
>>>>>>> Morten Olav Hansen
>>>>>>> Senior Engineer, DHIS 2
>>>>>>> University of Oslo
>>>>>>> http://www.dhis2.org
>>>>>>>
>>>>>>> On Fri, Apr 22, 2016 at 6:31 PM, Ibrahim Bayoh <
>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> @Jason,Knut and Morten, Having analytics table run every 10 mins
>>>>>>>> sounds like a good place to start. but am kind of not sure how to implement
>>>>>>>> this with the API calls and am sure this is not possible through the user
>>>>>>>> interface. If you guys can point me in the right direction with an example
>>>>>>>> of some sort that will greatly helpful and highly appreciated.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> On Fri, Apr 22, 2016 at 11:23 AM, Jason Pickering <
>>>>>>>> jason.p.pickering@xxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> No they are swapped now, so first they are built as temp tables,
>>>>>>>>> then swapped. So, this is the brief point in time in which things may not
>>>>>>>>> be available.
>>>>>>>>>
>>>>>>>>> Every 10 minutes might be OK, depending on your server, loading,
>>>>>>>>> and amount of data. It just requires some experimentation.
>>>>>>>>>
>>>>>>>>> Point is, "real time" analytics is not possible. Near-real time
>>>>>>>>> may be.
>>>>>>>>>
>>>>>>>>> On Fri, Apr 22, 2016 at 1:03 PM, Morten Olav Hansen <
>>>>>>>>> morten@xxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> But I thought all analytic tables was cleared out during
>>>>>>>>>> re-generation? is this not true?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Morten Olav Hansen
>>>>>>>>>> Senior Engineer, DHIS 2
>>>>>>>>>> University of Oslo
>>>>>>>>>> http://www.dhis2.org
>>>>>>>>>>
>>>>>>>>>> On Fri, Apr 22, 2016 at 6:00 PM, Knut Staring <knutst@xxxxxxxxx>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Depends on the data. I trigger analytics every 10 min.
>>>>>>>>>>> Hi All,
>>>>>>>>>>> Initially i thought DHIS2 had fully or out-of-the-box  realtime
>>>>>>>>>>> analytics and Dashboards. But after further inspection i realized that
>>>>>>>>>>> current data entry will not available for analysis or dashboards until the
>>>>>>>>>>> next day or by manually doing export tables.  This is a real bottle neck to
>>>>>>>>>>> the intended implementation am working on. I have looked at Scheduling, but
>>>>>>>>>>> the options for analytics table is not ideal in my case. *Is
>>>>>>>>>>> there way this can be reduced or removed to gain real-time dashboards and
>>>>>>>>>>> analytics?*
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Bayoh.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Ibrahim Rashid Bayoh
>>>>>>>>>>> Information Systems Coordinator,
>>>>>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>>>>>> Mobile: +232 88-765-638
>>>>>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> http://ehealthafrica.org/
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jason P. Pickering
>>>>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>>>>> tel:+46764147049
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Ibrahim Rashid Bayoh
>>>>>>>> Information Systems Coordinator,
>>>>>>>> eHealth Africa(Sierra Leone)
>>>>>>>> *117 Wilkinson Rd, Freetown, Sierra Leone*
>>>>>>>> Mobile: +232 88-765-638
>>>>>>>> ibrahim.bayoh@xxxxxxxxxxxxxxxxxxxx
>>>>>>>> http://ehealthafrica.org/
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jason P. Pickering
>>>>>> email: jason.p.pickering@xxxxxxxxx
>>>>>> tel:+46764147049
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Alex Tumwesigye
>>>>
>>>> Technical Advisor - DHIS2 (Consultant),
>>>> Ministry of Health/AFENET
>>>> Kampala
>>>> Uganda
>>>> +256 774149 775, + 256 759 800161
>>>> Skype ID: talexie
>>>>
>>>> IT Consultant (Servers, Networks and Security, Health Information
>>>> Systems - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>>>>
>>>>
>>>> "I don't want to be anything other than what I have been - one tree
>>>> hill "
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>> --
>>> Knut Staring
>>> Dept. of Informatics, University of Oslo
>>> Norway: +4791880522
>>> Skype: knutstar
>>> http://dhis2.org
>>>
>>
>>
>>
>> --
>> Alex Tumwesigye
>>
>> Technical Advisor - DHIS2 (Consultant),
>> Ministry of Health/AFENET
>> Kampala
>> Uganda
>> +256 774149 775, + 256 759 800161
>> Skype ID: talexie
>>
>> IT Consultant (Servers, Networks and Security, Health Information Systems
>> - DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant
>>
>>
>> "I don't want to be anything other than what I have been - one tree hill "
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Jason P. Pickering
> email: jason.p.pickering@xxxxxxxxx
> tel:+46764147049
>



-- 
Alex Tumwesigye

Technical Advisor - DHIS2 (Consultant),
Ministry of Health/AFENET
Kampala
Uganda
+256 774149 775, + 256 759 800161
Skype ID: talexie

IT Consultant (Servers, Networks and Security, Health Information Systems -
DHIS2, Disease Outbreak & Surveillance Systems) & Solar Consultant


"I don't want to be anything other than what I have been - one tree hill "