dhis2-users team mailing list archive

["Friedman, Roger (CDC/OID/NCHHSTP) (CTR)" <rdf4@xxxxxxx>]

My concern would be DB performance, there's no telling what kind of locks R or any other product using odbc/jdbc is going to use.  I'm already worried about simultaneous transactional and reporting use.  Have there been any large-volume performance tests?  Has any thought been given to splitting reporting and data entry between different DB servers?  I know everyone has been focused on getting the distributed DB aspects right, but assuming universal availability of internet, how would DHIS2 perform on a single (possibly clustered) national DB server?

-----Original Message-----
From: dhis2-users-bounces+rdf4=cdc.gov@xxxxxxxxxxxxxxxxxxx [mailto:dhis2-users-bounces+rdf4=cdc.gov@xxxxxxxxxxxxxxxxxxx] On Behalf Of Jason Pickering
Sent: Thursday, May 27, 2010 7:03 AM
To: Bob Jolliffe
Cc: dhis2-users@xxxxxxxxxxxxxxxxxxx; dhis2-devs
Subject: Re: [Dhis2-users] [Dhis2-devs] DHIS2 with R

Yeah, this I guess comes back time and time again, with my some what
uncomfortable relationship with Hibernate and Java. Clearly, we need
to think about how to make certain procedures crossplatform compatible
(cross platform in the sense of working between Postgres/MySQL and
other DBs) with the need to offer advanced analysis capabilities, with
acceptable performance.

There could be multiple ways of doing it, but in the absense of having
R integrated into DHIS2, I think the most likely shorterm use case
would be just some documentation on how to use the R client with the
DHIS2 database. Perhaps those users that use R over time with DHIS2
could contribute their procedures, which should be able to be
generalized either with PL/R.

Of course the difference with using Postgres, is that R procedures can
be embedded as a new language inside the DB. I am not really sure this
is possible with MySQL. This of course reduces the internal overhead
of getting the data out of Postgres, through Java, and into the R
interpreter, but I am not sure really what the impact of this might be
without testing it.



On Thu, May 27, 2010 at 12:26 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
> On 27 May 2010 11:15, Jason Pickering <jason.p.pickering@xxxxxxxxx> wrote:
>> Hi Bob,
>>
>> Yes, I suspect that most R users would probably want to do things
>> their own way. It has a rather steep learning curve. :)
>>
>> As for canned R scripts, the best way would probably with with PL/R, a
>> procedural Postgresql language which utilizes R.
>>
>> http://www.joeconway.com/plr/doc/index.html
>>
>> I have done some very basic testing and it seems to work just fine on
>> the server side.
>
> Swings and roundabouts to a certain extent.  The main thing is that
> the r scripts are evaluated using the r c library.  If they were
> invoked from within java/dhis then I guess data access would be slower
> than from pl/r (we'd need to have a way to get the data to the r
> interpreter), but number crunching would be similar and would also
> work with mysql and friends.  Not sure which of these are bigger
> problems in typical/possible scenarios.
>
>>
>> I think they are two separate problems really, but I totally agree, C
>> is likely going to be faster than Java for big operations. However, I
>> do think (as all of you know) that the use of stored procedures (with
>> the wrapper facade type of approach) for certain functions (like
>> aggregation and heavy cross tab operations) would be much better to be
>> executed on the database server as a native stored procedure.
>>
>> Regards,
>> Jason
>>
>>
>>
>>
>> On Thu, May 27, 2010 at 11:45 AM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>>> We've talked before about integrating scripting engine (such as R)
>>> into dhis : http://www.rforge.net/rscript/
>>>
>>> But my guess is that most R users are going to be of a level of
>>> sophistication that they would be most comfortable doing the kind of
>>> thing you describe - conecting directly to db with r client and doing
>>> their stuff.
>>>
>>> OTOH if there were sufficiently useful "canned" dhis R scripts which
>>> could take some number crunching load off the jvm and produce canned
>>> useful analysis then that would be different.
>>>
>>> Sadly I don't know sufficient about R to know.  But I sense it ...
>>>
>>> Regards
>>> Bob
>>>
>>> On 27 May 2010 10:08, Jason Pickering <jason.p.pickering@xxxxxxxxx> wrote:
>>>> Hi everyone. I have had a recent question from a user about how DHIS2
>>>> can be used with R. I am including a trivial example here about how to
>>>> use R as as a client to access data and produce a graph in DHIS2.
>>>>
>>>> Just get a copy of R  and install the DBI and RPostregSQL packages with
>>>>
>>>>>install.packages()
>>>>
>>>>
>>>> After that, just connect to the DB, retrieve your data (in this case
>>>> from a report table) and produce a graph.
>>>>
>>>>>library(DBI)
>>>>
>>>>>library(RPostgreSQL)
>>>>
>>>>>drv <- dbDriver("PostgreSQL")
>>>>
>>>>>con <- dbConnect(drv, dbname="dhis2_zm_prod2", user="postgres", password="postgres")
>>>>
>>>>>rs <- dbSendQuery(con, "SELECT * FROM _report_malaria_indicators_district where
>>>> organisationunitid = 3904")
>>>>
>>>>>data <- fetch(rs,n=-1)
>>>>
>>>>>barplot(data$malaria_confirm_incidence, names.arg=as.character(data$periodname), main=as.character(data$organisationunitname[1]),las=2)
>>>>
>>>>>dev.print(png, file="/home/jason/test.png")
>>>>
>>>> Regards,
>>>> Jason
>>>>
>>>> ---
>>>> Jason P. Pickering
>>>> email: jason.p.pickering@xxxxxxxxx
>>>> tel:+260968395190
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>
>>
>>
>> --
>> --
>> Jason P. Pickering
>> email: jason.p.pickering@xxxxxxxxx
>> tel:+260968395190
>>
>



-- 
--
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+260968395190

_______________________________________________
Mailing list: https://launchpad.net/~dhis2-users
Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~dhis2-users
More help   : https://help.launchpad.net/ListHelp