dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

2010/7/23 Bob Jolliffe <bobjolliffe@xxxxxxxxx>

> Alright writing is pretty slow.  Writing 10million records take two hours.
>
> But reading is pretty darn quick.  Iterating through 10million rows
> and dumping each row to stdout (redirected to /dev/null) takes about 1
> minute +- 5 seconds.   Throughout which memory use is constant and
> low: less than 1.5%.  That's got to be quicker than anything using
> jdbc which is obliged to push everything up and down the tcp/ip stack.
>
> Not looking any more at this now - just wanted to see the numbers.
> Looks like a good addition to Knut's student project list.  Refactor
> dhis14 file import to inject a jackcess backend in place of the
> hibernate one.
>
> Cheers
> Bob
>

Thanks for testing this. Looks promising.

Actually what I was most curious about was how data is fetched from such
large tables. By doing a "select * from DataValue" on a 10 mill table we
will run out of memory one way or the other. Doing a "select * from
DataValue order by DataValueID limit 0, 10000" and then increase the limit
takes a long time due to the ordering. Ibatis (what we currenty use) has a
RowHandler interface which deals with this and lets you handle each row
individually with apparently quite effective memory management (I don't know
how this is implemented though.)



>
> 2010/7/23 Bob Jolliffe <bobjolliffe@xxxxxxxxx>:
> > 2010/7/22 Lars Helge Øverland <larshelge@xxxxxxxxx>:
> >>
> >> No doubt this looks much simpler.
> >> Would be interesting to do a test with a large table (>10 mill) and see
> how
> >> it performs in terms of memory usage.
> >
> > 10 million records is a lot of records!  I have it whirring away in
> > the background as I get on with other stuff.  Started 1.5 hours ago
> > and still writing records ... db file up to 250M ... hasn't started
> > reading back yet but thus far memory usage is low and constant.  I'll
> > let you know when/if it finishes :-)
> >
> >> Lars
> >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi <asangansi@xxxxxxxxx>
> wrote:
> >>>
> >>> Impressive!
> >>> First time seeing that clean functionality!
> >>> I see potential there to move data between both systems :)
> >>>
> >>> Ime
> >>>
> >>>
> >>> --- On Tue, 7/20/10, Knut Staring <knutst@xxxxxxxxx> wrote:
> >>>
> >>> > From: Knut Staring <knutst@xxxxxxxxx>
> >>> > Subject: Re: [Dhis2-devs] dhis14 import
> >>> > To: "Bob Jolliffe" <bobjolliffe@xxxxxxxxx>
> >>> > Cc: "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
> >>> > Date: Tuesday, July 20, 2010, 3:32 PM
> >>> > That sounds really great - it has
> >>> > been problematic to require Windows for this.
> >>> >
> >>> > k
> >>> >
> >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx
> >
> >>> > wrote:
> >>> > > Just some throwaway code testing out jackcess for
> >>> > reading dhis14 (and
> >>> > > potentially modulo basico files):
> >>> > >
> >>> > > http://pastebin.com/wMv1SZqq
> >>> > >
> >>> > > I'm pretty impressed.  It works well and I suspect
> >>> > also much faster
> >>> > > than accessing via odbc/ibatis or whatever it is.
> >>> >  Never mind the
> >>> > > nonsense of what this code actually does - the point
> >>> > is that it can
> >>> > > iterate over access tables using java (on ubuntu).
> >>> >  Kind of nice.
> >>> > >
> >>> > > Cheers
> >>> > > Bob
> >>> > >
> >>> > > _______________________________________________
> >>> > > Mailing list: https://launchpad.net/~dhis2-devs
> >>> > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> > > More help   : https://help.launchpad.net/ListHelp
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Cheers,
> >>> > Knut Staring
> >>> >
> >>> > _______________________________________________
> >>> > Mailing list: https://launchpad.net/~dhis2-devs
> >>> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> > Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> > More help   : https://help.launchpad.net/ListHelp
> >>> >
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~dhis2-devs
> >>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >
>