dhis2-devs team mailing list archive

[Bob Jolliffe <bobjolliffe@xxxxxxxxx>]

Alright writing is pretty slow.  Writing 10million records take two hours.

But reading is pretty darn quick.  Iterating through 10million rows
and dumping each row to stdout (redirected to /dev/null) takes about 1
minute +- 5 seconds.   Throughout which memory use is constant and
low: less than 1.5%.  That's got to be quicker than anything using
jdbc which is obliged to push everything up and down the tcp/ip stack.

Not looking any more at this now - just wanted to see the numbers.
Looks like a good addition to Knut's student project list.  Refactor
dhis14 file import to inject a jackcess backend in place of the
hibernate one.

Cheers
Bob

2010/7/23 Bob Jolliffe <bobjolliffe@xxxxxxxxx>:
> 2010/7/22 Lars Helge Øverland <larshelge@xxxxxxxxx>:
>>
>> No doubt this looks much simpler.
>> Would be interesting to do a test with a large table (>10 mill) and see how
>> it performs in terms of memory usage.
>
> 10 million records is a lot of records!  I have it whirring away in
> the background as I get on with other stuff.  Started 1.5 hours ago
> and still writing records ... db file up to 250M ... hasn't started
> reading back yet but thus far memory usage is low and constant.  I'll
> let you know when/if it finishes :-)
>
>> Lars
>> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi <asangansi@xxxxxxxxx> wrote:
>>>
>>> Impressive!
>>> First time seeing that clean functionality!
>>> I see potential there to move data between both systems :)
>>>
>>> Ime
>>>
>>>
>>> --- On Tue, 7/20/10, Knut Staring <knutst@xxxxxxxxx> wrote:
>>>
>>> > From: Knut Staring <knutst@xxxxxxxxx>
>>> > Subject: Re: [Dhis2-devs] dhis14 import
>>> > To: "Bob Jolliffe" <bobjolliffe@xxxxxxxxx>
>>> > Cc: "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>> > Date: Tuesday, July 20, 2010, 3:32 PM
>>> > That sounds really great - it has
>>> > been problematic to require Windows for this.
>>> >
>>> > k
>>> >
>>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>>> > wrote:
>>> > > Just some throwaway code testing out jackcess for
>>> > reading dhis14 (and
>>> > > potentially modulo basico files):
>>> > >
>>> > > http://pastebin.com/wMv1SZqq
>>> > >
>>> > > I'm pretty impressed.  It works well and I suspect
>>> > also much faster
>>> > > than accessing via odbc/ibatis or whatever it is.
>>> >  Never mind the
>>> > > nonsense of what this code actually does - the point
>>> > is that it can
>>> > > iterate over access tables using java (on ubuntu).
>>> >  Kind of nice.
>>> > >
>>> > > Cheers
>>> > > Bob
>>> > >
>>> > > _______________________________________________
>>> > > Mailing list: https://launchpad.net/~dhis2-devs
>>> > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
>>> > > More help   : https://help.launchpad.net/ListHelp
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Cheers,
>>> > Knut Staring
>>> >
>>> > _______________________________________________
>>> > Mailing list: https://launchpad.net/~dhis2-devs
>>> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> > Unsubscribe : https://launchpad.net/~dhis2-devs
>>> > More help   : https://help.launchpad.net/ListHelp
>>> >
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>