dhis2-devs team mailing list archive

[Ime Asangansi <asangansi@xxxxxxxxx>]

Hi Bob,

Mmmm..interesting.

Just to be sure.
Were you reading from 1.4 and writing to 2.0? mysql or postgres?

Were you writing the other way (to access)?

Thanks.

Ime


--- On Fri, 7/23/10, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:

> From: Bob Jolliffe <bobjolliffe@xxxxxxxxx>
> Subject: Re: [Dhis2-devs] dhis14 import
> To: "Lars Helge Øverland" <larshelge@xxxxxxxxx>
> Cc: "Ime Asangansi" <asangansi@xxxxxxxxx>, "Knut Staring" <knutst@xxxxxxxxx>, "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
> Date: Friday, July 23, 2010, 3:53 PM
> Alright writing is pretty slow. 
> Writing 10million records take two hours.
> 
> But reading is pretty darn quick.  Iterating through
> 10million rows
> and dumping each row to stdout (redirected to /dev/null)
> takes about 1
> minute +- 5 seconds.   Throughout which
> memory use is constant and
> low: less than 1.5%.  That's got to be quicker than
> anything using
> jdbc which is obliged to push everything up and down the
> tcp/ip stack.
> 
> Not looking any more at this now - just wanted to see the
> numbers.
> Looks like a good addition to Knut's student project
> list.  Refactor
> dhis14 file import to inject a jackcess backend in place of
> the
> hibernate one.
> 
> Cheers
> Bob
> 
> 2010/7/23 Bob Jolliffe <bobjolliffe@xxxxxxxxx>:
> > 2010/7/22 Lars Helge Øverland <larshelge@xxxxxxxxx>:
> >>
> >> No doubt this looks much simpler.
> >> Would be interesting to do a test with a large
> table (>10 mill) and see how
> >> it performs in terms of memory usage.
> >
> > 10 million records is a lot of records!  I have it
> whirring away in
> > the background as I get on with other stuff.  Started
> 1.5 hours ago
> > and still writing records ... db file up to 250M ...
> hasn't started
> > reading back yet but thus far memory usage is low and
> constant.  I'll
> > let you know when/if it finishes :-)
> >
> >> Lars
> >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi
> <asangansi@xxxxxxxxx>
> wrote:
> >>>
> >>> Impressive!
> >>> First time seeing that clean functionality!
> >>> I see potential there to move data between
> both systems :)
> >>>
> >>> Ime
> >>>
> >>>
> >>> --- On Tue, 7/20/10, Knut Staring <knutst@xxxxxxxxx>
> wrote:
> >>>
> >>> > From: Knut Staring <knutst@xxxxxxxxx>
> >>> > Subject: Re: [Dhis2-devs] dhis14 import
> >>> > To: "Bob Jolliffe" <bobjolliffe@xxxxxxxxx>
> >>> > Cc: "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
> >>> > Date: Tuesday, July 20, 2010, 3:32 PM
> >>> > That sounds really great - it has
> >>> > been problematic to require Windows for
> this.
> >>> >
> >>> > k
> >>> >
> >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob
> Jolliffe <bobjolliffe@xxxxxxxxx>
> >>> > wrote:
> >>> > > Just some throwaway code testing out
> jackcess for
> >>> > reading dhis14 (and
> >>> > > potentially modulo basico files):
> >>> > >
> >>> > > http://pastebin.com/wMv1SZqq
> >>> > >
> >>> > > I'm pretty impressed.  It works
> well and I suspect
> >>> > also much faster
> >>> > > than accessing via odbc/ibatis or
> whatever it is.
> >>> >  Never mind the
> >>> > > nonsense of what this code actually
> does - the point
> >>> > is that it can
> >>> > > iterate over access tables using
> java (on ubuntu).
> >>> >  Kind of nice.
> >>> > >
> >>> > > Cheers
> >>> > > Bob
> >>> > >
> >>> > >
> _______________________________________________
> >>> > > Mailing list: https://launchpad.net/~dhis2-devs
> >>> > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> > > More help   : https://help.launchpad.net/ListHelp
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Cheers,
> >>> > Knut Staring
> >>> >
> >>> >
> _______________________________________________
> >>> > Mailing list: https://launchpad.net/~dhis2-devs
> >>> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> > Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> > More help   : https://help.launchpad.net/ListHelp
> >>> >
> >>>
> >>>
> _______________________________________________
> >>> Mailing list: https://launchpad.net/~dhis2-devs
> >>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~dhis2-devs
> >>> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >
>