dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Bob,

I think this is indeed interesting, but is it really worth spending
any time on? We currently have something that works, and for our 10
million row database in Zambia it took a few hours to import. OK,
fine, it took time, but it was a one off deal.  Is there much to be
gained from a full refactor? I think the typical use case would be a
one time bulk import, which more or less works. There are still some
hiccups, but these are more a function of the differences in the
metadata models between 1.4 and 2.0.

IMHO, what does need to work, is bidirectional syncronization with XML
between the two systems. Although it is theoretically possible to
export data from 2.0 into 1.4, the last time i checked a few weeks
back, it does not work. This funtionality, for me at least, is much
much more important than simply being able to import DHIS 1.4 data
without windows (which I would need in the first place to create 1.4
data!).

Not to be a party pooper or anything, but just trying to give some perspective.

Regards,
Jason


On Fri, Jul 23, 2010 at 10:05 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
> Hi Ime
>
> On 23 July 2010 19:23, Ime Asangansi <asangansi@xxxxxxxxx> wrote:
>> Hi Bob,
>>
>> Mmmm..interesting.
>>
>> Just to be sure.
>> Were you reading from 1.4 and writing to 2.0? mysql or postgres?
>>
>> Were you writing the other way (to access)?
>
> Nothing so grand as any of the above.  Though I am reading and writing
> access.  Reading/writing mysql and postgres will be much more of a
> bottleneck than reading jackcess 'coz of the tcp requirement of jdbc.
>
> In this scenario I'm just running scratch code in the background to
> keep my laptop warm while I try to write :-)  See attached.  jackcess
> is very maven friendly.
>
> mvn assembly:assembly
> java -jar target/dhaccess-jar-with-dendencies.jar
>
> Drink lots of coffee, take a walk, have lunch .... etc
>
> Cheers.
> Bob
>
>
> package org.hisp.dhis;
>
>
> import com.healthmarketscience.jackcess.ColumnBuilder;
> import com.healthmarketscience.jackcess.DataType;
> import java.io.File;
> import java.io.IOException;
> import java.util.Map;
>
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
>
> import com.healthmarketscience.jackcess.Database;
> import com.healthmarketscience.jackcess.Table;
> import com.healthmarketscience.jackcess.TableBuilder;
>
> /**
>  * CPU warmer
>  *
>  */
> public class App
> {
>
>    private static final Log log = LogFactory.getLog( App.class );
>
>    private static long MAXROWS = 10000000;
>
>    public static void main( String[] args )
>    {
>        try
>        {
>            File dbFile = new File( "/home/bobj/src/dhaccess/test.mdb" );
>            Database db = Database.create( dbFile );
>            log.info( "Opened " + dbFile.getName() );
>
>            Map<String, Object> row = null;
>            Table table = new TableBuilder( "test" ).addColumn( new
> ColumnBuilder( "id", DataType.LONG ).toColumn() ).addColumn( new
> ColumnBuilder( "value", DataType.TEXT ).toColumn() ).toTable( db );
>
>
>            long i = 0;
>            for ( i = 0; i < MAXROWS; i++ )
>            {
>                table.addRow( i, "some text" );
>            }
>
>            log.info( "Reading data" );
>            int count = 0;
>            while ( ( row = table.getNextRow() ) != null )
>            {
>                count++;
>                log.debug( row );
>                Integer id = (Integer) row.get( "id" );
>                String value = (String) row.get( "value" );
>
>                System.err.println( id + " : " + value );
>            }
>            log.info( count + " datavalues" );
>        } catch ( IOException ex )
>        {
>            log.info("Ouch: "+ex);
>        }
>
>    }
> }
>
>
>
>
>>
>> Thanks.
>>
>> Ime
>>
>>
>> --- On Fri, 7/23/10, Bob Jolliffe <bobjolliffe@xxxxxxxxx> wrote:
>>
>>> From: Bob Jolliffe <bobjolliffe@xxxxxxxxx>
>>> Subject: Re: [Dhis2-devs] dhis14 import
>>> To: "Lars Helge Øverland" <larshelge@xxxxxxxxx>
>>> Cc: "Ime Asangansi" <asangansi@xxxxxxxxx>, "Knut Staring" <knutst@xxxxxxxxx>, "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>> Date: Friday, July 23, 2010, 3:53 PM
>>> Alright writing is pretty slow.
>>> Writing 10million records take two hours.
>>>
>>> But reading is pretty darn quick.  Iterating through
>>> 10million rows
>>> and dumping each row to stdout (redirected to /dev/null)
>>> takes about 1
>>> minute +- 5 seconds.   Throughout which
>>> memory use is constant and
>>> low: less than 1.5%.  That's got to be quicker than
>>> anything using
>>> jdbc which is obliged to push everything up and down the
>>> tcp/ip stack.
>>>
>>> Not looking any more at this now - just wanted to see the
>>> numbers.
>>> Looks like a good addition to Knut's student project
>>> list.  Refactor
>>> dhis14 file import to inject a jackcess backend in place of
>>> the
>>> hibernate one.
>>>
>>> Cheers
>>> Bob
>>>
>>> 2010/7/23 Bob Jolliffe <bobjolliffe@xxxxxxxxx>:
>>> > 2010/7/22 Lars Helge Øverland <larshelge@xxxxxxxxx>:
>>> >>
>>> >> No doubt this looks much simpler.
>>> >> Would be interesting to do a test with a large
>>> table (>10 mill) and see how
>>> >> it performs in terms of memory usage.
>>> >
>>> > 10 million records is a lot of records!  I have it
>>> whirring away in
>>> > the background as I get on with other stuff.  Started
>>> 1.5 hours ago
>>> > and still writing records ... db file up to 250M ...
>>> hasn't started
>>> > reading back yet but thus far memory usage is low and
>>> constant.  I'll
>>> > let you know when/if it finishes :-)
>>> >
>>> >> Lars
>>> >> On Wed, Jul 21, 2010 at 3:42 AM, Ime Asangansi
>>> <asangansi@xxxxxxxxx>
>>> wrote:
>>> >>>
>>> >>> Impressive!
>>> >>> First time seeing that clean functionality!
>>> >>> I see potential there to move data between
>>> both systems :)
>>> >>>
>>> >>> Ime
>>> >>>
>>> >>>
>>> >>> --- On Tue, 7/20/10, Knut Staring <knutst@xxxxxxxxx>
>>> wrote:
>>> >>>
>>> >>> > From: Knut Staring <knutst@xxxxxxxxx>
>>> >>> > Subject: Re: [Dhis2-devs] dhis14 import
>>> >>> > To: "Bob Jolliffe" <bobjolliffe@xxxxxxxxx>
>>> >>> > Cc: "dhis2-devs" <dhis2-devs@xxxxxxxxxxxxxxxxxxx>
>>> >>> > Date: Tuesday, July 20, 2010, 3:32 PM
>>> >>> > That sounds really great - it has
>>> >>> > been problematic to require Windows for
>>> this.
>>> >>> >
>>> >>> > k
>>> >>> >
>>> >>> > On Tue, Jul 20, 2010 at 3:23 PM, Bob
>>> Jolliffe <bobjolliffe@xxxxxxxxx>
>>> >>> > wrote:
>>> >>> > > Just some throwaway code testing out
>>> jackcess for
>>> >>> > reading dhis14 (and
>>> >>> > > potentially modulo basico files):
>>> >>> > >
>>> >>> > > http://pastebin.com/wMv1SZqq
>>> >>> > >
>>> >>> > > I'm pretty impressed.  It works
>>> well and I suspect
>>> >>> > also much faster
>>> >>> > > than accessing via odbc/ibatis or
>>> whatever it is.
>>> >>> >  Never mind the
>>> >>> > > nonsense of what this code actually
>>> does - the point
>>> >>> > is that it can
>>> >>> > > iterate over access tables using
>>> java (on ubuntu).
>>> >>> >  Kind of nice.
>>> >>> > >
>>> >>> > > Cheers
>>> >>> > > Bob
>>> >>> > >
>>> >>> > >
>>> _______________________________________________
>>> >>> > > Mailing list: https://launchpad.net/~dhis2-devs
>>> >>> > > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> >>> > > Unsubscribe : https://launchpad.net/~dhis2-devs
>>> >>> > > More help   : https://help.launchpad.net/ListHelp
>>> >>> > >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > --
>>> >>> > Cheers,
>>> >>> > Knut Staring
>>> >>> >
>>> >>> >
>>> _______________________________________________
>>> >>> > Mailing list: https://launchpad.net/~dhis2-devs
>>> >>> > Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> >>> > Unsubscribe : https://launchpad.net/~dhis2-devs
>>> >>> > More help   : https://help.launchpad.net/ListHelp
>>> >>> >
>>> >>>
>>> >>>
>>> _______________________________________________
>>> >>> Mailing list: https://launchpad.net/~dhis2-devs
>>> >>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> >>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> >>> More help   : https://help.launchpad.net/ListHelp
>>> >>
>>> >>
>>> >
>>>
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-devs
> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-devs
> More help   : https://help.launchpad.net/ListHelp
>
>



-- 
Jason P. Pickering
email: jason.p.pickering@xxxxxxxxx
tel:+17069260025