dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

Jason,


o I have now imported the database I downloaded last night without problems
in 32 minutes, giving 4,9 million data values.

- The code now manages missing calculated data element identifiers. Will be
a part of next DHIS 2 release. If you remove them manually it will work
anyway.

- I had to remove the data values larger than 2^31, which is basically the
budget data, from the database manually.


o To speed things up you can do the following (you have probably done a few
of them):

- Set the environment variable JAVA_OPTS, mine is      -Xms128m -Xmx512m
-XX:PermSize=128m -XX:MaxPermSize=1024m

- Give Postgres more memory in pginstalldir/data/postgresql.conf by setting
the   shared_buffers    property, mine is   256MB

- For inital imports (!), in the DHIS 2 import screen - "Show advanced
options" - set "Skip check for matching datavalues" to "yes (empty
database)"


o Regarding db maintenance I did a few things under "Data administration" -
"Maintenance":

- I ran "Clear zero values" which reduced the total amount of data values
from 4.9 to 3.3 millions.

- I ran "Prune periods" which reduced the total amount of periods from 99 to
68, all of this will speed up the datamart process.

- I entered the orgunit level names manually under "Organisation Unit" -
"Organisation Unit Levels".


o What you need to do is the following:

- Have a look at the indicator and calculated data element identifiers. I am
now replacing missing identifiers with "0", which might imply unwanted
results or division-by-zero.

- Change the aggregation operator from SUM to AVERAGE for data elements
where you want to do the average when aggregating, eg. Age and Population
related elements. The sustainable solution to this is to change this in DHIS
1.4, which simply always sets this property to SUM.


o I am not sure why you didn't get it to work. This sounds a little silly
but are you sure you waited long enough? My import on a high-end maching
took 32 min. Is the GUI message saying "Import process complete", or is it
still saying "Importing routine data values" ?  This is the end of my log.
Yours stops at "Imported Periods":


-------------

* INFO  12:58:25,726 Imported Periods (DefaultDhis14FileImportService.java
[Thre
ad-9])
* INFO  13:27:31,520 Imported RoutineDataValues
(DefaultDhis14FileImportService.
java [Thread-9])
* INFO  13:27:34,427 Imported OnChangePeriods
(DefaultDhis14FileImportService.ja
va [Thread-9])
* INFO  13:27:58,880 Imported SemiPermanentDataValues
(DefaultDhis14FileImportSe
rvice.java [Thread-9])
* INFO  13:27:58,958 Import process completed: 0:32:01.611
(ImportInternalProces
s.java [Thread-9])

------------------



You can download the pg backup file here (created in windows/pgadmin):

http://folk.uio.no/larshelg/files/dhis2zambiaMarch2009.backup



best regards, Lars