dhis2-devs team mailing list archive

[Jason Pickering <jason.p.pickering@xxxxxxxxx>]

Hi Lars,
Thanks for this. It is a huge relief to know that it will work, albeit
only with the developer massaging it at this point. :)


> - I had to remove the data values larger than 2^31, which is basically the
> budget data, from the database manually.

We will need to change this in DHIS 1.4 somehow. Would it be possible
for DHIS to implement a check to ensure that the value is not out of
the allowable range? If it is, then perhaps it could simply ignore
these values, and inform the user.


> - Set the environment variable JAVA_OPTS, mine is      -Xms128m -Xmx512m
> -XX:PermSize=128m -XX:MaxPermSize=1024m

After looking at the logs again, I saw some PermGenSpace errors that
occurred directly after the import started. I am not sure if they are
related to the import or something else. It could be related but there
were no timestamps on the errors, so it is difficult to tell. I assume
they are related though, because I did not see the last part of what
showed up in your log, in my log.  I will try and up the memory a bit
more on this machine and see what happens.


> - Give Postgres more memory in pginstalldir/data/postgresql.conf by setting
> the   shared_buffers    property, mine is   256MB

Thanks. I will take a look at it.

> - For inital imports (!), in the DHIS 2 import screen - "Show advanced
> options" - set "Skip check for matching datavalues" to "yes (empty
> database)"
>

Could you explain this option?  I would assume that it will check for
existing data values and not overwrite them. If I choose no, I will
assume it will attempt to insert everything?

> o Regarding db maintenance I did a few things under "Data administration" -
> "Maintenance":
>
> - I ran "Clear zero values" which reduced the total amount of data values
> from 4.9 to 3.3 millions.
>
> - I ran "Prune periods" which reduced the total amount of periods from 99 to
> 68, all of this will speed up the datamart process.

I think we can do this from the DHIS 1.4 side as well. WIll take a look at it.

>
> - I entered the orgunit level names manually under "Organisation Unit" -
> "Organisation Unit Levels".
>
>
> o What you need to do is the following:
>
> - Have a look at the indicator and calculated data element identifiers. I am
> now replacing missing identifiers with "0", which might imply unwanted
> results or division-by-zero.

We are working on this now. Am having some difficulties with DHIS 1.4
in this regard, but we know it is an issue with this database.


>
> - Change the aggregation operator from SUM to AVERAGE for data elements
> where you want to do the average when aggregating, eg. Age and Population
> related elements. The sustainable solution to this is to change this in DHIS
> 1.4, which simply always sets this property to SUM.

Also known to us. We thought this had been fixed sometime ago by Ola
but it seems to have crept back into the system through a recent
update.

>
> o I am not sure why you didn't get it to work. This sounds a little silly
> but are you sure you waited long enough? My import on a high-end maching
> took 32 min. Is the GUI message saying "Import process complete", or is it
> still saying "Importing routine data values" ?  This is the end of my log.
> Yours stops at "Imported Periods"

Yes, I waited. Well, in fact ,I did not have to wait. It just executed
like lightning and gave  me a "null" response in the browser window. I
did not get anything like "Import process complete".


>
> -------------
>
> * INFO  12:58:25,726 Imported Periods (DefaultDhis14FileImportService.java
> [Thre
> ad-9])
> * INFO  13:27:31,520 Imported RoutineDataValues
> (DefaultDhis14FileImportService.
> java [Thread-9])
> * INFO  13:27:34,427 Imported OnChangePeriods
> (DefaultDhis14FileImportService.ja
> va [Thread-9])
> * INFO  13:27:58,880 Imported SemiPermanentDataValues
> (DefaultDhis14FileImportSe
> rvice.java [Thread-9])
> * INFO  13:27:58,958 Import process completed: 0:32:01.611
> (ImportInternalProces
> s.java [Thread-9])
>
> ------------------
>
>
>
> You can download the pg backup file here (created in windows/pgadmin):
>
> http://folk.uio.no/larshelg/files/dhis2zambiaMarch2009.backup

Big help Lars. This will at least enable me to start having a look at
the development of the reports (which they need ASAP) and then we can
continue to clean up the other issues.

Tack,
Jason