dhis2-devs team mailing list archive

[Lars Helge Øverland <larshelge@xxxxxxxxx>]

Hi,

this update is intended for server administrators looking to upgrade to
DHIS 2.11.

In version 2.11, DHIS is performing on-the-fly data aggregation as part of
the new analytics API. This is in contrast to the datamart API where data
is been pre-aggregated for a set of dimensions. The analytics tables
contain indexed columns for each data dimension together with a column for
the "raw" data values. The tables are partitioned for each year of data.
Partition means a part of a table which has been split horizontally on rows
- you will find tables like analytics_2011, analytics_2012, analytics_2013
in the database. This has some consequences for server configuration:


1) It is strongly recommended to set up a cache server to reduce the
pressure on your web servers. First, caching must be enabled in DHIS from
"System general settings" by setting "Cache strategy" to "Cache until 6 am
tomorrow". The simplest way to set up a server-side cache if you already
have a reverse proxy like nginx and apache running is to configure it
directly in the proxy. Enabling caching for nginx is documented here:

http://dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html#d5e580

Apache caching can be done through the mod_cache module:
http://httpd.apache.org/docs/2.2/mod/mod_cache.html

Otherwise, dedicated caches like varnish (https://www.varnish-cache.org/) and
squid (http://www.squid-cache.org/) could be set up in front of your web
servers.



2) Run latest version of PostgreSQL which is 9.2. It has some nice
performance improvements over earlier versions.



3) Make sure a lot of memory is available to the database / PostgreSQL. If
the analytics tables for the last 2-3 years can fit in the memory cache
then on-the-fly aggregation will perform much better. You can see the size
of the analytics tables and indexes using this SQL on PostgreSQL:

select pg_size_pretty( pg_total_relation_size( 'analytics_2013' ) );
*

The more memory you have the better. Documentation has been updated, have a
look at the values for shared_buffers and effective_cache_size settings and
update your configurations adjusted for your available memory:

http://dhis2.org/doc/snapshot/en/implementer/html/ch08.html#d5e413



4) Make sure you have enough disk space as t
he size of the database will increase with the analytics tables. Also, when
doing database backups it's not required to include the analytics nor the
datamart tables. An efficient command for creating backups which omits
these tables and compresses the output in one go (replace db-name and
db-user) is:

pg_dump db-name -U db-user -T aggregated* -T analytics* -T completeness* |
gzip > dhis-pgdump.gz


5) Optional. Use a fast hard disk type/drive. It will help performance both
on analytics table generation and read queries if you have a fast disk
type, ideally SSD.


regards,

Lars



*