dhis2-devs team mailing list archive

[David Muturi <dnmuturi@xxxxxxxxx>]

Hi Paulo,
Here it is...
this update is intended for server administrators looking to upgrade
to DHIS 2.11.

In version 2.11, DHIS is performing on-the-fly data aggregation as
part of the new analytics API. This is in contrast to the datamart API
where data is been pre-aggregated for a set of dimensions. The
analytics tables contain indexed columns for each data dimension
together with a column for the "raw" data values. The tables are
partitioned for each year of data. Partition means a part of a table
which has been split horizontally on rows - you will find tables like
analytics_2011, analytics_2012, analytics_2013 in the database. This
has some consequences for server configuration:


1) It is strongly recommended to set up a cache server to reduce the
pressure on your web servers. First, caching must be enabled in DHIS
from "System general settings" by setting "Cache strategy" to "Cache
until 6 am tomorrow". The simplest way to set up a server-side cache
if you already have a reverse proxy like nginx and apache running is
to configure it directly in the proxy. Enabling caching for nginx is
documented here:

http://dhis2.org/doc/snapshot/en/implementer/html/ch08s02.html#d5e580

Apache caching can be done through the mod_cache module:
http://httpd.apache.org/docs/2.2/mod/mod_cache.html

Otherwise, dedicated caches like varnish
(https://www.varnish-cache.org/) and squid
(http://www.squid-cache.org/) could be set up in front of your web
servers.



2) Run latest version of PostgreSQL which is 9.2. It has some nice
performance improvements over earlier versions.



3) Make sure a lot of memory is available to the database /
PostgreSQL. If the analytics tables for the last 2-3 years can fit in
the memory cache then on-the-fly aggregation will perform much better.
You can see the size of the analytics tables and indexes using this
SQL on PostgreSQL:

select pg_size_pretty( pg_total_relation_size( 'analytics_2013' ) );

The more memory you have the better. Documentation has been updated,
have a look at the values for shared_buffers and effective_cache_size
settings and update your configurations adjusted for your available
memory:

http://dhis2.org/doc/snapshot/en/implementer/html/ch08.html#d5e413



4) Make sure you have enough disk space as t
he size of the database will increase with the analytics tables. Also,
when doing database backups it's not required to include the analytics
nor the datamart tables. An efficient command for creating backups
which omits these tables and compresses the output in one go (replace
db-name and db-user) is:

pg_dump db-name -U db-user -T aggregated* -T analytics* -T
completeness* | gzip > dhis-pgdump.gz


5) Optional. Use a fast hard disk type/drive. It will help performance
both on analytics table generation and read queries if you have a fast
disk type, ideally SSD.


rgds
David

2013/3/25 Paulo Grácio <paulogracio@xxxxxxxxx>:
> Congratulations for the achivement!
>
> Is there any information about how to upgrade from version 2.10 to 2.11?
>
> Kind regards,
> Paulo Grácio
>
> On 22 March 2013 19:05, Lars Helge Øverland <larshelge@xxxxxxxxx> wrote:
>>
>> Hi all,
>>
>> DHIS version 2.11 is out and we have some great features for you:
>>
>>
>> - Web-based pivot table: Allows you to create pivot tables with data
>> aggregated along all data dimensions, including data element/organisation
>> unit group sets. Data dimensions can be arranged freely on columns, rows and
>> as filters. You can include totals and subtotals on all axis and control the
>> table density and font size under options. Your pivots can be saved as
>> favorites and downloaded to your local computer. This design goal is to
>> allow users to create their own reports with the data they need without any
>> involvement of super-users.
>>
>> A pivot table with immunization indicators for last 4 quarters by facility
>> ownership, type and location can be made like this (log in with
>> admin/district):
>>
>> http://apps.dhis2.org/demo/dhis-web-pivot/app/index.html?id=XWXTmFUffng
>>
>> A pivot table with number of diagnosis by facility ownership and type with
>> totals and subtotals looks like this (notice that diagnosis are data element
>> groups):
>>
>> http://apps.dhis2.org/demo/dhis-web-pivot/app/index.html?id=vlnZeGhgw1T
>>
>> A pivot table for vaccines, broken down on categories, age group and point
>> of service, last 12 months:
>>
>> http://apps.dhis2.org/demo/dhis-web-pivot/app/index.html?id=z31v4RMNqnu
>>
>> A pivot table with reporting rates, major forms for public facilities by
>> location and type, last 12 months:
>>
>> http://apps.dhis2.org/demo/dhis-web-pivot/app/index.html?id=VHewKMjyLnR
>>
>> We encourage you to play around - select dimension items for the left menu
>> and arrange dimensions from "Layout" in the top menu. Documentation can be
>> found here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch15.html
>>
>>
>>
>> - Data analytics Web-API resource: A flexible API for retrieving aggregated,
>> analytical data for all data dimensions using a simple query syntax. The
>> pivot table module is built on top of this API. It allows developers of
>> third-party apps and web portals to connect to DHIS 2 and query for any type
>> of aggregated data, where any number of dimensions and filters can be
>> specified. Data can be retrieved in json, jsonp, xml, csv, html and xls
>> (Excel) format and responses can be constrained with criteria like greater
>> than, equals and so on.
>>
>> A query for retrieving data for certain data elements, periods and
>> organisation units might look like this:
>>
>> http://apps.dhis2.org/demo/api/analytics?dimension=dx:fbfJHSPpUQD;cYeuwXTCPkU&dimension=pe:2012Q1;2012Q2&dimension=ou:O6uvpzGd5pu;lc3eMKXaEfw
>>
>> Documentation can be found here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch25s18.html
>>
>>
>>
>> - Sharing: Several objects including chart, pivot table, map, indicator and
>> data set can now be shared and have fine-grained access control. An object
>> can be made private to yourself, shared with any number of user groups or be
>> made available for all. You can grant edit or view-only rights to other
>> people. You can create user groups directly from the dashboard. Who should
>> be allowed to create public objects can be controlled through user roles.
>>
>> This functionality is useful for example in a scenario where entities like
>> health programs and implementing partners need to create indicators and
>> reports which should be available only for members of their organisation. In
>> this case, an organisation can create a user group for their staff, then
>> develop charts and maps which could be made available for that organisation
>> only. General reports made by the ministry could still be shared with all
>> and made publicly available.
>>
>> Another use-case is that of an international NGO with operations in multiple
>> countries. Here, user groups could be set up for each country and
>> indicators, resources, charts and reports which are country-specific could
>> be made available for staff working in that country only, without appearing
>> in lists for staff working in other countries. Indicators and reports which
>> are applicable for all counties could be made accessible for all. The
>> sharing solution also opens up for great possibilities when combined with
>> the self-registration of user account feature.
>>
>> Documentation here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch07.html
>>
>>
>>
>> - DXF 2 meta-data exchange: You can now import and export a complete set of
>> objects in the aggregate part of the system, including data sets, custom
>> data entry forms, maps, charts and option sets. Import/export of meta-data
>> objects now uses the new DXF 2 format. Import/export can also be done
>> through the meta-data Web API resource. This greatly improves the ability
>> for meta-data exchange between DHIS 2 instances and third-party systems.
>>
>> Documentation here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch19.html
>>
>>
>> - HTML-based standard reports: You can now develop standard reports in pure
>> HTML and Javascript. This requires a little bit of development skills but
>> allows for creating pixel-perfect reports tailored to most needs. It is
>> especially flexible when used in combination with the analytics Web API,
>> where aggregated data can be pulled out of the system and positioned in the
>> report according to your needs.
>>
>> An example of such a report can be viewed here:
>>
>> http://apps.dhis2.org/demo/dhis-web-reporting/generateHtmlReport.action?uid=fqERdm6UtkI&ou=ImspTQPwCqd
>>
>> Documentation here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch14s04.html#d5e2332
>>
>>
>> - Multi-organisation unit data entry: You can now do data entry for multiple
>> organisation unit in the same form. This is useful for instance in scenarios
>> where you have section forms with few data elements and an organisational
>> hierarchy with hospitals with multiple wards below, as it allows you to do
>> fast data entry for all wards at the same time. To enable it, go the system
>> general settings and enable "multi-organisation unit forms", then go to data
>> entry and select the organisation unit above the ones you want to enter data
>> for.
>>
>> Documentation here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch09s02.html#d5e1122
>>
>>
>> - Tracker ad-hoc aggregate reports: You can now do on-the-fly aggregation of
>> tracker data in tabular reports. Define aggregation queries with filters or
>> group statements and view the results instantly. Create reports like top 10
>> diagnoses from inpatient admissions, save as favorite and run the report
>> again whenever you like. You can check it out here:
>>
>> http://apps.dhis2.org/demo/dhis-web-caseentry/app/index.html
>>
>>
>> - Tracker program statistics reports: Tracker now comes with better out of
>> the box statistics to program managers. Monitor new enrollments, programs
>> completed, and drop-outs. Also analyse in detail from each stage how many
>> visits are done, completed and overdue. Try it out on the demo here:
>>
>> http://apps.dhis2.org/demo/dhis-web-caseentry/statisticalProgramReportSelect.action
>>
>>
>> - Tracker general improvements: The tracker module is rapidly being improved
>> and the user experience is improved in person dashboard, case-based tabular
>> reports, person aggregation and program maintenance.
>>
>> The tracker user guide can be found here:
>>
>> http://dhis2.org/doc/snapshot/en/user/html/ch24s02.html#d5e5474
>>
>>
>> - Facility registry Web API resource: DHIS 2 now implements the API
>> specification made by the FRED project. This means that DHIS 2 can be
>> considered a feasible facility registry. More info here:
>>
>> http://facilityregistry.org
>>
>>
>> - Model changes: Indicator formulas, validation rule formulas and custom
>> data entry forms are now using UIDs (stable identifiers) which makes it
>> possible to transfer these easily between systems. Category options can now
>> be re-used within many categories which improves analysis and naming
>> conventions.
>>
>>
>> - Miscellaneous: Report table supports weekly periods. Data visualizer has
>> an improved favorite user interface, making it consistent with pivot table
>> and GIS. Touch-friendly navigation for help and profile items (click on help
>> or profile in top menu).
>>
>> http://apps.dhis2.org/demo/dhis-web-commons-about/functions.action
>>
>>
>> - The scheduling functionality has been improved. As a consequence you must
>> re-set your scheduled jobs after upgrading in data administration ->
>> scheduling.
>>
>>
>> WAR file, Live package, sample data and source code can be downloaded from
>> here:
>>
>> http://dhis2.org/downloads
>>
>>
>> Documentation and Javadocs can be found here:
>>
>> http://dhis2.org/documentation
>>
>>
>> Launchpad source code page:
>>
>> https://launchpad.net/dhis2/+milestone/2.11
>>
>>
>> Launchpad blueprint page:
>>
>> https://launchpad.net/dhis2/+milestone/2.11
>>
>>
>>
>> best regards,
>>
>> the DHIS 2 development team
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-users
>> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-users
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
> --
> Paulo Grácio
>
> _______________________________________________
> Mailing list: https://launchpad.net/~dhis2-users
> Post to     : dhis2-users@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~dhis2-users
> More help   : https://help.launchpad.net/ListHelp