dhis2-devs team mailing list archive

[Orvalho Augusto <orvaquim@xxxxxxxxx>]

OK! We must start from somewhere. Thank you!

I am not sure about thes calculations. And this is considering that we want
to guarantee that Mr PG runs the query with all data on RAM. That to happen
one  needs special tunes  for PG.

As a simple rule to guide us it is fine. And we need more benchmarks,
simulations and more facts to evaluate variables.

Caveman




On Thu, May 2, 2013 at 2:22 PM, Lars Helge Øverland <larshelge@xxxxxxxxx>wrote:

> Hi Caveman,
>
> thanks for the input. The memory recommendations are based on the size of
> the indexes which are created for analytics tables - ideally the database
> indexes on the analytics tables for the last 2 years should fit in memory
> for optimal performance. But this depends on the number of dynamic
> dimensions you have set up for your system - hence they should be
> considered a rule-of-thumb as I said. If you want to do testing
> specifically for your system then you can measure the size of the db
> indexes with SQL:
>
> *
>
> select pg_size_pretty(pg_total_relation_size('analytics_2012'));
>
> minus this value
>
> select pg_size_pretty(pg_relation_size('analytics_2012'));
>
> Lars
>
> *
>
>
>
>
> On Thu, May 2, 2013 at 2:01 PM, Bob Jolliffe <bobjolliffe@xxxxxxxxx>wrote:
>
>> Hi Orvalho
>>
>> On 1 May 2013 15:07, Orvalho Augusto <orvaquim@xxxxxxxxx> wrote:
>>
>>> It is true that hardware prices have decreased a lot.
>>>
>>> But for me it seems too excessive these recommendations. So I am
>>> wondering how we arrived to these recommendations. Would be nice if we
>>> share that.
>>>
>>> And for professional solutions it is not only one big machine you buy.
>>> You must have two. So when one dies you have the other ready.
>>>
>>> Some other things to be tried are:
>>> Instead of using one big muscular server you might need to consider to
>>> use one machine for webapplication and other database.
>>>
>>
>> I agree that having a separate machine for the database can be a good
>> thing.  As long as they are connected with a good Gbs backbone.  This also
>> allows you to optimize the two machines differently.  I know in Rwanda they
>> have got away with fairly humble hardware so far by doing this.  Though
>> moving to 2.11 could still prove challenging.
>>
>> Bob
>>
>>
>>
>>>
>>> And the webapplication server actually can be a group of machines doing
>>> it; And for databases you can have just one machine for writing queries
>>> which is master to other servers that are only queried for reading only
>>> (this for MySQL have been done) or other high availability solution.
>>>
>>> For the tools to study your server, let me add some more:
>>> - top - The basic to see the processes and CPU usage
>>>
>>> - vmstat - This is verry important to see how your server is using the
>>> memory. Repeat it and check our the blocks are being used.
>>>
>>> - plotting tools based on snmp may help to profile your server; To see
>>> correlations between traffic/connections and CPU or memory. So you can
>>> consider what to change.
>>>
>>> Hope it helps.
>>>
>>> Caveman
>>>
>>>
>>>
>>>
>>>
>>> On Wed, May 1, 2013 at 12:21 PM, Dayo Adeyomoye <deemoyes@xxxxxxxxx>wrote:
>>>
>>>> Yes you are right, hardware prices are coming down plus a very good
>>>> server can run VMWare to properly allocate resources
>>>>
>>>> BlackBerry: 3114F90C
>>>> Phone: 08028851441, 08035560463
>>>> emails: deemoyes@xxxxxxxxx, addyraph@xxxxxxxxx
>>>> Yahoo IM: deemoyes
>>>> Facebook: pharael
>>>> twitter:pharaell
>>>>   ------------------------------
>>>>  *From:* Lars Helge Øverland <larshelge@xxxxxxxxx>
>>>> *To:* DHIS 2 developers <dhis2-devs@xxxxxxxxxxxxxxxxxxx>; "
>>>> dhis2-users@xxxxxxxxxxxxxxxxxxx" <dhis2-users@xxxxxxxxxxxxxxxxxxx>
>>>> *Sent:* Wednesday, May 1, 2013 11:16 AM
>>>> *Subject:* [Dhis2-users] Server specifications
>>>>
>>>> Hi all,
>>>>
>>>> in version 2.11 and later DHIS performs aggregation on-the-fly as
>>>> opposed to pre-aggregating data into data marts.
>>>>
>>>> This raises the bar a little on your server's specifications, mostly in
>>>> terms of available RAM and disk read speeds.
>>>>
>>>>
>>>> Some rules-of-thumb server recommendations:
>>>>
>>>>
>>>> - RAM: At least 1 GB memory per 1 million captured data records per
>>>> month or per 1000 concurrent users. At least 4 GB for a small instance, 12
>>>> GB for a medium instance.
>>>>
>>>> - CPU cores: 4 CPU cores for a small instance, 8 CPU cores for a medium
>>>> or large instance.
>>>>
>>>> - Disk: Ideally use an SSD. Otherwise use a 7200 rpm disk. Minimum read
>>>> speed is 150 Mb/s, 220 Mb/s is good, 350 Mb/s or better is ideal.
>>>>
>>>>
>>>> So we encourage you to invest in appropriate hardware to continue to
>>>> benefit from all the new features in DHIS. Hardware prices are coming down
>>>> and value for money is increasing at vps' like linode<http://linode.com/>
>>>>  and digitalocean <http://digitalocean.com/>.
>>>>
>>>>
>>>> regards,
>>>>
>>>> Lars
>>>>
>>>> -----
>>>>
>>>> PS. For server admins, here are some tips to measure hard disk/drive
>>>> read speeds on Linux:
>>>>
>>>> - Use the hdparm utility which will give you approximate read speed:
>>>>
>>>> sudo hdparm -Tt /dev/sda
>>>>
>>>>
>>>> - If you are on a virtual server image without /dev/sda you can
>>>> generate a random file and test write and read speed like this:
>>>>
>>>> dd if=/dev/zero of=testfile bs=4k count=2000000
>>>>
>>>> dd if=testfile of=/dev/null bs=4k count=2000000
>>>>
>>>>
>>>> - To do a real-life test, you can install iotop (sudo apt-get install
>>>> iotop) and run it with
>>>>
>>>> sudo iotop
>>>>
>>>> then start DHIS in a new terminal and load a very large pivot table,
>>>> then monitor read speed from iotop. You should be seeing read speeds around
>>>> 12 - 25 Mb/s. Be aware that database queries will be cached after some time.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-users
>>>> Post to    : dhis2-users@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-users
>>>> More help  : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~dhis2-devs
>>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~dhis2-devs
>>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~dhis2-devs
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~dhis2-devs
>> Post to     : dhis2-devs@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~dhis2-devs
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>