maria-developers team mailing list archive

["Mark Wadham" <ubuntu@xxxxxx>]

Hi,

Apologies if this is the wrong place to ask this question but it seemed 
like the most apt.

We are working on operational processes for online schema changes with 
MariaDB 10.1.16 and have encountered memory issues which are proving 
difficult to reliably solve.

If anyone on this list could explain to me in detail how to calculate 
the memory footprint of modifying a large table with DDL statements I 
would be very grateful.

Our test case is a 3-node galera cluster on which we want to alter a 
~160GB table that has .  Each node has 64GB ram and (originally) no swap 
space allocated.  Even with the innodb buffer pool lowered to 16GB we 
consistently reached an OOM condition trying to add a column to the 
table.  If we lower the innodb buffer pool to 16GB *and* provision 16GB 
of swap, the DDL completes.  With the innodb buffer pool left at 50GB 
and allocated swap of 16GB, same OOM condition.

Since this is an operational procedure we are testing the process under 
heavy load, but the node that is applying the DDL is first placed into 
RSU mode so the writesets simply build up in the gcache while it's 
executing (and obviously traffic from the production environment would 
be directed away from the node while this is happening).  We are also 
increasing the gcache to 800GB on the node as part of this process to 
ensure that we have plenty of time for it to complete.

I'm fairly sure that the gcache is simply a continuous write-to-disk 
process rather than anything involving significant memory, so *hope* 
that's not a contributing factor here.  I am however quite surprised 
that executing DDL on a table can be so memory-intensive.

Our next test run will be with the innodb buffer pool set to 0 on the 
node executing the DDL with no swap allocated to see if it can complete 
purely in-memory.  Logically it should be able to if this is a memory 
constraint rather than a bug.

But fundamentally we need to be able to determine with confidence how 
much memory we need to successfully execute DDL on a table of {n} size.

Thanks,
Mark