← Back to team overview

maria-developers team mailing list archive

MariaDB memory usage when rebuilding a large table

 

Hi,

Apologies if this is the wrong place to ask this question but it seemed like the most apt.

We are working on operational processes for online schema changes with MariaDB 10.1.16 and have encountered memory issues which are proving difficult to reliably solve.

If anyone on this list could explain to me in detail how to calculate the memory footprint of modifying a large table with DDL statements I would be very grateful.

Our test case is a 3-node galera cluster on which we want to alter a ~160GB table that has . Each node has 64GB ram and (originally) no swap space allocated. Even with the innodb buffer pool lowered to 16GB we consistently reached an OOM condition trying to add a column to the table. If we lower the innodb buffer pool to 16GB *and* provision 16GB of swap, the DDL completes. With the innodb buffer pool left at 50GB and allocated swap of 16GB, same OOM condition.

Since this is an operational procedure we are testing the process under heavy load, but the node that is applying the DDL is first placed into RSU mode so the writesets simply build up in the gcache while it's executing (and obviously traffic from the production environment would be directed away from the node while this is happening). We are also increasing the gcache to 800GB on the node as part of this process to ensure that we have plenty of time for it to complete.

I'm fairly sure that the gcache is simply a continuous write-to-disk process rather than anything involving significant memory, so *hope* that's not a contributing factor here. I am however quite surprised that executing DDL on a table can be so memory-intensive.

Our next test run will be with the innodb buffer pool set to 0 on the node executing the DDL with no swap allocated to see if it can complete purely in-memory. Logically it should be able to if this is a memory constraint rather than a bug.

But fundamentally we need to be able to determine with confidence how much memory we need to successfully execute DDL on a table of {n} size.

Thanks,
Mark


Follow ups