maria-developers team mailing list archive

[Sergei Golubchik <serg@xxxxxxxxxxx>]

Hi, andrei.elkin!

On Oct 18, andrei.elkin@xxxxxxxxxx wrote:
> Sergei,
> 
> > Hi, Andrei!
> >
> > First, when I run it I got:
> >
> > Warning: 112 bytes lost at 0x7f212f26a670, allocated by T@0 at
> > sql/sql_binlog.cc:68, sql/sql_binlog.cc:181, sql/sql_parse.cc:5625,
> > sql/sql_parse.cc:7465, sql/sql_parse.cc:1497, sql/sql_parse.cc:1124,
> > sql/sql_connect.cc:1330, sql/sql_connect.cc:1243
> >
> > so you have a memory leak with your thd->lex->comment.str
> 
> Turns out LEX cleaned up at the end of decoded event execution. Thanks
> for alerting!
> 
> Btw, do you compile with clang or gcc? Asking seeing my stack unresolved.

gcc. It should not matter, I think. You need to build with -g and have
addr2line in the path.

> > Also, I'm thinking whether BINLOG (in this last form) could
> > automatically clear its variables @a and @b. Without an explicit SET
> > @a=NULL. It's a bit weird to have such a side effect. But it'll help
> > to save memory - the event size it over 3GB, right? So you'll have
> > it split in two variables. Then you create a concatenated copy (it's
> > over 3GB more), then you decode it (already more than 9GB in
> > memory), and then it's executed where copies of huge blob values are
> > created again, may be more than once. Freeing @a and @b and the
> > concatenated copy as soon as possible will save over 6GB in RAM
> > usage.
> 
> Well, in the regular case of replaying all three phases of
>  SET, BINLOG, UNSET
> time between the last two is not really big.

It's not a question of time, but of how much memory you need *at the
same time*. Now you do

1. allocate >3GB for both variables
2. allocate >3GB for a combined concatenated event
3. allocate ~3GB for a base64-decoded event
4. allocate memory for copies of blobs (2*3GB?)
--- now you have 15GB allocated
5. free blobs
6. free base64 decoded event
7. free concatenated event
8. free variables

And I suggest to do it like

1. allocate >3GB for both variables
2. allocate >3GB for a combined concatenated event
3. free variables
4. allocate ~3GB for a base64-decoded event
5. free concatenated event
6. allocate memory for copies of blobs (2*3GB?)
--- now you have 9GB allocated
7. free blobs
8. free base64 decoded event

So the peak memory consumption is >6GB less.

Regards,
Sergei
Chief Architect MariaDB
and security@xxxxxxxxxxx