maria-developers team mailing list archive

[Kristian Nielsen <knielsen@xxxxxxxxxxxxxxx>]

"Rasmus Johansson" <rasmus@xxxxxxxxxxxxxxxx> writes:

> I would need your help to understand the different replication topics that
> are very active now:
>
> - The replication work  that you have been doing 

As far as implementation goes, it is these four worklogs, which are in code
review and available in the feature preview tree
lp:~maria-captains/maria/mariadb-5.2-rpl:

    http://askmonty.org/worklog/Server-Sprint/?tid=116
    http://askmonty.org/worklog/Server-RawIdeaBin/?tid=136
    http://askmonty.org/worklog/Server-Sprint/?tid=163
    http://askmonty.org/worklog/Server-Sprint/?tid=132

This work improves replication in a number of ways:

 - Implement working group commit for the replication binlog. This greatly
   improves performance when running with sync_binlog=1, which is necessary to
   ensure the ability to reliably recover replication state after a master
   crash.

 - Ensure same commit order between innodb and binlog, without the need for
   the expensive prepare_commit_mutex. This is needed by XtraBackup and InnoDB
   hot backup to do a non-blocking online backup that can be used to provision
   a slave. The other existing patches I have seen solve this by having the
   user need to turn on/off the prepare_commit_mutex depending on whether
   backup is going on or not.

 - Allow to obtain an InnoDB consistent read snapshot and the corresponding
   binlog position in a fully non-blocking way. This allows for example to
   provision a slave from a fully non-blocking `mysqldump --master-data
   --single-transaction`. Previously, this required FLUSH TABLES WITH READ
   LOCK, which blocks queries in the server for some time.

 - Port the Facebook patch to release InnoDB row locks early, already during
   the prepare phase. This can improve performance in the presense of hot-spot
   rows in busy OLTP applications. (This feature must be enabled explicitly,
   it is off by default).

 - Make the transaction coordinator inside MariaDB plugable, allowing to plug
   in different implementations. This is mostly preparatory work for pluggable
   replication, as the replication binlog functions as a transaction
   coordinator. So with this, a new binlog implementation can be plugged in
   and do the same transaction coordinator role that the legacy binlog
   normally handles.

In addition to this, there was a long discussion on the MariaDB mailing list
about various replication topics, and I wrote up the following specifications
(but did not start on the implementation yet):

    http://askmonty.org/worklog/?tid=107
    http://askmonty.org/worklog/?tid=120
    http://askmonty.org/worklog/?tid=133

The idea with these is to gradually make the existing legacy MySQL replication
and binlog pluggable, opening the way for writing completely new replication
implementations. It builds on top of the implementation work above.

I also worked on the specifications of the parallel replication worklogs
MWL#169 and MWL#170 mentioned below.

> - The parallel replication that Oracle is talking about for MySQL 5.6

Do you mean this?

    http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave
    http://forge.mysql.com/worklog/task.php?id=4648

The idea is to take one transaction at a time, and run the operations inside
it in parallel. That is a bit unusual, most other projects work by running
multiple transaction in parallel.

I think this is motivated by NDB (though it is just me reading between the
lines, I do not know for sure). Since in NDB, multiple parallel transactions
on the master are replicated as a single, bigger transaction called an
epoch. So they need something on the slave that can take apart the epoch again
and make it parallel.

Frankly, when I first saw this, I dismissed it, thinking it was the wrong
approach to the problem (outside of NDB replication at least).

I am still quite sceptical. Though it may be possible to use this for normal
OLTP workloads (with small transactions) by artificially combining multiple
transactions, at the cost of higher delay between master and slave.

> - The replication functionality that <customer> was interested in together
> with <partner> and us

This is a request for implementing another algorithm for parallel
replication. I wrote up some designs for this:

    http://askmonty.org/worklog/Server-RawIdeaBin/?tid=169
    http://askmonty.org/worklog/Server-RawIdeaBin/?tid=170

This overlaps with the above-mentioned parallel replication project in Oracle,
however the chosen approach/algorithm is quite different between the two. In
this work, it would be the responsibility of the application to ensure that
parallel transactions modifying tables in two different databases are
independent. Then such transactions can be replicated in parallel on the
slave.

> Could you just give a short description on each of those and then describe
> how they connect to each other or are there something that duplicates?

Hope this helps,

 - Kristian.