maria-discuss team mailing list archive

[Igor Mazur <igor.kozlov@xxxxxxxxx>]

Sorry, I've forgotten to say that I use xtrabackup.

2016-04-14 2:04 GMT+03:00 Andrew Garner <muzazzi@xxxxxxxxx>:

> What SST method are you using? Based on this I expect you are using rsync.
>
> I imagine someone knows more detailed answers here, but from my
> understanding:
>
> Internally the bin/wsrep_sst_rsync script indicates it needs "flush
> tables" to the server and the server runs this "sst_flush_tables()"
> function that effectively runs:
>
> FLUSH TABLES WITH READ LOCK
> + reload_acl_and_cache(thd, REFRESH_ENGINE_LOG | REFRESH_BINARY_LOG, ...)
>
> Which I think is equivalent to FLUSH ENGINE LOGS / FLUSH BINARY LOGS.
> I think around this point it also sets innodb_disallow_writes to stop
> InnoDB's background I/O (so rsync can be consistent).
>
> At that point the bin/wsrep_sst_rsync script graps the last
> $BINLOG_N_FILES binary logs, which I believe is hardcoded to '1' and
> seems to be evaluated after this flush condition.   But internally I
> guess InnoDB can have a prepared transaction related to a previous
> binlog.
>
> I've wrestled with similar problems writing backup recovery routines
> for MariaDB + snapshot backups (typically via LVM snapshots), where I
> typically have to go 2 logs back and it seems to eliminate this kind
> of failure.  You might be able to bump up BINLOG_N_FILES in
> /usr/bin/wsrep_sst_rsync (or wherever it is located) and see if that
> resolves your race condition.
>
> Alternatively I think percona-xtrabackup has solved this problem[1] a
> while back as well, so a different wsrep-sst-method may be an option.
>  Although xtrabackup has its own problems - in particularly I've found
> that xtrabackup currently doesn't handle lost+found directories too
> well[2], so you either have to avoid having the datadir directly under
> the mountpoint or use some other filesystem (such as XFS) that doesn't
> create lost+found.  I suppose if you're using encryption, xtrabackup
> is also not currently an option. :)
>
> ~Andrew
>
> [1] https://bugs.launchpad.net/percona-xtrabackup/+bug/1254227
> [2] https://bugs.launchpad.net/percona-xtrabackup/+bug/1272329
>
> On Wed, Apr 13, 2016 at 4:27 PM, Igor Mazur <igor.kozlov@xxxxxxxxx> wrote:
> > Hi,
> >
> > Almost everytime when I join node to Galera cluster (MariaDB 10.1.13) I'm
> > getting an error after SST:
> >
> > 2016-04-13 21:09:37 139932281419712 [Note] Recovering after a crash using
> > /var/log/mysql/mariadb-bin
> > 2016-04-13 21:09:37 139932281419712 [ERROR] Binlog file
> > '/var/log/mysql/mariadb-bin.000023' not found in binlog index, needed for
> > recovery. Aborting.
> > 2016-04-13 21:09:37 139932281419712 [ERROR] Crash recovery failed. Either
> > correct the problem (if it's, for example, out of memory error) and
> restart,
> > or delete (or rename) binary log and start mysqld with
> > --tc-heuristic-recover={commit|rollback}
> > 2016-04-13 21:09:37 139932281419712 [ERROR] Can't init tc log
> > 2016-04-13 21:09:37 139932281419712 [ERROR] Aborting
> >
> > As I see, almost every SST rotates a log file and downloads last (in
> example
> > above I have mariadb-bin.000024), and not previous. But for recovering it
> > needs previous.
> > Sometimes I'm lucky and it restores.
> >
> > Anyone knows why it rotates a log and is it possible to force it to load
> > more than last log file?
> >
> > Igor Mazur
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~maria-discuss
> > Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~maria-discuss
> > More help   : https://help.launchpad.net/ListHelp
> >
>