← Back to team overview

maria-discuss team mailing list archive

Re: SST with 10.1.13 + xtrabackup-v2 is failing


Yes, the datadir was missing in the config which Guillaume was kind enough to point out to me.


On 05/06/2016 01:49 PM, Andrew Garner wrote:
This looks curiously similar to this xtrabackup bug:


Are you defining a datadir explicitly in your config?  You might check
if it's being picked up right in "xtrabackup --help | grep ^datadir"


On Fri, May 6, 2016 at 9:52 AM, Erik Mathis
<erik.mathis@xxxxxxxxxxxxxxxxx> wrote:
I cant seem to get past this problem when using xtrabackup-v2 to preform an
SST. Everything works fine when I use rsync method.
This is a new cluster. I know the sst user/password is working which I

2016-05-06 14:34:31 140258805017344 [Note] WSREP: State transfer required:
     Group state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8
     Local state: 00000000-0000-0000-0000-000000000000:-1
2016-05-06 14:34:31 140258805017344 [Note] WSREP: New cluster view: global
state: b5013090-12e9-11e6-ba99-bf6bb7612fa3:8, view# 43: Primary, number of
nodes: 3, my index: 0, protocol version 3
2016-05-06 14:34:31 140258805017344 [Warning] WSREP: Gap in state sequence.
Need state transfer.
2016-05-06 14:34:31 140258796631808 [Note] WSREP: Running:
'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '' --datadir
'/var/lib/mysql/'   --parent '28455' --binlog 'mysql-bin''
WSREP_SST: [INFO] Streaming with xbstream (20160506 14:34:32.597)
WSREP_SST: [INFO] Using socat as streamer (20160506 14:34:32.605)
WSREP_SST: [INFO] Evaluating timeout -s9 100 socat -u
TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} )
(20160506 14:34:32.740)
2016-05-06 14:34:32 140258805017344 [Note] WSREP: Prepared SST request:
2016-05-06 14:34:32 140258805017344 [Note] WSREP: wsrep_notify_cmd is not
defined, skipping notification.
2016-05-06 14:34:32 140258805017344 [Note] WSREP: REPL Protocols: 7 (3, 2)
2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue
2016-05-06 14:34:32 140258805017344 [Note] WSREP: Assign initial position
for certification: 8, protocol version: 3
2016-05-06 14:34:32 140258905679616 [Note] WSREP: Service thread queue
2016-05-06 14:34:32 140258805017344 [Warning] WSREP: Failed to prepare for
incremental state transfer: Local state UUID
(00000000-0000-0000-0000-000000000000) does not match group state UUID
(b5013090-12e9-11e6-ba99-bf6bb7612fa3): 1 (Operation not permitted)
      at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be
2016-05-06 14:34:32 140258832279296 [Note] WSREP: Member 0.0 () requested
state transfer from '*any*'. Selected 1.0 ()(SYNCED) as donor.
2016-05-06 14:34:32 140258832279296 [Note] WSREP: Shifting PRIMARY -> JOINER
(TO: 8)
2016-05-06 14:34:32 140258805017344 [Note] WSREP: Requesting state transfer:
success, donor: 1
WSREP_SST: [INFO] Proceeding with SST (20160506 14:34:33.153)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio |
xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160506 14:34:33.152)
WSREP_SST: [INFO] Cleaning the existing datadir and innodb-data/log
directories (20160506 14:34:33.163)
removed `/var/lib/mysql/ib_logfile0'
removed `/var/lib/mysql/aria_log_control'
removed `/var/lib/mysql/aria_log.00000001'
removed `/var/lib/mysql/mysql-bin.index'
removed `/var/lib/mysql/ib_logfile1'
removed `/var/lib/mysql/ibdata1'
removed `/var/lib/mysql/mysql.sock'
removed `/var/lib/mysql/mysql-bin.state'
removed `/var/lib/mysql/mysql-bin.000001'
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20160506
2016-05-06 14:34:34 140258842769152 [Note] WSREP: (a4dbb726,
'tcp://') turning message relay requesting off
2016-05-06 14:34:44 140258832279296 [Note] WSREP: 1.0 (): State transfer to
0.0 () complete.
2016-05-06 14:34:44 140258832279296 [Note] WSREP: Member 1.0 () synced with
ls: cannot access mysql-bin.[0-9]*: No such file or directory
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20160506
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log
$rebuildcmd ${DATA} &>${DATA}/innobackup.prepare.log (20160506 14:34:45.021)
rm: cannot remove `/var/lib/mysql//innobackup.prepare.log': No such file or
rm: cannot remove `/var/lib/mysql//innobackup.move.log': No such file or
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20160506
WSREP_SST: [INFO] Evaluating innobackupex   --no-version-check --move-back
--force-non-empty-directories ${DATA} &>${DATA}/innobackup.move.log
(20160506 14:34:50.135)
WSREP_SST: [ERROR] Cleanup after exit with status:2 (20160506 14:34:50.167)
2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Process completed with
error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address ''
--datadir '/var/lib/mysql/'   --parent '28455' --binlog 'mysql-bin' : 2 (No
such file or directory)
2016-05-06 14:34:50 140258796631808 [ERROR] WSREP: Failed to read uuid:seqno
and wsrep_gtid_domain_id from joiner script.
2016-05-06 14:34:50 140260071184416 [ERROR] WSREP: SST failed: 2 (No such
file or directory)
2016-05-06 14:34:50 140260071184416 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

The Joiner innobackup.move.log has this error

innobackupex version 2.4.2 based on MySQL server 5.7.11 Linux (x86_64)
(revision id: 8e86a84)
Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file
14:34:50 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000

Mailing list: https://launchpad.net/~maria-discuss
Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~maria-discuss
More help   : https://help.launchpad.net/ListHelp