enterprise-support team mailing list archive

[Przemek <1703346@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

In a situation where disk partition used for tmpdir has little free
space, and large enough write  transaction is executed, which does not
fit in binlog_cache_size, an individual write fails with error, but
actual binary log gets corrupted, causing the slave to fail.

Reproduced for InnoDB tables with compound primary key, with standard
replication as well as for PXC.

Example error messages when tmpdir gets full (ML* of MY* files gets
created, apparently depending on whether total transaction file can't be
increased (ML) or individual update query (MY)):

master [localhost] {msandbox} (test) > update test1 set a=a+1,b=b+1,c=c+2 limit 110000;
ERROR 3 (HY000): Error writing file '/home/przemek/sandboxes/rsandbox_mysql-5_6_36/tmp/MLFgYjKP' (Errcode: 28 - No space left on device)
master [localhost] {msandbox} (test) > update test1 set a=a+1,b=b+1,c=c+2 limit 210000;
ERROR 3 (HY000): Error writing file '/home/przemek/sandboxes/rsandbox_mysql-5_6_36/tmp/MYUTuIOS' (Errcode: 28 - No space left on device)

Example error on the slave side:

                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'binlog truncated in the middle of event; consider out of disk space on master; the first event 'mysql-bin.000008' at 12574553, the last event read from './mysql-bin.000009' at 10668433, the last byte read from './mysql-bin.000009' at 10668452.'
               Last_SQL_Errno: 1032
               Last_SQL_Error: Could not execute Update_rows event on table test.u1; Can't find record in 'u1', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log mysql-bin.000009, end_log_pos 10668433

Example error on PXC slave node:

2017-07-07 18:50:39 8665 [ERROR] Error in Log_event::read_log_event(): 'Sanity check failed', data_len: 0, event_type: 0
2017-07-07 18:50:39 8665 [ERROR] WSREP: applier could not read binlog event, seqno: 378308, len: 2680586
2017-07-07 18:50:47 8665 [Warning] WSREP: Failed to apply app buffer: seqno: 378308, status: 1
         at galera/src/trx_handle.cpp:apply():351
Retrying 2th time

Example output from mysqlbinlog tool:

ERROR: Error in Log_event::read_log_event(): 'Found invalid event in binary log', data_len: 2180319, event_type: 0
ERROR: Could not read entry at offset 7233086: Error in log format or read error.
WARNING: The range of printed events ends with a row event or a table map event that does not have the STMT_END_F flag set. This might be because the last statement was not fully written to the log, or because you are using a --stop-position or --stop-datetime that refers to an event in the middle of a statement. The event(s) from the partial statement have not been written to output.

Test case to be found in upstream bug report:
https://bugs.mysql.com/bug.php?id=86991

** Affects: mysql-server
     Importance: Unknown
         Status: Unknown

** Affects: percona-server
     Importance: Undecided
         Status: New

** Affects: percona-server/5.6
     Importance: Undecided
         Status: Confirmed

** Affects: percona-server/5.7
     Importance: Undecided
         Status: New

** Affects: percona-xtradb-cluster
     Importance: Undecided
         Status: Confirmed

** Affects: percona-xtradb-cluster/5.6
     Importance: Undecided
         Status: Confirmed

** Affects: percona-xtradb-cluster/5.7
     Importance: Undecided
         Status: New


** Tags: i195910

** Also affects: percona-xtradb-cluster
   Importance: Undecided
       Status: New

** Also affects: percona-xtradb-cluster/5.6
   Importance: Undecided
       Status: New

** Also affects: percona-xtradb-cluster/5.7
   Importance: Undecided
       Status: New

** Also affects: percona-server/5.6
   Importance: Undecided
       Status: New

** Also affects: percona-server/5.7
   Importance: Undecided
       Status: New

** Bug watch added: MySQL Bug System #72457
   http://bugs.mysql.com/bug.php?id=72457

** Also affects: mysql-server via
   http://bugs.mysql.com/bug.php?id=72457
   Importance: Unknown
       Status: Unknown

** Changed in: percona-xtradb-cluster/5.6
       Status: New => Confirmed

** Changed in: percona-server/5.6
       Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Server/Client Support Team, which is subscribed to MySQL.
Matching subscriptions: Ubuntu Server/Client Support Team
https://bugs.launchpad.net/bugs/1703346

Title:
  Binlog corruption when tmpdir gets full

To manage notifications about this bug go to:
https://bugs.launchpad.net/mysql-server/+bug/1703346/+subscriptions