maria-discuss team mailing list archive

Thread
Date
New Question: MariaDB service fails to start after InnoDB/XtraDB crash

To: maria-discuss@xxxxxxxxxxxxxxxxxxx
From: AskMonty KB <noreply@xxxxxxxxxxxx>
Date: Thu, 11 Jun 2015 14:50:03 -0000
Hello,

A new question has been asked in "XtraDB and InnoDB" by kushkash. Please answer it at http://mariadb.com/kb/en/mariadb-service-fails-to-start-after-innodbxtradb-crash/ as the person asking the question may not be subscribed to the mailing list.

--------------------------------
Hi,
I've been running a script, inserting a large amount of data to several MySQL tables. Apparently, InnoDB crashed at some point (when ibdata1 reached ~1.6 Gb). I had to restart the computer and when I tried to restart MariaDB service, I got the following errors:

Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: 150611  8:58:02  InnoDB: Operating system error number 5 in a file operation.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Error number 5 means 'Input/output error'.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Some operating system error numbers are described at
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: File operation call: 'read'.
Jun 11 08:58:02 honolulu.bi.technion.ac.il mysqld_safe[7970]: InnoDB: Cannot continue operation.'
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: mariadb.service: control process exited, code=exited status=1
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: Failed to start MariaDB database server.
Jun 11 08:58:02 honolulu.bi.technion.ac.il systemd[1]: Unit mariadb.service entered failed state.

I checked /var/log/mariadb/mariadb.log to find out what might have caused the crash:

140911 10:40:38  InnoDB: Error: Write to file ./ibdata1 failed at offset 0 220200960.
InnoDB: 1048576 bytes should have been written, only 0 were written.
InnoDB: Operating system error number 28.
InnoDB: Check that your OS and file system support files of this size.
InnoDB: Check also that the disk is not full or a disk quota exceeded.
InnoDB: Error number 28 means 'No space left on device'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
140911 10:40:38  InnoDB: Assertion failure in thread 139942049851136 in file os0file.c line 4382
InnoDB: Failing assertion: ret
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately a150505 07:59:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150505 07:59:38 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.SFCcCu' --pid-file='/var/lib/mysql/honolulu.bi.technion.ac.il-recover.pid'
150505 07:59:41 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
150505  7:59:41 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
150505  7:59:41 InnoDB: The InnoDB memory heap is disabled
150505  7:59:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150505  7:59:41 InnoDB: Compressed tables use zlib 1.2.8
150505  7:59:41 InnoDB: Using Linux native AIO
150505  7:59:41 InnoDB: Initializing buffer pool, size = 128.0M
150505  7:59:41 InnoDB: Completed initialization of buffer pool
150505  7:59:41 InnoDB: highest supported file format is Barracuda.
150505  7:59:41  InnoDB: Waiting for the background threads to start
150505  7:59:42 Percona XtraDB (http://www.percona.com) 5.5.37-MariaDB-35.0 started; log sequence number 458740332
150505  7:59:42 [Note] Plugin 'FEEDBACK' is disabled.
150505  7:59:42 [Note] Server socket created on IP: '0.0.0.0'.
150505  7:59:42 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired
150505  7:59:42 [Warning] Checking table:   './mysql/user'
150505  7:59:42 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly
150505  7:59:42 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired
150505  7:59:42 [Warning] Checking table:   './mysql/db'
150505  7:59:42 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly
150505  7:59:42 [Note] Event Scheduler: Loaded 0 events
150505  7:59:42 [Note] WSREP: Read nil XID from storage engines, skipping position init
150505  7:59:42 [Note] WSREP: wsrep_load(): loading provider library 'none'
150505  7:59:42 [Note] [Debug] WSREP: dummy_init
150505  7:59:42 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.38-MariaDB-wsrep'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server, wsrep_25.10.r3997
150608 04:43:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150608 04:43:00 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.7Yb3bw' --pid-file='/var/lib/mysql/honolulu.bi.technion.ac.il-recover.pid'
150608 04:43:13 mysqld_safe WSREP: Failed to recover position: 
'150608  4:43:01 InnoDB: The InnoDB memory heap is disabled
150608  4:43:01 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150608  4:43:01 InnoDB: Compressed tables use zlib 1.2.8
150608  4:43:01 InnoDB: Using Linux native AIO
150608  4:43:01 InnoDB: Initializing buffer pool, size = 128.0M
150608  4:43:01 InnoDB: Completed initialization of buffer pool
150608  4:43:01 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 4676403586
150608  4:43:01  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Error: tried to read 1048576 bytes at offset 0 1048576.
InnoDB: Was only able to read 655360.
150608  4:43:13  InnoDB: Operating system error number 5 in a file operation.
InnoDB: Error number 5 means 'Input/output error'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
InnoDB: File operation call: 'read'.
InnoDB: Cannot continue operation.'

I have to say that there shouldn't be any disk space problem:
df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       819G   27G  750G   4% /
devtmpfs         7.9G       0   7.9G    0% /dev
tmpfs              7.9G        0   7.9G    0% /dev/shm
tmpfs              7.9G  8.9M   7.9G    1% /run
tmpfs              7.9G        0   7.9G    0% /sys/fs/cgroup
tmpfs              7.9G   12K    7.9G   1% /tmp
/dev/sda3       1.9T  123M   1.8T    1% /home
/dev/sda2      477M  112M  336M  25% /boot

The MariaDB is currently installed in default configuration.

I really need help on this issue...
1. What might have cause the problem? Do I have to configure some InnoDB parameters? Maybe the ibdata file size is a problem?
2. How to recover my data and restart the MariaDB service? (I tried all kind of tips I found in other forums, but I couldn't backup the ibdata1 file or start InnoDB in recovery mode).





--------------------------------

To view or answer this question please visit: http://mariadb.com/kb/en/mariadb-service-fails-to-start-after-innodbxtradb-crash/