maria-developers team mailing list archive

[김경남 <knamkim@xxxxxxxxx>]

Hi, 

We're use mariadb 10.1.3 as a main database of our system.
We migrated data from develpment server to product server a week ago.


Below is our error case : 

MariaDB shutdown with signal 6 when some query executes. (insert or
update...)

The error log is below.
------------------------------------------------------------------------
----------------------------------
...
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: However key
management plugin or used key_id 286 is not found or used encryption
algorithm or method does not match.
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: However key
management plugin or used key_id 286 is not found or used encryption
algorithm or method does not match.
2016-07-18  9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
2016-07-18 09:44:14 7f38f3c36b00  InnoDB: Assertion failure in thread
139882584566528 in file buf0buf.cc line 4803
...
...
------------------------------------------------------------------------
-----------------------------------------------

We check out system innodb table space with innochecksum.
The result is below.
------------------------------------------------------------------------
-----
 /engn001/masvc01/mysql/bin/innochecksum -v ibdata1

 InnoDB offline file checksum utility.
 
 Variables (--variable-name=value)
 and boolean options {FALSE|TRUE}  Value (after reading options)
 ---------------------------------
----------------------------------------
 verbose                           TRUE
 debug                             FALSE
 skip-corrupt                      FALSE
 count                             FALSE
 start-page                        0
 end-page                          0
 page                              0
 per-page-details                  FALSE
 leaf                              FALSE
 merge                             0
 Table is uncompressed
 Page size is 16384
 file ibdata1_20160718 = 3221225472 bytes (196608 pages)...
 InnoChecksum; checking pages in range 0 to 196607
 Fail; page 0 invalid (fails innodb and crc32 checksum)
------------------------------------------------------------------------
----
We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of
innochecksum are same.

We compiled mariadb with debug option and test under same enviroment.
Below is the mysql.trace when db crash.
-------------------------------------------------
========================================================================
==================================================
T@10   : | | | | | | | | | | | | | | | | | >print_buffer_to_file
T@10   : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB:
Marking tablespace as missing. You may drop this table or install
correct key management plugin and key file.
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) locking
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) locked
T@10   : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log
(0x15b4b60) unlocking
T@10   : | | | | | | | | | | | | | | | | | <print_buffer_to_file
T@10   : | | | | | | | | | | | | | | | | <vprint_msg_to_log
T@10   : | | | | | | | | | | | | | | | <sql_print_error
T@10   : | | | | | | | | | | | | | | | >my_malloc
T@10   : | | | | | | | | | | | | | | | | my: size: 4096  my_flags: 16
T@10   : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690
T@10   : | | | | | | | | | | | | | | | <my_malloc
T@10   : | | | | | | | | | | | | | | | >push_warning_printf
T@10   : | | | | | | | | | | | | | | | | enter: warning: 192
T@10   : | | | | | | | | | | | | | | | | >push_warning
T@10   : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table
in tablespace 0 encrypted.However key management plugin or used key_id
290 is not found or used encryption algorithm or method does not match.
Can't continue opening the table.
T@10   : | | | | | | | | | | | | | | | | | >THD::raise_condition
T@10   : | | | | | | | | | | | | | | | | | |
>mysql_audit_acquire_plugins
T@10   : | | | | | | | | | | | | | | | | | |
<mysql_audit_acquire_plugins
T@10   : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock
(0x5dcaeb00) locking
T@10   : | | | | | | | | | | | | | | | | | | >my_multi_malloc
T@10   : | | | | | | | | | | | | | | | | | | | >my_malloc
T@10   : | | | | | | | | | | | | | | | | | | | | my: size: 208
my_flags: 24
T@10   : | | | | | | | | | | | | | | | | | | | | exit: ptr:
0x7f9e8415a710
T@10   : | | | | | | | | | | | | | | | | | | | <my_malloc
T@10   : | | | | | | | | | | | | | | | | | | <my_multi_malloc
T@10   : | | | | | | | | | | | | | | | | | | >my_hash_init
T@10   : | | | | | | | | | | | | | | | | | | | enter: hash:
0x7f9e8415a710  size: 128
T@10   : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2
T@10   : | | | | | | | | | | | | | | | | | | | | >my_malloc
------------------------------------------------------------------------
---------------------------------------------------


So we export data from mariadb and import to new table space. 
After migration, we shutdown mariadb and restart. but signal 11
occurred and db crash. 
Below is our error log.

------------------------------------------------------------------------
--------------------------
2016-07-19  8:55:09 140556344748800 [Note]
/engn001/masvc01/mysql/bin/mysqld: Normal shutdown

2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Killing the
scheduler thread, thread id 1
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Waiting for
the scheduler thread to reply
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Stopped
2016-07-19  8:55:09 140556344748800 [Note] Event Scheduler: Purging the
queue. 0 events
160719  8:55:09 server_audit: STOPPED
2016-07-19  8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread
exiting.
2016-07-19  8:55:09 140556344748800 [Note] InnoDB: Starting shutdown...
2016-07-19  8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer
pool(s) to /data001/masvc01/ib_buffer_pool
2016-07-19  8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s)
dump completed at 160719  8:55:10
2016-07-19  8:55:15 140556344748800 [Note] InnoDB: Shutdown completed;
log sequence number 399145649446
2016-07-19  8:55:15 140556344748800 [Note]
/engn001/masvc01/mysql/bin/mysqld: Shutdown complete

160719 08:55:16 mysqld_safe mysqld from pid file
/engn001/masvc01/mysql/mysqld.pid ended
160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from
/data001/masvc01
2016-07-19  8:57:05 140304014350176 [Note]
/engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as
process 2608 ...
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref
count buffer pool pages
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory
heap is disabled
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks
use GCC atomic builtins
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Memory barrier is
not used
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Compressed tables
use zlib 1.2.3
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using Linux native
AIO
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32
instructions
2016-07-19  8:57:06 140304014350176 [Note] InnoDB: Initializing buffer
pool, size = 24.0G
2016-07-19  8:57:08 140304014350176 [Note] InnoDB: Completed
initialization of buffer pool
2016-07-19  8:57:10 140304014350176 [Note] InnoDB: Highest supported
file format is Barracuda.
2016-07-19  8:57:13 140304014350176 [Note] InnoDB: 128 rollback
segment(s) are active.
2016-07-19  8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to
start
2016-07-19  8:57:13 140304014350176 [Note] InnoDB:  Percona XtraDB
(http://www.percona.com <http://www.percona.com/> ) 5.6.29-76.2
started; log sequence number 399145649446
2016-07-19  8:57:13 140273638147840 [Note] InnoDB: Dumping buffer
pool(s) not yet started
2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from
/data001/masvc01/ib_buffer_pool
2016-07-19  8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is
disabled.
160719  8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0
STARTED.
2016-07-19  8:57:13 140304014350176 [Note] Server socket created on IP:
'::'.
2016-07-19  8:57:13 140304014350176 [Warning] 'user' entry
'root@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19  8:57:13 140304014350176 [Warning] 'user' entry
'@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19  8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@%
root@lgedglap02v' ignored in --skip-name-resolve mode.
2016-07-19  8:57:13 140273595853568 [Note] Event Scheduler: scheduler
thread started with id 1
2016-07-19  8:57:13 140304014350176 [Note] Reading of all Master_info
entries succeded
2016-07-19  8:57:13 140304014350176 [Note] Added new Master_info '' to
hash table
2016-07-19  8:57:13 140304014350176 [Note]
/engn001/masvc01/mysql/bin/mysqld: ready for connections.
Version: '10.1.14-MariaDB'  socket:
'/engn001/masvc01/mysql/mysqld.sock'  port: 3310  MariaDB Server
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: However key
management plugin or used key_id 92 is not found or used encryption
algorithm or method does not match.
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0
in file /data001/masvc01/ibdata1 encrypted.
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: However key
management plugin or used key_id 92 is not found or used encryption
algorithm or method does not match.
2016-07-19  8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace
as missing. You may drop this table or install correct key management
plugin and key file.
160719  8:57:47 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this
binary
or one of the libraries it was linked against is corrupt, improperly
built,
or misconfigured. This error can also be caused by malfunctioning
hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.1.14-MariaDB
key_buffer_size=67108864
read_buffer_size=50331648
max_used_connections=3
max_threads=102
thread_count=4
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads =
10094612 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400
/engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae]
/engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4]
/lib64/libpthread.so.0[0x3ffe40f7e0]
/engn001/masvc01/mysql/bin/mysqld[0x8fa6d3]
/engn001/masvc01/mysql/bin/mysqld[0xa3ba47]
/engn001/masvc01/mysql/bin/mysqld[0xa51a2e]
/engn001/masvc01/mysql/bin/mysqld[0xa52bc9]
/engn001/masvc01/mysql/bin/mysqld[0xa433ff]
/engn001/masvc01/mysql/bin/mysqld[0xa440b7]
/lib64/libpthread.so.0[0x3ffe407aa1]
/lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html
contains
information that should help you find out what is causing the crash.
.....

------------------------------------------------------------------------
--------------------------------------------------
I can't find solution for it. 
Is it a bug  or hardware fault ? 

Regards.

 





  김경남 차장
    
   CTO 아키텍처담당 
   Post Architecture팀
  02-2099-6177 / 010-2208-3006



 

 


  _____  

상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약
송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기
바랍니다.
상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS.
This message and its attachments may contain confidential information,
and they are intended to be viewed or used by only the individuals
specified in the message. If you have received this message in an error
from the sender, please contact the sender immediately to notify the
error and delete all of the message and its copies. It is prohibited to
view, use, make public and/or distribute part or whole of this message
without written permission.