maria-discuss team mailing list archive

Thread
Date
One node out of cluster keeps crashing

To: "maria-discuss@xxxxxxxxxxxxxxxxxxx" <maria-discuss@xxxxxxxxxxxxxxxxxxx>
From: Momin Rashid <mrashid@xxxxxxxxxxxxxxxxx>
Date: Tue, 31 Mar 2015 06:49:36 +0000
Accept-language: en-US
Authentication-results: lists.launchpad.net; dkim=none (message not signed) header.d=none;
Thread-index: AQHQa37a40IH8PbheU+UIB5PDPWJuA==
Thread-topic: One node out of cluster keeps crashing
Hello everyone,

MariaDB-Galera-server-10.0.17-1.el7.centos.x86_64
MariaDB-client-10.0.17-1.el7.centos.x86_64
galera-25.3.9-1.rhel7.el7.centos.x86_64

We have a three node MariaDB-galera cluster, and one of our nodes keeps
crashing.  The other two nodes have been running fine for weeks without
issues.  They all have the exact same specifications.  I have even
rebuilt this node from ground up, and the new node still crashed.

Could someone take a look at the following logs and help me figure out
what is wrong, and how can we avoid this in future?



=====================================
2015-03-30 10:13:05 7f5be57fc700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 19 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 116110 srv_active, 0 srv_shutdown, 343879
srv_idle
srv_master_thread log flush and writes: 459986
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 56843
OS WAIT ARRAY INFO: signal count 804274
Mutex spin waits 1039838, rounds 335818, OS waits 4493
RW-shared spins 393078, rounds 1863774, OS waits 46608
RW-excl spins 34511, rounds 1424782, OS waits 5051
Spin rounds per wait: 0.32 mutex, 4.74 RW-shared, 41.28 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 3156054
Purge done for trx's n:o < 3155992 undo n:o < 0 state: running but idle
History list length 814
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 3155294, not started
MySQL thread id 246025, OS thread handle 0x7f5d84124700, query id
32716796 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3154332, not started
MySQL thread id 245589, OS thread handle 0x7f5d4c963700, query id
32642939 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3150705, not started
MySQL thread id 245568, OS thread handle 0x7f5d840db700, query id
32777452 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3152515, not started
MySQL thread id 245295, OS thread handle 0x7f5d851ff700, query id
32560015 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3155566, not started
MySQL thread id 245281, OS thread handle 0x7f5d86d23700, query id
32774720 ha-proxy.prod.lan 10.0.2.11 appl_01 cleaning up
---TRANSACTION 3156038, not started

TOO MANY LOCKS PRINTED FOR THIS TRX: SUPPRESSING FURTHER PRINTS

--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer
thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
 ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
294306 OS file reads, 2485633 OS file writes, 532545 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 63, seg size 65, 28112 merges
merged operations:
 insert 42583, delete mark 939, delete 29
discarded operations:
 insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 121964244699
Log flushed up to   121964244699
Pages flushed up to 121964244699
Last checkpoint at  121964244699
Max checkpoint age    216721613
Checkpoint age target 209949063
Modified age          0
Checkpoint age        0
0 pending log writes, 0 pending chkp writes
153001 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 6263013376; in additional pool allocated 0
Total memory allocated by read views 3288
Internal hash tables (constant factor + variable factor)
    Adaptive hash index 423323504       (96884488 + 326439016)
    Page hash           757784 (buffer pool 0 only)
    Dictionary cache    25112304        (24222544 + 889760)
    File system         854680  (812272 + 42408)
    Lock system         15172408        (15139192 + 33216)
    Recovery system     0       (0 + 0)
Dictionary memory allocated 889760
Buffer pool size        373496
Buffer pool size, bytes 6119358464
Free buffers            34210
Database pages          319362
Old database pages      118034
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 1891, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 294208, created 25154, written 2272207
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 319362, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4184
Database pages          40015
Old database pages      14791
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 237, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36928, created 3087, written 294016
LRU len: 40015, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4881
Database pages          39304
Old database pages      14528
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 232, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36418, created 2886, written 189961
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39304, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4278
Database pages          39906
Old database pages      14748
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 215, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36782, created 3124, written 154398
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39906, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4286
Database pages          39917
Old database pages      14754
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 247, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36327, created 3590, written 392683
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39917, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4215
Database pages          39956
Old database pages      14769
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 276, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36833, created 3123, written 258687
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39956, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4082
Database pages          40127
Old database pages      14831
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 243, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36928, created 3199, written 252207
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 40127, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4010
Database pages          40224
Old database pages      14860
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 223, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 37120, created 3104, written 527443
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 40224, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size        46687
Buffer pool size, bytes 764919808
Free buffers            4274
Database pages          39913
Old database pages      14753
Modified db pages       0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 218, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 36872, created 3041, written 202812
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read
ahead 0.00/s
LRU len: 39913, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
2 read views open inside InnoDB
8 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
8 out of 1000 descriptors used
---OLDEST VIEW---
Normal read view
Read view low limit trx n:o 3156042
Read view up limit trx id 3155989
Read view low limit trx id 3156042
Read view individually stored trx ids:
Read view trx id 3155989
Read view trx id 3156029
Read view trx id 3156038
-----------------
Main thread process no. 2488, id 140032660715264, state: sleeping
Number of rows inserted 1182044, updated 1504189, deleted 4658, read
2328285511
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long
WSREP: BF lock wait long