maria-discuss team mailing list archive

[Ljr Yang <dbmsprog@xxxxxxxxx>]

Hi, All
     Recently, We use Sysbench  Test  MariaDB 10.1.13.  In Test ,We found
server lock and can't write .
Testing environment:
server1:  10.2.1.242    (HP OEM Fusion IO PCIE Card  3.2TB )
server2:  10.2.1.239    (ScanDisk Fusion IO PCIE Card 3.2TB)

sysbench test script:
 sysbench --test=/usr/share/doc/sysbench/tests/db/oltp.lua --num-threads=16
 --mysql-host=localhost --mysql-port=3306
--mysql-socket=/data/mysql2/data/mysql.sock --mysql-user=root
--mysql-password=123456 --mysql-db=test --max-time=0 --max-requests=0
 --report-interval=1 --oltp-tables-count=16 run

when *wsrep_on = on *,  after run a few minutes or a few hours, both
server1 and server2  were stucked, all write stoped ,only select statement
run normal  :
................
[ 815s] threads: 16, tps: 6176.00, reads: 86431.00, writes: 24699.00,
response time: 3.37ms (95%), errors: 0.00, reconnects:  0.00
[ 816s] threads: 16, tps: 6064.99, reads: 84907.86, writes: 24256.96,
response time: 3.66ms (95%), errors: 0.00, reconnects:  0.00
[ 817s] threads: 16, tps: 6097.00, reads: 85394.05, writes: 24394.02,
response time: 3.54ms (95%), errors: 0.00, reconnects:  0.00
[ 818s] threads: 16, tps: 6131.97, reads: 85819.52, writes: 24529.86,
response time: 3.65ms (95%), errors: 0.00, reconnects:  0.00
[ 819s] threads: 16, tps: 6143.04, reads: 86036.63, writes: 24575.18,
response time: 3.44ms (95%), errors: 0.00, reconnects:  0.00
[ 820s] threads: 16, tps: 5819.80, reads: 81532.22, writes: 23332.20,
response time: 3.60ms (95%), errors: 0.00, reconnects:  0.00
[ 821s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
[ 822s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
[ 823s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
[ 824s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
[ 825s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
[ 826s] threads: 16, tps: 0.00, reads: 0.00, writes: 0.00, response time:
0.00ms (95%), errors: 0.00, reconnects:  0.00
................

In error log file not error info , after execute* pstack `pidof mysqld` >
pstack.log*,  sysbench run normal again , and start write data.

Every time have this problem,  in pstack.log file contains a few lock:
#0  0x00007f2b04fe0334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2b04fdb5d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007f2b04fdb4a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
....................

When *set wsrep_on = off*,  Sysbench test MariaDB 10.1.13 normal.

As contrast ,We download
Percona-XtraDB-Cluster-5.6.28-rel76.1-25.14.1.Linux.x86_64.ssl101.tar.gz
and run sysbench test, found same problem too.

We can't confirm , is it hardware or galera question ?   please help
analyze ,thank you.

Reference link:
https://en.wikibooks.org/wiki/Linux_Applications_Debugging_Techniques/Deadlocks
http://www.linuxprogrammingblog.com/threads-and-fork-think-twice-before-using-them
-- 
杨建军