maria-developers team mailing list archive

[Rich Prohaska <prohaska@xxxxxxxxxxx>]

we have a customer that has observed problems related to the
interaction of the thread pools in mariadb 5.5.30 and tokudb 7.1.0.
the problem is that connections that stuck for 100's of seconds in the
Killed state when a big tokudb transaction commit is in progress.  the
customer claims that this problem was resolved by turning the thread
pool OFF.  have there been any fixes to the thread pool implementation
post the 5.5.30 release?

here is some information that may be useful.

the processlist showed 2838 client connections of which:
851 Killed
483 Sleep
1467 Connect
1 show processlist
1 processing a commit on a big delete from a tokudb table
35 blocked on a row lock held by the big delete

the Killed connections look like:
84847752 sfi_mysql 10.0.0.60:1814 sfi Killed 93 NULL 0.000
with the connection address and time slightly different

a gdb snapshot of the system at the time of the failure:
291 total threads
130 threads waiting on fil_aio_wait
52 threads waiting on tokudb's work_on_kibbutz
16 thread tokudb deserialization thread waiting
36 mysql update thread blocked on the tokudb lock tree
1thread executing a big tokudb delete transaction
40 threads in the thread pool get_event function
1 tokudb checkpoint thread blocked waiting for the LP MO lock, which
is held by the big txn commit thread
that leaves 15 misc worker threads that are not doing anything interesting


the gdb stacks for the thread pool threads are:
Thread 78 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 71 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 70 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 69 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 68 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 62 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 57 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 52 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 51 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 49 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 45 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 44 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 43 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 42 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 41 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 39 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 37 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 36 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 35 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 33 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 32 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 31 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 26 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 24 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 22 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 21 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 20 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 19 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 18 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 17 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 16 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 13 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 12 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 11 : epoll_wait , io_poll_wait , listener , get_event ,
worker_main , start_thread , clone
Thread 8 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 7 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 5 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 4 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 3 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone
Thread 2 : pthread_cond_timedwait@@GLIBC_2.3.2 ,
inline_mysql_cond_timedwait , get_event , worker_main , start_thread ,
clone


thanks

rich prohaska