maria-developers team mailing list archive

Thread
Date

Re: Query Stuck in "closing table" State

To: Sergei Golubchik <serg@xxxxxxxxxxxx>
From: Gordan Bobic <gordan@xxxxxxxxxx>
Date: Wed, 20 Mar 2013 12:00:31 +0000
Cc: maria-developers@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20130320114248.GA1149@meddwl.fritz.box>
User-agent: Roundcube Webmail/0.4.2

On Wed, 20 Mar 2013 12:42:48 +0100, Sergei Golubchik<serg@xxxxxxxxxxxx> wrote:

Hi, Gordan!

On Mar 20, Gordan Bobic wrote:

 I have a situation with MariaDB 10.0.0 where an INSERT INTO ...
SELECT query that takes 3 seconds to run ends up stuck in theclosing
 tables state for 15+ minutes while burning 100% of CPU.

 I have seen bug reports dating back 5 years that sound very similar
 but none of them are listed as resolved.

 Is there a way to debug/troubleshoot this further? The problem is
quite intermittent. My data is in a state of constant flux sofindingan exact reproducible test case based on data lone isn't likely tobe
 viable. Is there a reasonable way to capture more detailed debug
 information than what is available from "show full processlist"?


Next time you see that, you could try a poor man's profiler of Domas:
http://poormansprofiler.org/

Simply, collect all the stack traces with

  gdb -ex "set pagination 0" -ex "thread apply all bt" \
    --batch -p $(pidof mysqld) > stack.traces

You can also repeat that few times, like, every five minutes.


Thanks for the tip.

Stack trace attached in a text file.
Does this provide enough information for a worthwhile bug report?

Gordan

[Thread debugging using libthread_db enabled]
[New Thread 0x7fa4f8297700 (LWP 29012)]
[New Thread 0x7fa4f83bb700 (LWP 28473)]
[New Thread 0x7fa4f8372700 (LWP 26598)]
[New Thread 0x7fa4fc4f9700 (LWP 26160)]
[New Thread 0x7fa4fc542700 (LWP 26012)]
[New Thread 0x7fa4fc58b700 (LWP 25937)]
[New Thread 0x7fa1a35fe700 (LWP 25936)]
[New Thread 0x7fa1a3fff700 (LWP 25935)]
[New Thread 0x7fa1a8f3a700 (LWP 25934)]
[New Thread 0x7fa1a993b700 (LWP 25933)]
[New Thread 0x7fa1ac13f700 (LWP 25932)]
[New Thread 0x7fa1aa33c700 (LWP 25931)]
[New Thread 0x7fa1aad3d700 (LWP 25930)]
[New Thread 0x7fa1ab73e700 (LWP 25929)]
[New Thread 0x7fa1acb40700 (LWP 25925)]
[New Thread 0x7fa1ad541700 (LWP 25924)]
[New Thread 0x7fa1adf42700 (LWP 25923)]
[New Thread 0x7fa1ae943700 (LWP 25922)]
[New Thread 0x7fa1af344700 (LWP 25921)]
[New Thread 0x7fa1afd45700 (LWP 25920)]
[New Thread 0x7fa1b0746700 (LWP 25919)]
[New Thread 0x7fa1b1147700 (LWP 25918)]
[New Thread 0x7fa1b1b48700 (LWP 25917)]
[New Thread 0x7fa1b2549700 (LWP 25916)]
[New Thread 0x7fa1b2f4a700 (LWP 25915)]
[New Thread 0x7fa1b394b700 (LWP 25914)]
[New Thread 0x7fa1b434c700 (LWP 25913)]
[New Thread 0x7fa1b4d4d700 (LWP 25912)]
[New Thread 0x7fa1b574e700 (LWP 25911)]
[New Thread 0x7fa1b614f700 (LWP 25910)]
[New Thread 0x7fa1b6b50700 (LWP 25909)]
[New Thread 0x7fa1b7551700 (LWP 25908)]
[New Thread 0x7fa1b7f52700 (LWP 25907)]
[New Thread 0x7fa1b8953700 (LWP 25906)]
[New Thread 0x7fa1b9354700 (LWP 25905)]
[New Thread 0x7fa1b9d55700 (LWP 25904)]
[New Thread 0x7fa1ba756700 (LWP 25903)]
[New Thread 0x7fa1bb157700 (LWP 25902)]
[New Thread 0x7fa1bbb58700 (LWP 25901)]
[New Thread 0x7fa1bc559700 (LWP 25900)]
[New Thread 0x7fa1bcf5a700 (LWP 25899)]
[New Thread 0x7fa1bd95b700 (LWP 25898)]
[New Thread 0x7fa1be35c700 (LWP 25897)]
[New Thread 0x7fa1bed5d700 (LWP 25896)]
[New Thread 0x7fa1bf75e700 (LWP 25895)]
[New Thread 0x7fa1c015f700 (LWP 25894)]
[New Thread 0x7fa1c0b60700 (LWP 25893)]
[New Thread 0x7fa1c1561700 (LWP 25892)]
[New Thread 0x7fa4f9267700 (LWP 25891)]
0x00007fa4fa956a63 in poll () from /lib64/libc.so.6

Thread 50 (Thread 0x7fa4f9267700 (LWP 25891)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000088f4d7 in my_service_thread_sleep ()
#2  0x0000000000887a83 in ma_checkpoint_background ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7fa1c1561700 (LWP 25892)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7fa1c0b60700 (LWP 25893)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7fa1c015f700 (LWP 25894)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7fa1bf75e700 (LWP 25895)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7fa1bed5d700 (LWP 25896)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7fa1be35c700 (LWP 25897)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7fa1bd95b700 (LWP 25898)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7fa1bcf5a700 (LWP 25899)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7fa1bc559700 (LWP 25900)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7fa1bbb58700 (LWP 25901)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7fa1bb157700 (LWP 25902)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7fa1ba756700 (LWP 25903)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7fa1b9d55700 (LWP 25904)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7fa1b9354700 (LWP 25905)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7fa1b8953700 (LWP 25906)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7fa1b7f52700 (LWP 25907)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7fa1b7551700 (LWP 25908)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7fa1b6b50700 (LWP 25909)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7fa1b614f700 (LWP 25910)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7fa1b574e700 (LWP 25911)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fa1b4d4d700 (LWP 25912)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fa1b434c700 (LWP 25913)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fa1b394b700 (LWP 25914)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fa1b2f4a700 (LWP 25915)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fa1b2549700 (LWP 25916)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fa1b1b48700 (LWP 25917)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fa1b1147700 (LWP 25918)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fa1b0746700 (LWP 25919)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fa1afd45700 (LWP 25920)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fa1af344700 (LWP 25921)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fa1ae943700 (LWP 25922)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fa1adf42700 (LWP 25923)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fa1ad541700 (LWP 25924)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fa1acb40700 (LWP 25925)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fa1ab73e700 (LWP 25929)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009590e4 in lock_wait_timeout_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fa1aad3d700 (LWP 25930)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009be2d0 in srv_error_monitor_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fa1aa33c700 (LWP 25931)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009c08c6 in srv_monitor_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fa1ac13f700 (LWP 25932)):
#0  0x00007fa4fa958d03 in select () from /lib64/libc.so.6
#1  0x0000000000971c0f in os_thread_sleep(unsigned long) ()
#2  0x00000000009c1b64 in srv_master_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fa1a993b700 (LWP 25933)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970bfb in os_event_wait_low(os_event_struct*, long) ()
#2  0x00000000009c1668 in srv_purge_coordinator_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fa1a8f3a700 (LWP 25934)):
#0  0x00007fa4fa958d03 in select () from /lib64/libc.so.6
#1  0x0000000000971c0f in os_thread_sleep(unsigned long) ()
#2  0x0000000000a183db in buf_flush_page_cleaner_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fa1a3fff700 (LWP 25935)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970bfb in os_event_wait_low(os_event_struct*, long) ()
#2  0x0000000000a14f3e in buf_dump_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fa1a35fe700 (LWP 25936)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009e96e0 in ib_wqueue_timedwait(ib_wqueue_struct*, long) ()
#3  0x0000000000a699e4 in fts_optimize_thread(void*) ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fa4fc58b700 (LWP 25937)):
#0  0x00007fa4fc1f32a5 in sigwait () from /lib64/libpthread.so.0
#1  0x00000000004fd64b in signal_hand ()
#2  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fa4fc542700 (LWP 26012)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004ffaaf in one_thread_per_connection_end(THD*, bool) ()
#2  0x0000000000639475 in do_handle_one_connection(THD*) ()
#3  0x0000000000639602 in handle_one_connection ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fa4fc4f9700 (LWP 26160)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fa4f8372700 (LWP 26598)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fa4f83bb700 (LWP 28473)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004ffaaf in one_thread_per_connection_end(THD*, bool) ()
#2  0x0000000000639475 in do_handle_one_connection(THD*) ()
#3  0x0000000000639602 in handle_one_connection ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fa4f8297700 (LWP 29012)):
#0  0x00007fa4fa901050 in memcpy () from /lib64/libc.so.6
#1  0x000000000065ff6b in JOIN_CACHE::get_record_by_pos(unsigned char*) ()
#2  0x000000000066037c in JOIN_CACHE::get_record() ()
#3  0x000000000065e521 in JOIN_CACHE::join_matching_records(bool) ()
#4  0x000000000065b398 in JOIN_CACHE::join_records(bool) ()
#5  0x000000000065b3d4 in JOIN_CACHE::join_records(bool) ()
#6  0x00000000005a656f in evaluate_join_record(JOIN*, st_join_table*, int) ()
#7  0x00000000005a68a5 in sub_select(JOIN*, st_join_table*, bool) ()
#8  0x00000000005acdbd in do_select(JOIN*, List<Item>*, TABLE*, Procedure*) ()
#9  0x00000000005cbc7a in JOIN::exec_inner() ()
#10 0x00000000005cdb4a in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#11 0x00000000005ce81d in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#12 0x0000000000582b98 in mysql_execute_command(THD*) ()
#13 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#14 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#15 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#16 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#17 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#18 0x0000000000580839 in mysql_execute_command(THD*) ()
#19 0x000000000058fe42 in Prepared_statement::execute(String*, bool) ()
#20 0x00000000005942d7 in Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*) ()
#21 0x000000000059477b in mysql_sql_stmt_execute(THD*) ()
#22 0x000000000057fc01 in mysql_execute_command(THD*) ()
#23 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#24 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#25 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#26 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#27 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#28 0x0000000000580839 in mysql_execute_command(THD*) ()
#29 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#30 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#31 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#32 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#33 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#34 0x0000000000622ddd in Event_job_data::execute(THD*, bool) ()
#35 0x00000000007f9d86 in Event_worker_thread::run(THD*, Event_queue_element_for_exec*) ()
#36 0x00000000007f9e21 in event_worker_thread ()
#37 0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#38 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa4fc60e7e0 (LWP 25889)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x00000000004fe074 in handle_connections_sockets() ()
#2  0x00000000005052f2 in mysqld_main(int, char**) ()
#3  0x00007fa4fa896cdd in __libc_start_main () from /lib64/libc.so.6
#4  0x00000000004f87d9 in _start ()

Follow ups

Re: Query Stuck in "closing table" State
From: Sergei Golubchik, 2013-03-20

References

Query Stuck in "closing table" State
From: Gordan Bobic, 2013-03-20
Re: Query Stuck in "closing table" State
From: Sergei Golubchik, 2013-03-20