maria-developers team mailing list archive

Thread
Date

Re: Query Stuck in "closing table" State

To: Sergei Golubchik <serg@xxxxxxxxxxxx>
From: Gordan Bobic <gordan@xxxxxxxxxx>
Date: Wed, 20 Mar 2013 12:51:50 +0000
Cc: maria-developers@xxxxxxxxxxxxxxxxxxx
In-reply-to: <20130320122028.GA4027@meddwl.fritz.box>
User-agent: Roundcube Webmail/0.4.2

On Wed, 20 Mar 2013 13:20:28 +0100, Sergei Golubchik<serg@xxxxxxxxxxxx> wrote:

Hi, Gordan!

On Mar 20, Gordan Bobic wrote:

> On Mar 20, Gordan Bobic wrote:
>>  I have a situation with MariaDB 10.0.0 where an INSERT INTO ...
>>  SELECT query that takes 3 seconds to run ends up stuck in the
>>  closing tables state for 15+ minutes while burning 100% of CPU.
>>
 Stack trace attached in a text file.
 Does this provide enough information for a worthwhile bug report?


Was this stack trace created when you had your problem with a stuck
query?

Yes. And the same query is still stuck in the same state 35 minuteslater.


I have attached a fresh stack trace.

Because it doesn't look like there's any thread stuck in the "closing
table" state.


In which case the state shown in "show full processlist" is incorrect.
Note that I think that helps.

There's one event running, a couple of idle connection threads, acoupleof dead connection thread (waiting in the thread cache forresurrection),
and internal service threads, aria, innodb, signal thread, etc.


The one event that is running must be it, then. I specifically disabled

everything else on the box (slaves, event schduler) to get a cleantrace.


Gordan

[Thread debugging using libthread_db enabled]
[New Thread 0x7fa4fc4b0700 (LWP 31206)]
[New Thread 0x7fa4f844d700 (LWP 31205)]
[New Thread 0x7fa4f824e700 (LWP 31204)]
[New Thread 0x7fa4f8404700 (LWP 31203)]
[New Thread 0x7fa4f8297700 (LWP 29012)]
[New Thread 0x7fa4f83bb700 (LWP 28473)]
[New Thread 0x7fa4f8372700 (LWP 26598)]
[New Thread 0x7fa4fc4f9700 (LWP 26160)]
[New Thread 0x7fa4fc542700 (LWP 26012)]
[New Thread 0x7fa4fc58b700 (LWP 25937)]
[New Thread 0x7fa1a35fe700 (LWP 25936)]
[New Thread 0x7fa1a3fff700 (LWP 25935)]
[New Thread 0x7fa1a8f3a700 (LWP 25934)]
[New Thread 0x7fa1a993b700 (LWP 25933)]
[New Thread 0x7fa1ac13f700 (LWP 25932)]
[New Thread 0x7fa1aa33c700 (LWP 25931)]
[New Thread 0x7fa1aad3d700 (LWP 25930)]
[New Thread 0x7fa1ab73e700 (LWP 25929)]
[New Thread 0x7fa1acb40700 (LWP 25925)]
[New Thread 0x7fa1ad541700 (LWP 25924)]
[New Thread 0x7fa1adf42700 (LWP 25923)]
[New Thread 0x7fa1ae943700 (LWP 25922)]
[New Thread 0x7fa1af344700 (LWP 25921)]
[New Thread 0x7fa1afd45700 (LWP 25920)]
[New Thread 0x7fa1b0746700 (LWP 25919)]
[New Thread 0x7fa1b1147700 (LWP 25918)]
[New Thread 0x7fa1b1b48700 (LWP 25917)]
[New Thread 0x7fa1b2549700 (LWP 25916)]
[New Thread 0x7fa1b2f4a700 (LWP 25915)]
[New Thread 0x7fa1b394b700 (LWP 25914)]
[New Thread 0x7fa1b434c700 (LWP 25913)]
[New Thread 0x7fa1b4d4d700 (LWP 25912)]
[New Thread 0x7fa1b574e700 (LWP 25911)]
[New Thread 0x7fa1b614f700 (LWP 25910)]
[New Thread 0x7fa1b6b50700 (LWP 25909)]
[New Thread 0x7fa1b7551700 (LWP 25908)]
[New Thread 0x7fa1b7f52700 (LWP 25907)]
[New Thread 0x7fa1b8953700 (LWP 25906)]
[New Thread 0x7fa1b9354700 (LWP 25905)]
[New Thread 0x7fa1b9d55700 (LWP 25904)]
[New Thread 0x7fa1ba756700 (LWP 25903)]
[New Thread 0x7fa1bb157700 (LWP 25902)]
[New Thread 0x7fa1bbb58700 (LWP 25901)]
[New Thread 0x7fa1bc559700 (LWP 25900)]
[New Thread 0x7fa1bcf5a700 (LWP 25899)]
[New Thread 0x7fa1bd95b700 (LWP 25898)]
[New Thread 0x7fa1be35c700 (LWP 25897)]
[New Thread 0x7fa1bed5d700 (LWP 25896)]
[New Thread 0x7fa1bf75e700 (LWP 25895)]
[New Thread 0x7fa1c015f700 (LWP 25894)]
[New Thread 0x7fa1c0b60700 (LWP 25893)]
[New Thread 0x7fa1c1561700 (LWP 25892)]
[New Thread 0x7fa4f9267700 (LWP 25891)]
0x00007fa4fa956a63 in poll () from /lib64/libc.so.6

Thread 54 (Thread 0x7fa4f9267700 (LWP 25891)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000088f4d7 in my_service_thread_sleep ()
#2  0x0000000000887a83 in ma_checkpoint_background ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 53 (Thread 0x7fa1c1561700 (LWP 25892)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 52 (Thread 0x7fa1c0b60700 (LWP 25893)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 51 (Thread 0x7fa1c015f700 (LWP 25894)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 50 (Thread 0x7fa1bf75e700 (LWP 25895)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 49 (Thread 0x7fa1bed5d700 (LWP 25896)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 48 (Thread 0x7fa1be35c700 (LWP 25897)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 47 (Thread 0x7fa1bd95b700 (LWP 25898)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 46 (Thread 0x7fa1bcf5a700 (LWP 25899)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 45 (Thread 0x7fa1bc559700 (LWP 25900)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 44 (Thread 0x7fa1bbb58700 (LWP 25901)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 43 (Thread 0x7fa1bb157700 (LWP 25902)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 42 (Thread 0x7fa1ba756700 (LWP 25903)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 41 (Thread 0x7fa1b9d55700 (LWP 25904)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 40 (Thread 0x7fa1b9354700 (LWP 25905)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 39 (Thread 0x7fa1b8953700 (LWP 25906)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 38 (Thread 0x7fa1b7f52700 (LWP 25907)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 37 (Thread 0x7fa1b7551700 (LWP 25908)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 36 (Thread 0x7fa1b6b50700 (LWP 25909)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 35 (Thread 0x7fa1b614f700 (LWP 25910)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 34 (Thread 0x7fa1b574e700 (LWP 25911)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 33 (Thread 0x7fa1b4d4d700 (LWP 25912)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 32 (Thread 0x7fa1b434c700 (LWP 25913)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 31 (Thread 0x7fa1b394b700 (LWP 25914)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7fa1b2f4a700 (LWP 25915)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7fa1b2549700 (LWP 25916)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7fa1b1b48700 (LWP 25917)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fa1b1147700 (LWP 25918)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fa1b0746700 (LWP 25919)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fa1afd45700 (LWP 25920)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fa1af344700 (LWP 25921)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7fa1ae943700 (LWP 25922)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fa1adf42700 (LWP 25923)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fa1ad541700 (LWP 25924)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fa1acb40700 (LWP 25925)):
#0  0x00007fa4fbfe3614 in ?? () from /lib64/libaio.so.1
#1  0x000000000096c8fd in os_aio_linux_handle(unsigned long, fil_node_struct**, void**, unsigned long*) ()
#2  0x0000000000a43147 in fil_aio_wait(unsigned long) ()
#3  0x00000000009c22f8 in io_handler_thread ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fa1ab73e700 (LWP 25929)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009590e4 in lock_wait_timeout_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fa1aad3d700 (LWP 25930)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009be2d0 in srv_error_monitor_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fa1aa33c700 (LWP 25931)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009c08c6 in srv_monitor_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fa1ac13f700 (LWP 25932)):
#0  0x00007fa4fa958d03 in select () from /lib64/libc.so.6
#1  0x0000000000971c0f in os_thread_sleep(unsigned long) ()
#2  0x00000000009c1b64 in srv_master_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fa1a993b700 (LWP 25933)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970bfb in os_event_wait_low(os_event_struct*, long) ()
#2  0x00000000009c1668 in srv_purge_coordinator_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fa1a8f3a700 (LWP 25934)):
#0  0x00007fa4fa958d03 in select () from /lib64/libc.so.6
#1  0x0000000000971c0f in os_thread_sleep(unsigned long) ()
#2  0x0000000000a183db in buf_flush_page_cleaner_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fa1a3fff700 (LWP 25935)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970bfb in os_event_wait_low(os_event_struct*, long) ()
#2  0x0000000000a14f3e in buf_dump_thread ()
#3  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fa1a35fe700 (LWP 25936)):
#0  0x00007fa4fc1ef7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000970e13 in os_event_wait_time_low(os_event_struct*, unsigned long, long) ()
#2  0x00000000009e96e0 in ib_wqueue_timedwait(ib_wqueue_struct*, long) ()
#3  0x0000000000a699e4 in fts_optimize_thread(void*) ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fa4fc58b700 (LWP 25937)):
#0  0x00007fa4fc1f32a5 in sigwait () from /lib64/libpthread.so.0
#1  0x00000000004fd64b in signal_hand ()
#2  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fa4fc542700 (LWP 26012)):
#0  0x00007fa4fc1ef43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000004ffaaf in one_thread_per_connection_end(THD*, bool) ()
#2  0x0000000000639475 in do_handle_one_connection(THD*) ()
#3  0x0000000000639602 in handle_one_connection ()
#4  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fa4fc4f9700 (LWP 26160)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fa4f8372700 (LWP 26598)):
#0  0x000000000082cb0c in heap_write ()
#1  0x00000000008281ac in ha_heap::write_row(unsigned char*) ()
#2  0x00000000005d2c5f in schema_table_store_record(THD*, TABLE*) ()
#3  0x00000000005df262 in show_status_array(THD*, char const*, st_mysql_show_var*, enum_var_type, system_status_var*, char const*, TABLE*, bool, Item*) ()
#4  0x00000000005e0910 in fill_status(THD*, TABLE_LIST*, Item*) ()
#5  0x00000000005d0a1d in get_schema_tables_result(JOIN*, enum_schema_table_state) ()
#6  0x00000000005cbe3d in JOIN::exec_inner() ()
#7  0x00000000005cdb4a in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#8  0x00000000005ce81d in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#9  0x000000000057a32c in execute_sqlcom_select(THD*, TABLE_LIST*) ()
#10 0x0000000000581a31 in mysql_execute_command(THD*) ()
#11 0x0000000000583611 in mysql_parse(THD*, char*, unsigned int, Parser_state*) ()
#12 0x0000000000584ab6 in dispatch_command(enum_server_command, THD*, char*, unsigned int) ()
#13 0x00000000006392de in do_handle_one_connection(THD*) ()
#14 0x0000000000639602 in handle_one_connection ()
#15 0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#16 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fa4f83bb700 (LWP 28473)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fa4f8297700 (LWP 29012)):
#0  0x00000000006e9f53 in Arg_comparator::compare_int_signed_unsigned() ()
#1  0x00000000006f1712 in Item_func_eq::val_int() ()
#2  0x000000000065dfcb in JOIN_CACHE::generate_full_extensions(unsigned char*) ()
#3  0x000000000065e52c in JOIN_CACHE::join_matching_records(bool) ()
#4  0x000000000065b398 in JOIN_CACHE::join_records(bool) ()
#5  0x000000000065b3d4 in JOIN_CACHE::join_records(bool) ()
#6  0x00000000005a656f in evaluate_join_record(JOIN*, st_join_table*, int) ()
#7  0x00000000005a68a5 in sub_select(JOIN*, st_join_table*, bool) ()
#8  0x00000000005acdbd in do_select(JOIN*, List<Item>*, TABLE*, Procedure*) ()
#9  0x00000000005cbc7a in JOIN::exec_inner() ()
#10 0x00000000005cdb4a in mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) ()
#11 0x00000000005ce81d in handle_select(THD*, LEX*, select_result*, unsigned long) ()
#12 0x0000000000582b98 in mysql_execute_command(THD*) ()
#13 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#14 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#15 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#16 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#17 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#18 0x0000000000580839 in mysql_execute_command(THD*) ()
#19 0x000000000058fe42 in Prepared_statement::execute(String*, bool) ()
#20 0x00000000005942d7 in Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*) ()
#21 0x000000000059477b in mysql_sql_stmt_execute(THD*) ()
#22 0x000000000057fc01 in mysql_execute_command(THD*) ()
#23 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#24 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#25 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#26 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#27 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#28 0x0000000000580839 in mysql_execute_command(THD*) ()
#29 0x00000000007bf38e in sp_instr_stmt::exec_core(THD*, unsigned int*) ()
#30 0x00000000007c4300 in sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*) ()
#31 0x00000000007c4804 in sp_instr_stmt::execute(THD*, unsigned int*) ()
#32 0x00000000007c1fb7 in sp_head::execute(THD*, bool) ()
#33 0x00000000007c2fb7 in sp_head::execute_procedure(THD*, List<Item>*) ()
#34 0x0000000000622ddd in Event_job_data::execute(THD*, bool) ()
#35 0x00000000007f9d86 in Event_worker_thread::run(THD*, Event_queue_element_for_exec*) ()
#36 0x00000000007f9e21 in event_worker_thread ()
#37 0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#38 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fa4f8404700 (LWP 31203)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fa4f824e700 (LWP 31204)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fa4f844d700 (LWP 31205)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fa4fc4b0700 (LWP 31206)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x0000000000adece1 in vio_io_wait ()
#2  0x0000000000adedf3 in vio_socket_io_wait ()
#3  0x0000000000adf7a8 in vio_read ()
#4  0x00000000005066e3 in my_real_read(st_net*, unsigned long*) ()
#5  0x0000000000506b4c in my_net_read ()
#6  0x0000000000585179 in do_command(THD*) ()
#7  0x00000000006392de in do_handle_one_connection(THD*) ()
#8  0x0000000000639602 in handle_one_connection ()
#9  0x00007fa4fc1eb851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fa4fa96011d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fa4fc60e7e0 (LWP 25889)):
#0  0x00007fa4fa956a63 in poll () from /lib64/libc.so.6
#1  0x00000000004fe074 in handle_connections_sockets() ()
#2  0x00000000005052f2 in mysqld_main(int, char**) ()
#3  0x00007fa4fa896cdd in __libc_start_main () from /lib64/libc.so.6
#4  0x00000000004f87d9 in _start ()

Follow ups

Re: Query Stuck in "closing table" State
From: Gordan Bobic, 2013-03-20

References

Query Stuck in "closing table" State
From: Gordan Bobic, 2013-03-20
Re: Query Stuck in "closing table" State
From: Sergei Golubchik, 2013-03-20
Re: Query Stuck in "closing table" State
From: Gordan Bobic, 2013-03-20
Re: Query Stuck in "closing table" State
From: Sergei Golubchik, 2013-03-20