maria-discuss team mailing list archive

Thread
Date

Re: MariaDB-10.2.11 Long Time To Start

To: Marko Mäkelä <marko.makela@xxxxxxxxxxx>
From: Michael Caplan <michael@xxxxxxxxxxx>
Date: Mon, 8 Jan 2018 12:58:24 -0400
Cc: Maria Discuss <maria-discuss@xxxxxxxxxxxxxxxxxxx>
In-reply-to: <CAJuX1hyD_r8MLybtZ+gVRCcL50rEgJ-+7MPYMJLTjgMupShjRQ@mail.gmail.com>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0

Here you go!

Thanks,

Mike


On 2018-01-08 12:38 PM, Marko Mäkelä wrote:

Hi Michael,

OK, all the interesting functions (InnoDB functions) are displayed as ??.
In the "perf top" it would have helped to drill into the mysqldexecutable, and only show its top functions. Apparently a lot of pagesare being read; that is all that I can tell from the screenshots.
Can you try the following:

1. Find the proces ID, e.g., ps ax|grep mysqld
2. gdb /usr/sbin/mysqld
set log file /tmp/gdb.txt
set log on
attach [pid]
thr a a bt
detach
quit
With this we should get detailed stack traces. I am interested in thethread whose stack trace includes plugin_init and mysqld_main.
Could you please try again?

Marko

Attaching to program: /usr/sbin/mysqld, process 31062
[New LWP 31067]
[New LWP 31068]
[New LWP 31069]
[New LWP 31070]
[New LWP 31071]
[New LWP 31072]
[New LWP 31073]
[New LWP 31074]
[New LWP 31075]
[New LWP 31076]
[New LWP 31077]
[New LWP 31078]
[New LWP 31079]
[New LWP 31080]
[New LWP 31081]
[New LWP 31082]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f1fd9bacd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.

Thread 17 (Thread 0x7f1156801700 (LWP 31082)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fd6e7bd826 in ?? ()
#2  0x000055fd6e8fc127 in ?? ()
#3  0x00007f1fd9ba36ba in start_thread (arg=0x7f1156801700) at pthread_create.c:333
#4  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 16 (Thread 0x7f1157002700 (LWP 31081)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fd6e7bd826 in ?? ()
#2  0x000055fd6e8fc127 in ?? ()
#3  0x00007f1fd9ba36ba in start_thread (arg=0x7f1157002700) at pthread_create.c:333
#4  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 15 (Thread 0x7f1157803700 (LWP 31080)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fd6e7bd826 in ?? ()
#2  0x000055fd6e8fc127 in ?? ()
#3  0x00007f1fd9ba36ba in start_thread (arg=0x7f1157803700) at pthread_create.c:333
#4  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 14 (Thread 0x7f1158004700 (LWP 31079)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x000055fd6e7bd826 in ?? ()
#2  0x000055fd6e8fcb0a in ?? ()
#3  0x00007f1fd9ba36ba in start_thread (arg=0x7f1158004700) at pthread_create.c:333
#4  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 13 (Thread 0x7f1158805700 (LWP 31078)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f1158805700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 12 (Thread 0x7f1159006700 (LWP 31077)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f1159006700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 11 (Thread 0x7f1159807700 (LWP 31076)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f1159807700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 10 (Thread 0x7f115a008700 (LWP 31075)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115a008700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 9 (Thread 0x7f115a809700 (LWP 31074)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115a809700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f115b00a700 (LWP 31073)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115b00a700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f115b80b700 (LWP 31072)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115b80b700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f115c00c700 (LWP 31071)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115c00c700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7f115c80d700 (LWP 31070)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115c80d700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f115d00e700 (LWP 31069)):
#0  0x00007f1fdacc764a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x000055fd6e7b6c48 in ?? ()
#2  0x000055fd6e7b7192 in ?? ()
#3  0x000055fd6e7bc0b9 in ?? ()
#4  0x000055fd6e9508a5 in ?? ()
#5  0x000055fd6e853bc8 in ?? ()
#6  0x00007f1fd9ba36ba in start_thread (arg=0x7f115d00e700) at pthread_create.c:333
#7  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f1fc7c10700 (LWP 31068)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055fd6e6d99ff in ?? ()
#2  0x000055fd6e6d14c7 in ?? ()
#3  0x00007f1fd9ba36ba in start_thread (arg=0x7f1fc7c10700) at pthread_create.c:333
#4  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f1fd693e700 (LWP 31067)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x000055fd6ea610f2 in ?? ()
#2  0x00007f1fd9ba36ba in start_thread (arg=0x7f1fd693e700) at pthread_create.c:333
#3  0x00007f1fd924e3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f1fdb26c8c0 (LWP 31062)):
#0  0x00007f1fd9bacd43 in pread64 () at ../sysdeps/unix/syscall-template.S:84
#1  0x000055fd6e7b7595 in ?? ()
#2  0x000055fd6e7b8140 in ?? ()
#3  0x000055fd6e7b83af in ?? ()
#4  0x000055fd6e970d7e in ?? ()
#5  0x000055fd6e9715a5 in ?? ()
#6  0x000055fd6e95391b in ?? ()
#7  0x000055fd6e9335c9 in ?? ()
#8  0x000055fd6e859376 in ?? ()
#9  0x000055fd6e73afb1 in ?? ()
#10 0x000055fd6e4f98c4 in ha_initialize_handlerton(st_plugin_int*) ()
#11 0x000055fd6e36e175 in ?? ()
#12 0x000055fd6e36eb4a in plugin_init(int*, char**, int) ()
#13 0x000055fd6e2ca200 in ?? ()
#14 0x000055fd6e2cdde7 in mysqld_main(int, char**) ()
#15 0x00007f1fd9167830 in __libc_start_main (main=0x55fd6e2acf00 <main>, argc=1, argv=0x7fff63d32608, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff63d325f8)
    at ../csu/libc-start.c:291
#16 0x000055fd6e2c1749 in _start ()
Detaching from program: /usr/sbin/mysqld, process 31062

Follow ups

Re: MariaDB-10.2.11 Long Time To Start
From: Marko Mäkelä, 2018-01-09

References

MariaDB-10.2.11 Long Time To Start
From: Michael Caplan, 2018-01-08
Re: MariaDB-10.2.11 Long Time To Start
From: Marko Mäkelä, 2018-01-08
Re: MariaDB-10.2.11 Long Time To Start
From: Michael Caplan, 2018-01-08
Re: MariaDB-10.2.11 Long Time To Start
From: Marko Mäkelä, 2018-01-08