← Back to team overview

maria-discuss team mailing list archive

Re: MariaDB-10.2.11 Long Time To Start

 

Hi Marko,

Thanks for your patience with me.  I think I have better material for you to look at now.

Best,

Mike


On 2018-01-09 03:17 AM, Marko Mäkelä wrote:
Michael,

Sorry, I now realize that you must likely have used a stripped executable, and therefore the symbols for the InnoDB functions are missing (displayed as ?? in the stack trace output from gdb). As far as I understand, gdb should be able to resolve the symbols if you install the separate debug symbol package:

sudo apt install mariadb-server-core-10.2-dbgsym

I tried to find the installation package corresponding to '10.2.11-MariaDB-10.2.11+maria~xenial-log' so that I would be able to resolve the stack trace manually. As a developer, normally I work with code that I build myself from the source code, and therefore I am a bit bad at giving advice that applies to the packaged code.

I downloaded and extracted the files from the following packages:
mariadb-server-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-10.2-dbgsym_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-10.2_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-10.2_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-core-10.2-dbgsym_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-core-10.2-dbgsym_10.2.11+maria%7Exenial_amd64.deb> mariadb-server-core-10.2_10.2.11+maria~xenial_amd64.deb <http://mirror.netinch.com/pub/mariadb//mariadb-10.2.11/repo/ubuntu/pool/main/m/mariadb-10.2/mariadb-server-core-10.2_10.2.11+maria%7Exenial_amd64.deb>

The server executable /usr/sbin/mysqld is included in the mysql-server-core package. Unfortunately I was unable to match that executable with the addresses from your gdb output. I did the following:

file mysqld
mysqld: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=7649cc717c0ee820185a633fd27692f3b9160bed, stripped

gdb mysqld
symbol-file ../lib/debug/.build-id/76/49cc717c0ee820185a633fd27692f3b9160bed.debug

followed by a number of gdb commands. I expected the least significant 12 bits of the addresses to match in the disassembly of ha_initialize_handlerton() and plugin_init(), but that was not the case. Maybe I did something wrong.

So, can you please install the debug symbol package and try again?

Best regards,

Marko

Attachment: Screenshot from 2018-01-09 10-23-46.png
Description: PNG image

Attachment: Screenshot from 2018-01-09 10-24-16.png
Description: PNG image

Attachment: Screenshot from 2018-01-09 10-24-35.png
Description: PNG image

Attachment: Screenshot from 2018-01-09 10-24-52.png
Description: PNG image

Attachment: Screenshot from 2018-01-09 10-28-03.png
Description: PNG image

Attaching to program: /usr/sbin/mysqld, process 2248
[New LWP 2250]
[New LWP 2251]
[New LWP 2252]
[New LWP 2253]
[New LWP 2254]
[New LWP 2255]
[New LWP 2256]
[New LWP 2257]
[New LWP 2258]
[New LWP 2259]
[New LWP 2260]
[New LWP 2261]
[New LWP 2262]
[New LWP 2263]
[New LWP 2264]
[New LWP 2265]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x0000558a37d6d377 in ut_fold_ulint_pair (n2=0, n1=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/include/ut0rnd.ic:189
189	/home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/include/ut0rnd.ic: No such file or directory.

Thread 17 (Thread 0x7f051f7fe700 (LWP 2265)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000558a37c3b826 in os_event::wait (this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:166
#2  os_event::wait_low (reset_sig_count=1, this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:336
#3  os_event_wait_low (event=0x558a8a6ee740, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:535
#4  0x0000558a37d7a127 in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0flu.cc:3518
#5  0x00007f13a6f576ba in start_thread (arg=0x7f051f7fe700) at pthread_create.c:333
#6  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 16 (Thread 0x7f051ffff700 (LWP 2264)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000558a37c3b826 in os_event::wait (this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:166
#2  os_event::wait_low (reset_sig_count=1, this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:336
#3  os_event_wait_low (event=0x558a8a6ee740, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:535
#4  0x0000558a37d7a127 in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0flu.cc:3518
#5  0x00007f13a6f576ba in start_thread (arg=0x7f051ffff700) at pthread_create.c:333
#6  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 15 (Thread 0x7f052481b700 (LWP 2263)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000558a37c3b826 in os_event::wait (this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:166
#2  os_event::wait_low (reset_sig_count=1, this=0x558a8a6ee740) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:336
#3  os_event_wait_low (event=0x558a8a6ee740, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:535
#4  0x0000558a37d7a127 in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0flu.cc:3518
#5  0x00007f13a6f576ba in start_thread (arg=0x7f052481b700) at pthread_create.c:333
#6  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 14 (Thread 0x7f052501c700 (LWP 2262)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000558a37c3b826 in os_event::wait (this=0x558a391df0e0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:166
#2  os_event::wait_low (reset_sig_count=1, this=0x558a391df0e0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:336
#3  os_event_wait_low (event=0x558a391df0e0, reset_sig_count=reset_sig_count@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0event.cc:535
#4  0x0000558a37d7ab0a in buf_flush_page_cleaner_coordinator () at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0flu.cc:3190
#5  0x00007f13a6f576ba in start_thread (arg=0x7f052501c700) at pthread_create.c:333
#6  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 13 (Thread 0x7f052581d700 (LWP 2261)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f052581cae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f052581cae0, m1=m1@entry=0x7f052581cb90, m2=m2@entry=0x7f052581cb98, request=request@entry=0x7f052581cba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f052581cba0, m2=0x7f052581cb98, m1=0x7f052581cb90, global_segment=9)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=9, m1=m1@entry=0x7f052581cb90, m2=m2@entry=0x7f052581cb98, request=request@entry=0x7f052581cba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=9) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f052581d700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 12 (Thread 0x7f052601e700 (LWP 2260)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f052601dae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f052601dae0, m1=m1@entry=0x7f052601db90, m2=m2@entry=0x7f052601db98, request=request@entry=0x7f052601dba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f052601dba0, m2=0x7f052601db98, m1=0x7f052601db90, global_segment=8)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=8, m1=m1@entry=0x7f052601db90, m2=m2@entry=0x7f052601db98, request=request@entry=0x7f052601dba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=8) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f052601e700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 11 (Thread 0x7f052681f700 (LWP 2259)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f052681eae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f052681eae0, m1=m1@entry=0x7f052681eb90, m2=m2@entry=0x7f052681eb98, request=request@entry=0x7f052681eba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f052681eba0, m2=0x7f052681eb98, m1=0x7f052681eb90, global_segment=7)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=7, m1=m1@entry=0x7f052681eb90, m2=m2@entry=0x7f052681eb98, request=request@entry=0x7f052681eba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=7) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f052681f700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 10 (Thread 0x7f0527020700 (LWP 2258)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f052701fae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f052701fae0, m1=m1@entry=0x7f052701fb90, m2=m2@entry=0x7f052701fb98, request=request@entry=0x7f052701fba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f052701fba0, m2=0x7f052701fb98, m1=0x7f052701fb90, global_segment=6)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=6, m1=m1@entry=0x7f052701fb90, m2=m2@entry=0x7f052701fb98, request=request@entry=0x7f052701fba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=6) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0527020700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 9 (Thread 0x7f0527821700 (LWP 2257)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f0527820ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f0527820ae0, m1=m1@entry=0x7f0527820b90, m2=m2@entry=0x7f0527820b98, request=request@entry=0x7f0527820ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f0527820ba0, m2=0x7f0527820b98, m1=0x7f0527820b90, global_segment=5)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=5, m1=m1@entry=0x7f0527820b90, m2=m2@entry=0x7f0527820b98, request=request@entry=0x7f0527820ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=5) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0527821700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 8 (Thread 0x7f0528022700 (LWP 2256)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f0528021ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f0528021ae0, m1=m1@entry=0x7f0528021b90, m2=m2@entry=0x7f0528021b98, request=request@entry=0x7f0528021ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f0528021ba0, m2=0x7f0528021b98, m1=0x7f0528021b90, global_segment=4)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=4, m1=m1@entry=0x7f0528021b90, m2=m2@entry=0x7f0528021b98, request=request@entry=0x7f0528021ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=4) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0528022700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 7 (Thread 0x7f0528823700 (LWP 2255)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f0528822ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f0528822ae0, m1=m1@entry=0x7f0528822b90, m2=m2@entry=0x7f0528822b98, request=request@entry=0x7f0528822ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f0528822ba0, m2=0x7f0528822b98, m1=0x7f0528822b90, global_segment=3)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=3, m1=m1@entry=0x7f0528822b90, m2=m2@entry=0x7f0528822b98, request=request@entry=0x7f0528822ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=3) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0528823700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 6 (Thread 0x7f0529024700 (LWP 2254)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f0529023ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f0529023ae0, m1=m1@entry=0x7f0529023b90, m2=m2@entry=0x7f0529023b98, request=request@entry=0x7f0529023ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f0529023ba0, m2=0x7f0529023b98, m1=0x7f0529023b90, global_segment=2)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=2, m1=m1@entry=0x7f0529023b90, m2=m2@entry=0x7f0529023b98, request=request@entry=0x7f0529023ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=2) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0529024700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 5 (Thread 0x7f0529825700 (LWP 2253)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f0529824ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f0529824ae0, m1=m1@entry=0x7f0529824b90, m2=m2@entry=0x7f0529824b98, request=request@entry=0x7f0529824ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f0529824ba0, m2=0x7f0529824b98, m1=0x7f0529824b90, global_segment=1)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=1, m1=m1@entry=0x7f0529824b90, m2=m2@entry=0x7f0529824b98, request=request@entry=0x7f0529824ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=1) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f0529825700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 4 (Thread 0x7f052a026700 (LWP 2252)):
#0  0x00007f13a807b64a in ?? () from target:/lib/x86_64-linux-gnu/libaio.so.1
#1  0x0000558a37c34c48 in LinuxAIOHandler::collect (this=this@entry=0x7f052a025ae0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:1893
#2  0x0000558a37c35192 in LinuxAIOHandler::poll (this=this@entry=0x7f052a025ae0, m1=m1@entry=0x7f052a025b90, m2=m2@entry=0x7f052a025b98, request=request@entry=0x7f052a025ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2038
#3  0x0000558a37c3a0b9 in os_aio_linux_handler (request=0x7f052a025ba0, m2=0x7f052a025b98, m1=0x7f052a025b90, global_segment=0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:2092
#4  os_aio_handler (segment=segment@entry=0, m1=m1@entry=0x7f052a025b90, m2=m2@entry=0x7f052a025b98, request=request@entry=0x7f052a025ba0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/os/os0file.cc:5720
#5  0x0000558a37dce8a5 in fil_aio_wait (segment=segment@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:5317
#6  0x0000558a37cd1bc8 in io_handler_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:343
#7  0x00007f13a6f576ba in start_thread (arg=0x7f052a026700) at pthread_create.c:333
#8  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 3 (Thread 0x7f13a0bb6700 (LWP 2251)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000558a37b579ff in inline_mysql_cond_timedwait (src_file=0x558a37ffb6b8 "/home/buildbot/buildbot/build/mariadb-10.2.12/storage/maria/ma_servicethread.c", src_line=116, abstime=0x7f13a0bb5c80, 
    mutex=0x558a38fb9760 <LOCK_checkpoint>, that=0x558a38fb9720 <COND_checkpoint>) at /home/buildbot/buildbot/build/mariadb-10.2.12/include/mysql/psi/mysql_thread.h:1186
#2  my_service_thread_sleep (control=control@entry=0x558a386fd6a0 <checkpoint_control>, sleep_time=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/maria/ma_servicethread.c:115
#3  0x0000558a37b4f4c7 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/maria/ma_checkpoint.c:709
#4  0x00007f13a6f576ba in start_thread (arg=0x7f13a0bb6700) at pthread_create.c:333
#5  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 2 (Thread 0x7f13a3cf2700 (LWP 2250)):
#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1  0x0000558a37edf0f2 in inline_mysql_cond_timedwait (that=0x558a39012b60 <COND_timer>, mutex=0x558a39012ba0 <LOCK_timer>, 
    src_file=0x558a38043b88 "/home/buildbot/buildbot/build/mariadb-10.2.12/mysys/thr_timer.c", src_line=292, abstime=0x7f13a3cf1d10)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/include/mysql/psi/mysql_thread.h:1186
#2  timer_handler (arg=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/mysys/thr_timer.c:292
#3  0x00007f13a6f576ba in start_thread (arg=0x7f13a3cf2700) at pthread_create.c:333
#4  0x00007f13a66023dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread 1 (Thread 0x7f13a86208c0 (LWP 2248)):
#0  0x0000558a37d6d377 in ut_fold_ulint_pair (n2=0, n1=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/include/ut0rnd.ic:189
#1  ut_fold_binary (len=16338, str=0x7f0c9fa2dea6 "") at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/include/ut0rnd.ic:215
#2  buf_calc_page_new_checksum (page=page@entry=0x7f0c9fa2c000 "\204H*�") at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0checksum.cc:105
#3  0x0000558a37d5b223 in buf_page_is_checksum_valid_innodb (read_buf=read_buf@entry=0x7f0c9fa2c000 "\204H*�", checksum_field1=checksum_field1@entry=2219322075, checksum_field2=checksum_field2@entry=1026603750)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0buf.cc:703
#4  0x0000558a37d5cff6 in buf_page_is_corrupted (check_lsn=check_lsn@entry=true, read_buf=read_buf@entry=0x7f0c9fa2c000 "\204H*�", page_size=..., space=space@entry=0x558a8d7a9250)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0buf.cc:1027
#5  0x0000558a37d608e4 in buf_page_check_corrupt (space=0x558a8d7a9250, bpage=0x7f0c997fd108) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0buf.cc:5834
#6  buf_page_io_complete (bpage=bpage@entry=0x7f0c997fd108, evict=evict@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0buf.cc:5974
#7  0x0000558a37d89b1a in buf_read_page_low (ignore_missing_space=false, unzip=false, page_size=..., page_id=..., mode=132, type=0, sync=true, err=0x7ffc00778ae0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0rea.cc:225
#8  buf_read_page (page_id=..., page_size=...) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0rea.cc:460
#9  0x0000558a37d62178 in buf_page_get_gen (page_id=..., page_size=..., rw_latch=rw_latch@entry=2, guess=<optimized out>, guess@entry=0x0, mode=mode@entry=10, 
    file=file@entry=0x558a380319b0 "/home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc", line=4626, mtr=0x7ffc00779150, err=0x0)
    at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/buf/buf0buf.cc:4320
#10 0x0000558a37dd1274 in fsp_flags_try_adjust (space_id=space_id@entry=1233316, flags=flags@entry=0) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:4624
#11 0x0000558a37dd1c24 in fil_ibd_open (validate=<optimized out>, validate@entry=true, fix_dict=<optimized out>, purpose=purpose@entry=FIL_TYPE_TABLESPACE, id=1233316, flags=<optimized out>, 
    space_name=<optimized out>, path_in=0x558a8d79af60 "./ddx_lab_801002/pickups.ibd") at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/fil/fil0fil.cc:4211
#12 0x0000558a37db15c9 in dict_check_sys_tables (validate=true) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/dict/dict0load.cc:1482
#13 dict_check_tablespaces_and_store_max_id (validate=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/dict/dict0load.cc:1537
#14 0x0000558a37cd7376 in innobase_start_or_create_for_mysql () at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/srv/srv0start.cc:2524
#15 0x0000558a37bb8fb1 in innobase_init (p=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.2.12/storage/innobase/handler/ha_innodb.cc:4371
#16 0x0000558a379778c4 in ha_initialize_handlerton (plugin=0x558a39089ba0) at /home/buildbot/buildbot/build/mariadb-10.2.12/sql/handler.cc:520
#17 0x0000558a377ec175 in plugin_initialize (tmp_root=tmp_root@entry=0x7ffc00784770, plugin=plugin@entry=0x558a39089ba0, argc=argc@entry=0x558a3878ab98 <remaining_argc>, argv=argv@entry=0x558a39040828, 
    options_only=options_only@entry=false) at /home/buildbot/buildbot/build/mariadb-10.2.12/sql/sql_plugin.cc:1411
#18 0x0000558a377ecb4a in plugin_init (argc=argc@entry=0x558a3878ab98 <remaining_argc>, argv=0x558a39040828, flags=2) at /home/buildbot/buildbot/build/mariadb-10.2.12/sql/sql_plugin.cc:1692
#19 0x0000558a37748200 in init_server_components () at /home/buildbot/buildbot/build/mariadb-10.2.12/sql/mysqld.cc:5258
#20 0x0000558a3774bde7 in mysqld_main (argc=54, argv=0x558a39040828) at /home/buildbot/buildbot/build/mariadb-10.2.12/sql/mysqld.cc:5850
#21 0x00007f13a651b830 in __libc_start_main (main=0x558a3772af00 <main(int, char**)>, argc=1, argv=0x7ffc00785368, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7ffc00785358) at ../csu/libc-start.c:291
#22 0x0000558a3773f749 in _start ()
Detaching from program: /usr/sbin/mysqld, process 2248


Follow ups

References