enterprise-support team mailing list archive

[Laurynas Biveinis <laurynas.biveinis@xxxxxxxxx>]

Public bug reported:

Copy of http://bugs.mysql.com/bug.php?id=82886:

[7 Sep 10:44] Laurynas Biveinis
Description:
glibc has a bug where a short-lived detached thread, that manages to complete before the caller thread finishes executing pthread_create, may crash server. In MySQL, at least the InnoDB full text parallel merge threads might be short-lived enough to hit this.

The glibc bug is https://sourceware.org/bugzilla/show_bug.cgi?id=20116
(https://sourceware.org/bugzilla/show_bug.cgi?id=19951 could be related
too)

How to repeat:
Make several copies of innodb.innodb-alter testcase (move out innodb-alter-kill etc out of the way for faster repro, or just write a proper --do-test regexp). Then

$ ./mtr --debug-server --parallel=8 --do-test=innodb-alter --repeat=9000
(...)
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
62	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fd4802f6700 (LWP 2706))]
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x00000000008f2f47 in my_write_core (sig=sig@entry=11) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/mysys/stacktrace.c:422
#2  0x000000000066ca0c in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/signal_handler.cc:236
#3  <signal handler called>
#4  __pthread_create_2_1 (newthread=newthread@entry=0x7fd4802f2268, attr=attr@entry=0x7fd4802f2270, start_routine=start_routine@entry=0x9e1350 <fts_parallel_merge(void*)>, arg=arg@entry=0x7fd4580dbe38) at pthread_create.c:713
#5  0x00000000009bb4b6 in os_thread_create_func (func=func@entry=0x9e1350 <fts_parallel_merge(void*)>, arg=arg@entry=0x7fd4580dbe38, thread_id=thread_id@entry=0x7fd4802f22e0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/os/os0thread.cc:193
#6  0x00000000009dfb04 in row_fts_start_parallel_merge (merge_info=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/row/row0ftsort.cc:984
#7  0x00000000009f7806 in row_merge_build_indexes (trx=0x7fd458089638, old_table=0x7fd458072798, new_table=0x7fd458072798, online=false, indexes=0x7fd4580d77a0, key_numbers=0x7fd4580d77a8, n_indexes=1, table=0x7fd4580b7700, add_cols=0x0, col_map=0x0, add_autoinc=18446744073709551615, sequence=...) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/row/row0merge.cc:3742
#8  0x000000000096b673 in ha_innobase::inplace_alter_table (this=0x7fd458042680, altered_table=0x7fd4580b7700, ha_alter_info=0x7fd4802f2630) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/handler/handler0alter.cc:3967
#9  0x0000000000747cad in handler::ha_inplace_alter_table (ha_alter_info=0x7fd4802f2630, altered_table=0x7fd4580b7700, this=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/handler.h:2974
#10 mysql_inplace_alter_table (target_mdl_request=0x7fd4802f2710, alter_ctx=0x7fd4802f2e20, inplace_supported=HA_ALTER_INPLACE_SHARED_LOCK_AFTER_PREPARE, ha_alter_info=0x7fd4802f2630, altered_table=0x7fd4580b7700, table=0x7fd4580c62a0, table_list=0x7fd458031ab0, thd=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_table.cc:6850
#11 mysql_alter_table (thd=thd@entry=0x1931e90, new_db=<optimized out>, new_name=<optimized out>, create_info=create_info@entry=0x7fd4802f3fa0, table_list=table_list@entry=0x7fd458031ab0, alter_info=alter_info@entry=0x7fd4802f3ee0, order_num=0, order=0x0, ignore=false) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_table.cc:8904
#12 0x000000000082954e in Sql_cmd_alter_table::execute (this=<optimized out>, thd=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_alter.cc:317
#13 0x00000000006f1cce in mysql_execute_command (thd=thd@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:5498
#14 0x00000000006f5da8 in mysql_parse (thd=thd@entry=0x1931e90, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7fd4802f5610) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:6972
#15 0x00000000006f70b1 in dispatch_command (command=COM_QUERY, thd=0x1931e90, packet=0x1a0daa1 "ALTER TABLE t1n ADD FULLTEXT INDEX(ct)", packet_length=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:1441
#16 0x00000000006f8fb9 in do_command (thd=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:1053
#17 0x00000000006c1722 in do_handle_one_connection (thd_arg=thd_arg@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_connect.cc:1541
#18 0x00000000006c17c0 in handle_one_connection (arg=arg@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_connect.cc:1444
#19 0x0000000000b43856 in pfs_spawn_thread (arg=0x1984ed0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/perfschema/pfs.cc:1860
#20 0x00007fd4877256fa in start_thread (arg=0x7fd4802f6700) at pthread_create.c:333
#21 0x00007fd486bbab5d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

The crash happens in __pthread_create_2_1, after the thread has been
launched, in dereferencing  struct pthread *pd pointer. Strace will show
its containing memory area to be munmap'ped by the spawned thread which
has quit already.

Suggested fix:
Implement a workaround to pthread_join selected (only extremely short lived ones) server threads instead of pthread_detach'ing them.

With a prototype patch to do so I'm unable to reproduce the crash
anymore.

** Affects: mysql-server
     Importance: Unknown
         Status: Unknown

** Affects: percona-server
     Importance: High
     Assignee: Laurynas Biveinis (laurynas-biveinis)
         Status: Triaged

** Affects: percona-server/5.5
     Importance: Undecided
     Assignee: Laurynas Biveinis (laurynas-biveinis)
         Status: Invalid

** Affects: percona-server/5.6
     Importance: High
     Assignee: Laurynas Biveinis (laurynas-biveinis)
         Status: In Progress

** Affects: percona-server/5.7
     Importance: High
     Assignee: Laurynas Biveinis (laurynas-biveinis)
         Status: Triaged


** Tags: ci innodb upstream

** Also affects: percona-server/5.5
   Importance: Undecided
       Status: New

** Also affects: percona-server/5.6
   Importance: Undecided
       Status: New

** Also affects: percona-server/5.7
   Importance: Undecided
       Status: New

** Changed in: percona-server/5.5
     Assignee: (unassigned) => Laurynas Biveinis (laurynas-biveinis)

** Changed in: percona-server/5.7
     Assignee: (unassigned) => Laurynas Biveinis (laurynas-biveinis)

** Changed in: percona-server/5.5
       Status: New => Invalid

** Changed in: percona-server/5.6
   Importance: Undecided => High

** Changed in: percona-server/5.6
     Assignee: (unassigned) => Laurynas Biveinis (laurynas-biveinis)

** Changed in: percona-server/5.6
       Status: New => In Progress

** Changed in: percona-server/5.7
   Importance: Undecided => High

** Changed in: percona-server/5.7
       Status: New => Triaged

** Tags added: ci innodb upstream

** Bug watch added: MySQL Bug System #82886
   http://bugs.mysql.com/bug.php?id=82886

** Also affects: mysql-server via
   http://bugs.mysql.com/bug.php?id=82886
   Importance: Unknown
       Status: Unknown

-- 
You received this bug notification because you are a member of Ubuntu
Server/Client Support Team, which is subscribed to MySQL.
Matching subscriptions: Ubuntu Server/Client Support Team
https://bugs.launchpad.net/bugs/1621012

Title:
  Server may crash due to a glibc bug in handling short-lived detached
  threads

To manage notifications about this bug go to:
https://bugs.launchpad.net/mysql-server/+bug/1621012/+subscriptions