maria-discuss team mailing list archive

[Tom Ryan <tom0ryan@xxxxxxxxx>]

On Fri, Sep 6, 2019 at 5:56 PM Sergei Golubchik <serg@xxxxxxxxxxx> wrote:

> Hi, Tom!
>
> On Sep 06, Tom Ryan wrote:
> >
> > Sep 6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8
> thread_stack 0x49000
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /usr/sbin/mysqld(+0x566623)[0x5644f6973623]
> > Sep 6 14:37:59 node-2 mysqld[1846]:
> /usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]
>
> This looks truncated. Could you post a complete stack trace please?
>

Sep  6 14:37:59 node-2 mysqld[1846]: 190906 14:37:59 [ERROR] mysqld got
signal 11 ;

Sep  6 14:37:59 node-2 mysqld[1846]: This could be because you hit a bug.
It is also possible that this binary

Sep  6 14:37:59 node-2 mysqld[1846]: or one of the libraries it was linked
against is corrupt, improperly built,

Sep  6 14:37:59 node-2 mysqld[1846]: or misconfigured. This error can also
be caused by malfunctioning hardware.

Sep  6 14:37:59 node-2 mysqld[1846]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs

Sep  6 14:37:59 node-2 mysqld[1846]: We will try our best to scrape up some
info that will hopefully help

Sep  6 14:37:59 node-2 mysqld[1846]: diagnose the problem, but since we
have already crashed,

Sep  6 14:37:59 node-2 mysqld[1846]: something is definitely wrong and this
may fail.

Sep  6 14:37:59 node-2 mysqld[1846]: Server version:
10.4.7-MariaDB-1:10.4.7+maria~xenial-log

Sep  6 14:37:59 node-2 mysqld[1846]: key_buffer_size=134217728

Sep  6 14:37:59 node-2 mysqld[1846]: read_buffer_size=2097152

Sep  6 14:37:59 node-2 mysqld[1846]: max_used_connections=30

Sep  6 14:37:59 node-2 mysqld[1846]: max_threads=4002

Sep  6 14:37:59 node-2 mysqld[1846]: thread_count=45

Sep  6 14:37:59 node-2 mysqld[1846]: It is possible that mysqld could use
up to

Sep  6 14:37:59 node-2 mysqld[1846]: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 24817728 K  bytes of memory

Sep  6 14:37:59 node-2 mysqld[1846]: Hope that's ok; if not, decrease some
variables in the equation.

Sep  6 14:37:59 node-2 mysqld[1846]: Thread pointer: 0x7f32052a0348

Sep  6 14:37:59 node-2 mysqld[1846]: Attempting backtrace. You can use the
following information to find out

Sep  6 14:37:59 node-2 mysqld[1846]: where mysqld died. If you see no
messages after this, something went

Sep  6 14:37:59 node-2 mysqld[1846]: terribly wrong...

Sep  6 14:37:59 node-2 mysqld[1846]: stack_bottom = 0x7f3271446dd8
thread_stack 0x49000

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5644f71dcf4e]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(handle_fatal_signal+0x307)[0x5644f6c70b07]

Sep  6 14:37:59 node-2 mysqld[1846]:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f33e141d890]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(handler_index_cond_check+0x8d)[0x5644f6c7b1fd]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0x566623)[0x5644f6973623]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0xbb4ba8)[0x5644f6fc1ba8]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0xad1cfa)[0x5644f6edecfa]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0xad1fa3)[0x5644f6edefa3]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_ZN7handler11ha_rnd_nextEPh+0x12f)[0x5644f6c75acf]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z13rr_sequentialP11READ_RECORD+0x22)[0x5644f6d988e2]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x164)[0x5644f6a97034]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xc9f)[0x5644f6ab85af]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x5644f6ab8833]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x5644

f6ab6b26]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x5644f6ab7456]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0x55756d)[0x5644f696456d]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7371)[0x5644f6a61741]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x5644f6a632da]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(+0x656b59)[0x5644f6a63b59]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x2633)[0x5644f6a66ae3]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5644f6a670ac]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x5644f6b43691]

Sep  6 14:37:59 node-2 mysqld[1846]:
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x5644f6b437bd]

Sep  6 14:37:59 node-2 mysqld[1846]:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f33e14126db]

Sep  6 14:37:59 node-2 mysqld[1846]:
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f33e081088f]

Sep  6 14:37:59 node-2 mysqld[1846]: Trying to get some variables.

Sep  6 14:37:59 node-2 mysqld[1846]: Some pointers may be invalid and cause
the dump to abort.

Sep  6 14:37:59 node-2 mysqld[1846]: Query (0x7f3205209000): SELECT
field_data_field_weight.field_weight_value AS
field_data_field_weight_field_weight_value, node.nid AS nid,
DATE_FORMAT((DATE_A

DD('19700101', INTERVAL node.created SECOND) + INTERVAL -14400 SECOND),
'%Y%m') AS node_created_month FROM  node node LEFT JOIN
field_data_field_weight field_data_field_weight ON node.nid = fiel

d_data_field_weight.entity_id AND (field_data_field_weight.entity_type =
'node' AND field_data_field_weight.deleted = '0') WHERE (( (node.status = 1
OR (node.uid = 0 AND 0 <> 0 AND 0 = 1) OR 0 =

 1) AND (node.nid IN  (SELECT tn.nid AS nid FROM  taxonomy_index tn WHERE (
(tn.tid = '3') ))) ))AND ( EXISTS  (SELECT na.nid AS nid FROM  node_access
na WHERE (( (na.gid = '0') AND (na.realm =

'all') )OR( (na.gid = '0') AND (na.realm = 'content_access_author') )OR(
(na.gid = '1') AND (na.realm = 'content_access_rid') ))AND (na.grant_view
>= '1') AND (node.nid = na.nid) ))  ORDER BY no

de_created_month DESC, field_data_field_weight_field_weight_value DESC
LIMIT 15 OFFSET 0

Sep  6 14:37:59 node-2 mysqld[1846]: Connection ID (thread ID): 15972

Sep  6 14:37:59 node-2 mysqld[1846]: Status: NOT_KILLED

Sep  6 14:37:59 node-2 mysqld[1846]: Optimizer switch:
index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio

n_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_ma

tch_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,

optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subqu

ery=on,rowid_filter=on,condition_pushdown_from_having=on

Sep  6 14:37:59 node-2 mysqld[1846]: The manual page at
http://dev.mysql.com/doc/mysql/en/crashing.html contains

Sep  6 14:37:59 node-2 mysqld[1846]: information that should help you find
out what is causing the crash.

Sep  6 14:37:59 node-2 mysqld[1846]: Writing a core file...

Sep  6 14:37:59 node-2 mysqld[1846]: Working directory at /var/lib/mysql

Sep  6 14:37:59 node-2 mysqld[1846]: Resource Limits:

Sep  6 14:37:59 node-2 mysqld[1846]: Limit                     Soft Limit
        Hard Limit           Units

Sep  6 14:37:59 node-2 mysqld[1846]: Max cpu time              unlimited
        unlimited            seconds

Sep  6 14:37:59 node-2 mysqld[1846]: Max file size             unlimited
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max data size             unlimited
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max stack size            8388608
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max core file size        0
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max resident set          unlimited
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max processes             128431
        128431               processes

Sep  6 14:37:59 node-2 mysqld[1846]: Max open files            16364
        16364                files

Sep  6 14:37:59 node-2 mysqld[1846]: Max locked memory         16777216
        16777216             bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max address space         unlimited
        unlimited            bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max file locks            unlimited
        unlimited            locks

Sep  6 14:37:59 node-2 mysqld[1846]: Max pending signals       128431
        128431               signals

Sep  6 14:37:59 node-2 mysqld[1846]: Max msgqueue size         819200
        819200               bytes

Sep  6 14:37:59 node-2 mysqld[1846]: Max nice priority         0
        0

Sep  6 14:37:59 node-2 mysqld[1846]: Max realtime priority     0
        0

Sep  6 14:37:59 node-2 mysqld[1846]: Max realtime timeout      unlimited
        unlimited            us

Sep  6 14:37:59 node-2 mysqld[1846]: Core pattern:
|/usr/share/apport/apport %p %s %c %d %P

Sep  6 14:37:59 node-2 kernel: [31368.222207] show_signal_msg: 7 callbacks
suppressed

Sep  6 14:37:59 node-2 kernel: [31368.222210] mysqld[3138]: segfault at 0
ip 00005644f6c7b1fd sp 00007f32714417b0 error 4

here's one from this morning

Sep  7 05:12:07 node-1 mysqld[18950]: This could be because you hit a bug.
It is also possible that this binary

Sep  7 05:12:07 node-1 mysqld[18950]: or one of the libraries it was linked
against is corrupt, improperly built,

Sep  7 05:12:07 node-1 mysqld[18950]: or misconfigured. This error can also
be caused by malfunctioning hardware.

Sep  7 05:12:07 node-1 mysqld[18950]: To report this bug, see
https://mariadb.com/kb/en/reporting-bugs

Sep  7 05:12:07 node-1 mysqld[18950]: We will try our best to scrape up
some info that will hopefully help

Sep  7 05:12:07 node-1 mysqld[18950]: diagnose the problem, but since we
have already crashed,

Sep  7 05:12:07 node-1 mysqld[18950]: something is definitely wrong and
this may fail.

Sep  7 05:12:07 node-1 mysqld[18950]: Server version:
10.4.7-MariaDB-1:10.4.7+maria~bionic-log

Sep  7 05:12:07 node-1 mysqld[18950]: key_buffer_size=134217728

Sep  7 05:12:07 node-1 mysqld[18950]: read_buffer_size=2097152

Sep  7 05:12:07 node-1 mysqld[18950]: max_used_connections=27

Sep  7 05:12:07 node-1 mysqld[18950]: max_threads=4002

Sep  7 05:12:07 node-1 mysqld[18950]: thread_count=42

Sep  7 05:12:07 node-1 mysqld[18950]: It is possible that mysqld could use
up to

Sep  7 05:12:07 node-1 mysqld[18950]: key_buffer_size + (read_buffer_size +
sort_buffer_size)*max_threads = 24817728 K  bytes of memory

Sep  7 05:12:07 node-1 mysqld[18950]: Hope that's ok; if not, decrease some
variables in the equation.

Sep  7 05:12:07 node-1 mysqld[18950]: Thread pointer: 0x7f1adc00b618

Sep  7 05:12:07 node-1 mysqld[18950]: Attempting backtrace. You can use the
following information to find out

Sep  7 05:12:07 node-1 mysqld[18950]: where mysqld died. If you see no
messages after this, something went

Sep  7 05:12:07 node-1 mysqld[18950]: terribly wrong...

Sep  7 05:12:07 node-1 mysqld[18950]: stack_bottom = 0x7f199fecadd8
thread_stack 0x49000

Sep  7 05:12:07 node-1 mysqld[18950]:
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a2e3675cee]

Sep  7 05:12:07 node-1 mysqld[18950]:
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55a2e30ed085]

Sep  7 05:12:07 node-1 mysqld[18950]:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f1b115d0890]

Sep  7 05:12:07 node-1 kernel: [84233.145921] mysqld[26579]: segfault at 0
ip 000055a2e30f7500 sp 00007f19905b9e10 error 4 in
mysqld[55a2e2896000+13d6000]

Sep  7 05:12:07 node-1 systemd[1]: mariadb.service: Main process exited,
code=killed, status=11/SEGV

Sep  7 05:12:07 node-1 systemd[1]: mariadb.service: Failed with result
'signal'.


>
> And please tell exactly what binaries you're using. Version,
> architecture, Linux distribution.
>
>
ubuntu 18.04
mariadb respository

/usr/sbin/mysqld: ELF 64-bit LSB shared object, x86-64, version 1
(GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0,
BuildID[sha1]=cdf04892e30322f0ee59a1aa604e6087ad040231, stripped





> > This is getting old and is unlike our usual mariadb experience.
> >
> > Looking at the jira for 10.4.8, it appears to fix many problems in
> > 10.4 and I'm hopeful it solves this.
> >
> > In the meantime, any ideas or suggestions for things to look for?
>
> 10.4.8 should be out any day now. But if I'd know more about the crash,
> I could tell whether it's fixed there or not.
>
> Regards,
> Sergei
>