← Back to team overview

maria-discuss team mailing list archive

Re: mariadb 10.0.17 keeps crashing

 

Hi,

No variables to tweak the tokudb memory allocation ?
By default tokudb_cache_size takes half of the total system memory.
With 6G allocated to InnoDB buffer pool, and 1G to the MyISAM key buffer, are you sure you have enough memory for TokuDB ?

  Jocelyn


Le 03/03/2015 17:37, Gabriel Sosa a écrit :
@joocelyn Here is the my.cfg file

@justin I will check that post. Thanks

[mysqld]

# -------------------------------------------------------------------------------
# ++++ General
# -------------------------------------------------------------------------------
datadir       = /var/lib/mysql/data
pid-file        = /var/run/mysqld/mysqld.pid
socket        = /var/lib/mysql/mysql.sock
tmpdir        = /var/lib/mysql/tmp
port        = 3306

# -------------------------------------------------------------------------------
# ++++ Logging
# -------------------------------------------------------------------------------
log-error       = /var/lib/mysql/log/mysqld-error.log
long_query_time       = 10
slow-query-log-file       = /var/lib/mysql/log/mysqld-queries_slow.log
log-slave-updates
log-bin       = /var/lib/mysql/log/bin/slave-bin
log-warnings        = 1

max_relay_log_size        = 200M
relay_log_space_limit       = 25000M

# -------------------------------------------------------------------------------
# ++++ Network
# -------------------------------------------------------------------------------
max_connections       = 3000
max_connect_errors        = 1000
wait_timeout        = 120
connect_timeout       = 30
interactive_timeout       = 3600
slave_net_timeout       = 120
back_log        = 50
max_allowed_packet        = 128M


# -------------------------------------------------------------------------------
# ++++ Misc
# -------------------------------------------------------------------------------
# Text Searchs
ft_min_word_len        = 2
plugin-load = ha_tokudb

# -------------------------------------------------------------------------------
# ++++ Threads
# -------------------------------------------------------------------------------
thread_concurrency        = 8
thread_cache        = 64

# -------------------------------------------------------------------------------
# ++++ Memory
# -------------------------------------------------------------------------------
# Tables
table_cache       = 4096
tmp_table_size        = 256M
# Memory per Thread
sort_buffer_size        = 8M
read_buffer_size        = 4M
read_rnd_buffer_size        = 16M
# Query Cache
query_cache_type        = 1
query_cache_limit       = 2M
query_cache_size        = 64M

# -------------------------------------------------------------------------------
# ++++ MyISAM Parameters
# -------------------------------------------------------------------------------
key_buffer_size       = 1024M
myisam_sort_buffer_size       = 64M
myisam_recover        = FORCE,BACKUP

# -------------------------------------------------------------------------------
# ++++ InnoDB Parameters
# -------------------------------------------------------------------------------
# General
innodb_data_home_dir        = /var/lib/mysql/innodb
innodb_log_group_home_dir       = /var/lib/mysql/innodblogs
innodb_file_per_table
innodb_data_file_path       = ibdata1:100M:autoextend
innodb_status_file        = ib_status
innodb_autoextend_increment     = 10M
innodb_support_xa       = 0
innodb_thread_concurrency       = 8
innodb_flush_method       = O_DIRECT
innodb_flush_log_at_trx_commit  = 2
# Memory
innodb_buffer_pool_size       = 6G
innodb_additional_mem_pool_size = 8M
innodb_open_files       = 512
# Logging
innodb_log_buffer_size        = 8M
innodb_log_file_size        = 256M
innodb_log_files_in_group       = 2


# -------------------------------------------------------------------------------
# ++++ Replication : SLAVE Profile
# -------------------------------------------------------------------------------
#skip-slave-start
server-id       = 124388
relay-log       = /var/lib/mysql/log/replication/slave-bin
relay-log-info-file = /var/lib/mysql/log/replication/slave-log.info <http://slave-log.info> master-info-file = /var/lib/mysql/log/replication/master-log.info <http://master-log.info>
max_binlog_size       = 20971520

read-only       = 1

[mysqldump]
quick
max_allowed_packet        = 128M

[isamchk]
key_buffer        = 256M
sort_buffer_size        = 256M
read_buffer       = 2M
write_buffer        = 2M

[myisamchk]
key_buffer        = 256M
sort_buffer_size        = 256M
read_buffer       = 2M
write_buffer        = 2M

[mysqlhotcopy]
interactive-timeout

On Tue, Mar 3, 2015 at 12:56 PM, Justin Swanhart <greenlion@xxxxxxxxx <mailto:greenlion@xxxxxxxxx>> wrote:

    Hi,

    You might try to find the source of the termination with this:
    http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/

    On Tue, Mar 3, 2015 at 8:45 AM, jocelyn fournier
    <jocelyn.fournier@xxxxxxxxx <mailto:jocelyn.fournier@xxxxxxxxx>>
    wrote:

        Hi,

        Could you send your my.cnf ?

        Thanks,
          Jocelyn

        Le 03/03/2015 16:24, Gabriel Sosa a écrit :
        Hello,

        I've been a proudly user of mariadb 5.5.x for a long time now
         but given the nice feature setI decided to give mariadb 10.x
        a try.

        I took one of our current slaves running mariadb 5.5.x (on
        centos 6.5) and followed the upgrade steps using yum and ran
        *mysql_upgrade*. The upgrade ran without any trouble...then I
        moved a huge table (about 1B records right now) from InnoDB
        to TokuDB.

        Now, every couple of hours I find that the replication is far
        behind the master and the reason is because this server keeps
        checking tables marked as crashed....

        I can't seems to find any indicator of OOM killer in the
        system logs NOR anything related to that in the mysql log,
        the only clue I have is:


        ---------------
        *150303 10:08:57 mysqld_safe Number of processes running now: 0*
        *150303 10:08:57 mysqld_safe mysqld restarted*
        150303 10:08:58 [Warning] 'THREAD_CONCURRENCY' is deprecated
        and will be removed in a future release.
        150303 10:08:59 [Warning] option 'innodb-status-file':
        boolean value 'ib_status' wasn't recognized. Set to OFF.
        150303 10:08:59 [Warning] option
        'innodb-autoextend-increment': unsigned value 10485760
        adjusted to 1000
        150303 10:08:59 [Note] InnoDB: Using mutexes to ref count
        buffer pool pages
        150303 10:08:59 [Note] InnoDB: The InnoDB memory heap is disabled
        150303 10:08:59 [Note] InnoDB: Mutexes and rw_locks use GCC
        atomic builtins
        150303 10:08:59 [Note] InnoDB: Memory barrier is not used
        150303 10:08:59 [Note] InnoDB: Compressed tables use zlib 1.2.3
        150303 10:08:59 [Note] InnoDB: Using Linux native AIO
        150303 10:08:59 [Note] InnoDB: Using CPU crc32 instructions
        150303 10:08:59 [Note] InnoDB: Initializing buffer pool, size
        = 6.0G
        150303 10:08:59 [Note] InnoDB: Completed initialization of
        buffer pool
        150303 10:08:59 [Note] InnoDB: Highest supported file format
        is Barracuda.
        150303 10:08:59 [Note] InnoDB: Log scan progressed past the
        checkpoint lsn 14178608217047
        150303 10:08:59 [Note] InnoDB: Database was not shutdown
        normally!
        150303 10:08:59 [Note] InnoDB: Starting crash recovery.
        150303 10:08:59 [Note] InnoDB: Reading tablespace information
        from the .ibd files...
        150303 10:09:05 [Note] InnoDB: Restoring possible
        half-written data pages
        150303 10:09:05 [Note] InnoDB: from the doublewrite buffer...
        InnoDB: Doing recovery: scanned up to log sequence number
        14178613459456
        InnoDB: Doing recovery: scanned up to log sequence number
        14178618702336
        InnoDB: Doing recovery: scanned up to log sequence number
        14178623945216
        InnoDB: Doing recovery: scanned up to log sequence number
        14178629188096
        InnoDB: Doing recovery: scanned up to log sequence number
        14178634430976
        InnoDB: Doing recovery: scanned up to log sequence number
        14178639673856
        InnoDB: Doing recovery: scanned up to log sequence number
        14178644916736
        ......
        ......
        ......
        ......
        150303 10:09:14 [Note] InnoDB: Starting an apply batch of log
        records to the database...
        InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13
        14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
        34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
        54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
        74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
        94 95 96 97 98 99
        InnoDB: Apply batch completed
        InnoDB: In a MySQL replication slave the last master binlog file
        InnoDB: position 0 16433109, file name slave-bin.852141
        InnoDB: Last MySQL binlog file position 0 16784214, file name
        /var/lib/mysql/log/bin/slave-bin.005211
        150303 10:09:34 [Note] InnoDB: 128 rollback segment(s) are
        active.
        150303 10:09:34 [Note] InnoDB: Waiting for purge to start
        150303 10:09:34 [Note] InnoDB:  Percona XtraDB
        (http://www.percona.com) 5.6.22-72.0 started; log sequence
        number 14178834821658
        Tue Mar  3 10:09:36 2015 TokuFT recovery starting in env
        /var/lib/mysql/data/
        Tue Mar  3 10:09:36 2015 TokuFT recovery scanning backward
        from 1330297
        Tue Mar  3 10:09:36 2015 TokuFT recovery bw_end_checkpoint at
        1330297 timestamp 1425395277901416 xid 1330281 (bw_newer)
        Tue Mar  3 10:09:36 2015 TokuFT recovery bw_begin_checkpoint
        at 1330281 timestamp 1425395273487081 (bw_between)
        Tue Mar  3 10:09:36 2015 TokuFT recovery turning around at
        begin checkpoint 1330281 time 4414335
        Tue Mar  3 10:09:36 2015 TokuFT recovery starts scanning
        forward to 1330297 from 1330281 left 16 (fw_between)
        Tue Mar  3 10:09:36 2015 TokuFT recovery closing 14 dictionaries
        Tue Mar  3 10:09:36 2015 TokuFT recovery making a checkpoint
        Tue Mar  3 10:09:36 2015 TokuFT recovery done
        150303 10:09:36 [Note] Recovering after a crash using
        /var/lib/mysql/log/bin/slave-bin

        ---------------

        The odd thing is that the 5.5.x version was working just fine
        (since about a year) in the same hardware...nothing changed
        in that front.

        Any clue?

        Thank you in advance



        _______________________________________________
        Mailing list:https://launchpad.net/~maria-discuss  <https://launchpad.net/%7Emaria-discuss>
        Post to     :maria-discuss@xxxxxxxxxxxxxxxxxxx  <mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
        Unsubscribe :https://launchpad.net/~maria-discuss  <https://launchpad.net/%7Emaria-discuss>
        More help   :https://help.launchpad.net/ListHelp


        _______________________________________________
        Mailing list: https://launchpad.net/~maria-discuss
        <https://launchpad.net/%7Emaria-discuss>
        Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
        <mailto:maria-discuss@xxxxxxxxxxxxxxxxxxx>
        Unsubscribe : https://launchpad.net/~maria-discuss
        <https://launchpad.net/%7Emaria-discuss>
        More help   : https://help.launchpad.net/ListHelp





--
Gabriel Sosa
Sometimes the questions are complicated and the answers are simple. -- Dr. Seuss


Follow ups

References