maria-discuss team mailing list archive

[Justin Swanhart <greenlion@xxxxxxxxx>]

Hi,

Presumably, if the server is using too much ram it will a) crash and leave
a stack trace (not happening) or b) invoke the OOMPK which leaves a trace
in syslog, which also, presumably, is not happening, since she can find no
trace of OOMPK activity.

So memory is a good guess (output of free -m would be useful) I doubt it is
memory related do to lack of evidence for it.

--Justin

On Tue, Mar 3, 2015 at 9:58 AM, jocelyn fournier <jocelyn.fournier@xxxxxxxxx
> wrote:

>  Hi,
>
> No variables to tweak the tokudb memory allocation ?
> By default tokudb_cache_size takes half of the total system memory.
> With 6G allocated to InnoDB buffer pool, and 1G to the MyISAM key buffer,
> are you sure you have enough memory for TokuDB ?
>
>   Jocelyn
>
>
> Le 03/03/2015 17:37, Gabriel Sosa a écrit :
>
>  @joocelyn Here is the my.cfg file
>
>  @justin I will check that post. Thanks
>
>  [mysqld]
>
>  #
> -------------------------------------------------------------------------------
> # ++++ General
> #
> -------------------------------------------------------------------------------
> datadir                         = /var/lib/mysql/data
> pid-file                        = /var/run/mysqld/mysqld.pid
> socket                          = /var/lib/mysql/mysql.sock
> tmpdir                          = /var/lib/mysql/tmp
> port                            = 3306
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Logging
> #
> -------------------------------------------------------------------------------
> log-error                       = /var/lib/mysql/log/mysqld-error.log
> long_query_time                 = 10
> slow-query-log-file             =
> /var/lib/mysql/log/mysqld-queries_slow.log
> log-slave-updates
> log-bin                         = /var/lib/mysql/log/bin/slave-bin
> log-warnings                    = 1
>
>  max_relay_log_size              = 200M
> relay_log_space_limit           = 25000M
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Network
> #
> -------------------------------------------------------------------------------
> max_connections                 = 3000
> max_connect_errors              = 1000
> wait_timeout                    = 120
> connect_timeout                 = 30
> interactive_timeout             = 3600
> slave_net_timeout               = 120
> back_log                        = 50
> max_allowed_packet              = 128M
>
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Misc
> #
> -------------------------------------------------------------------------------
> # Text Searchs
> ft_min_word_len                  = 2
> plugin-load  = ha_tokudb
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Threads
> #
> -------------------------------------------------------------------------------
> thread_concurrency              = 8
> thread_cache                    = 64
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Memory
> #
> -------------------------------------------------------------------------------
> # Tables
> table_cache                     = 4096
> tmp_table_size                  = 256M
> # Memory per Thread
> sort_buffer_size                = 8M
> read_buffer_size                = 4M
> read_rnd_buffer_size            = 16M
> # Query Cache
> query_cache_type                = 1
> query_cache_limit               = 2M
> query_cache_size                = 64M
>
>  #
> -------------------------------------------------------------------------------
> # ++++ MyISAM Parameters
> #
> -------------------------------------------------------------------------------
> key_buffer_size                 = 1024M
> myisam_sort_buffer_size         = 64M
> myisam_recover                  = FORCE,BACKUP
>
>  #
> -------------------------------------------------------------------------------
> # ++++ InnoDB Parameters
> #
> -------------------------------------------------------------------------------
> # General
> innodb_data_home_dir            = /var/lib/mysql/innodb
> innodb_log_group_home_dir       = /var/lib/mysql/innodblogs
> innodb_file_per_table
> innodb_data_file_path           = ibdata1:100M:autoextend
> innodb_status_file              = ib_status
> innodb_autoextend_increment     = 10M
> innodb_support_xa               = 0
> innodb_thread_concurrency       = 8
> innodb_flush_method             = O_DIRECT
> innodb_flush_log_at_trx_commit  = 2
> # Memory
> innodb_buffer_pool_size         = 6G
> innodb_additional_mem_pool_size = 8M
> innodb_open_files               = 512
> # Logging
> innodb_log_buffer_size          = 8M
> innodb_log_file_size            = 256M
> innodb_log_files_in_group       = 2
>
>
>  #
> -------------------------------------------------------------------------------
> # ++++ Replication : SLAVE Profile
> #
> -------------------------------------------------------------------------------
> #skip-slave-start
> server-id                       = 124388
> relay-log                       = /var/lib/mysql/log/replication/slave-bin
> relay-log-info-file             = /var/lib/mysql/log/replication/
> slave-log.info
> master-info-file                = /var/lib/mysql/log/replication/
> master-log.info
> max_binlog_size                 = 20971520
>
>  read-only                       = 1
>
>  [mysqldump]
> quick
> max_allowed_packet              = 128M
>
>  [isamchk]
> key_buffer                      = 256M
> sort_buffer_size                = 256M
> read_buffer                     = 2M
> write_buffer                    = 2M
>
>  [myisamchk]
> key_buffer                      = 256M
> sort_buffer_size                = 256M
> read_buffer                     = 2M
> write_buffer                    = 2M
>
>  [mysqlhotcopy]
> interactive-timeout
>
> On Tue, Mar 3, 2015 at 12:56 PM, Justin Swanhart <greenlion@xxxxxxxxx>
> wrote:
>
>> Hi,
>>
>>  You might try to find the source of the termination with this:
>>
>> http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/
>>
>> On Tue, Mar 3, 2015 at 8:45 AM, jocelyn fournier <
>> jocelyn.fournier@xxxxxxxxx> wrote:
>>
>>>  Hi,
>>>
>>> Could you send your my.cnf ?
>>>
>>> Thanks,
>>>   Jocelyn
>>>
>>> Le 03/03/2015 16:24, Gabriel Sosa a écrit :
>>>
>>>   Hello,
>>>
>>>  I've been a proudly user of mariadb 5.5.x for a long time now  but
>>> given the nice feature setI decided to give mariadb 10.x a try.
>>>
>>>  I took one of our current slaves running mariadb 5.5.x (on centos 6.5)
>>> and followed the upgrade steps using yum and ran *mysql_upgrade*. The
>>> upgrade ran without any trouble...then I moved a huge table (about 1B
>>> records right now) from InnoDB to TokuDB.
>>>
>>>  Now, every couple of hours I find that the replication is far behind
>>> the master and the reason is because this server keeps checking tables
>>> marked as crashed....
>>>
>>>  I can't seems to find any indicator of OOM killer in the system logs
>>> NOR anything related to that in the mysql log, the only clue I have is:
>>>
>>>
>>>  ---------------
>>> *150303 10:08:57 mysqld_safe Number of processes running now: 0*
>>> *150303 10:08:57 mysqld_safe mysqld restarted*
>>> 150303 10:08:58 [Warning] 'THREAD_CONCURRENCY' is deprecated and will be
>>> removed in a future release.
>>> 150303 10:08:59 [Warning] option 'innodb-status-file': boolean value
>>> 'ib_status' wasn't recognized. Set to OFF.
>>> 150303 10:08:59 [Warning] option 'innodb-autoextend-increment': unsigned
>>> value 10485760 adjusted to 1000
>>> 150303 10:08:59 [Note] InnoDB: Using mutexes to ref count buffer pool
>>> pages
>>> 150303 10:08:59 [Note] InnoDB: The InnoDB memory heap is disabled
>>> 150303 10:08:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic
>>> builtins
>>> 150303 10:08:59 [Note] InnoDB: Memory barrier is not used
>>> 150303 10:08:59 [Note] InnoDB: Compressed tables use zlib 1.2.3
>>> 150303 10:08:59 [Note] InnoDB: Using Linux native AIO
>>> 150303 10:08:59 [Note] InnoDB: Using CPU crc32 instructions
>>> 150303 10:08:59 [Note] InnoDB: Initializing buffer pool, size = 6.0G
>>> 150303 10:08:59 [Note] InnoDB: Completed initialization of buffer pool
>>> 150303 10:08:59 [Note] InnoDB: Highest supported file format is
>>> Barracuda.
>>> 150303 10:08:59 [Note] InnoDB: Log scan progressed past the checkpoint
>>> lsn 14178608217047
>>> 150303 10:08:59 [Note] InnoDB: Database was not shutdown normally!
>>> 150303 10:08:59 [Note] InnoDB: Starting crash recovery.
>>> 150303 10:08:59 [Note] InnoDB: Reading tablespace information from the
>>> .ibd files...
>>> 150303 10:09:05 [Note] InnoDB: Restoring possible half-written data
>>> pages
>>> 150303 10:09:05 [Note] InnoDB: from the doublewrite buffer...
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178613459456
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178618702336
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178623945216
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178629188096
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178634430976
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178639673856
>>> InnoDB: Doing recovery: scanned up to log sequence number 14178644916736
>>> ......
>>> ......
>>> ......
>>> ......
>>>  150303 10:09:14 [Note] InnoDB: Starting an apply batch of log records
>>> to the database...
>>> InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
>>> 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
>>> 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67
>>> 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
>>> 93 94 95 96 97 98 99
>>> InnoDB: Apply batch completed
>>> InnoDB: In a MySQL replication slave the last master binlog file
>>> InnoDB: position 0 16433109, file name slave-bin.852141
>>> InnoDB: Last MySQL binlog file position 0 16784214, file name
>>> /var/lib/mysql/log/bin/slave-bin.005211
>>> 150303 10:09:34 [Note] InnoDB: 128 rollback segment(s) are active.
>>> 150303 10:09:34 [Note] InnoDB: Waiting for purge to start
>>> 150303 10:09:34 [Note] InnoDB:  Percona XtraDB (http://www.percona.com)
>>> 5.6.22-72.0 started; log sequence number 14178834821658
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery starting in env
>>> /var/lib/mysql/data/
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery scanning backward from 1330297
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery bw_end_checkpoint at 1330297
>>> timestamp 1425395277901416 xid 1330281 (bw_newer)
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery bw_begin_checkpoint at 1330281
>>> timestamp 1425395273487081 (bw_between)
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery turning around at begin
>>> checkpoint 1330281 time 4414335
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery starts scanning forward to
>>> 1330297 from 1330281 left 16 (fw_between)
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery closing 14 dictionaries
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery making a checkpoint
>>> Tue Mar  3 10:09:36 2015 TokuFT recovery done
>>> 150303 10:09:36 [Note] Recovering after a crash using
>>> /var/lib/mysql/log/bin/slave-bin
>>>
>>>  ---------------
>>>
>>>  The odd thing is that the 5.5.x version was working just fine (since
>>> about a year) in the same hardware...nothing changed in that front.
>>>
>>>  Any clue?
>>>
>>>  Thank you in advance
>>>
>>>
>>>
>>>  _______________________________________________
>>> Mailing list: https://launchpad.net/~maria-discuss
>>> Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~maria-discuss
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~maria-discuss
>>> Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~maria-discuss
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>
>
>  --
> Gabriel Sosa
> Sometimes the questions are complicated and the answers are simple. -- Dr.
> Seuss
>
>
>