maria-discuss team mailing list archive
-
maria-discuss team
-
Mailing list archive
-
Message #04241
Re: limiting memory use by mariadb?
dataset size: 1000000 rows. 5 int cols. 7 char cols
I run mysqlslap for 99 concurrent clients ( out of 100 max)
Container memory is limited to 5.268G
Query cache configuration is default, i didn't try to change it
Why MyISAM engine is more suitable in resource constrained env? What's the
cons of using it?
Below is my "my.cnf":
# MariaDB database server configuration file.
#
# You can copy this file to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html
# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock
# Here is entries for some specific programs
# The following values assume you have at least 32M ram
# This was formally known as [safe_mysqld]. Both versions are currently
parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
[mysqld]
skip-host-cache
skip-name-resolve
#
# * Basic Settings
#
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mysql
lc_messages = en_US
skip-external-locking
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
#
# * Fine Tuning
#
max_connections = 100
connect_timeout = 5
wait_timeout = 600
max_allowed_packet = 16M
thread_cache_size = 128
sort_buffer_size = 4M
bulk_insert_buffer_size
= 16M
tmp_table_size = 32M
max_heap_table_size = 32M
#
# * MyISAM
#
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched. On error, make copy and try a repair.
myisam_recover_options = BACKUP
key_buffer_size = 128M
#open-files-limit = 2000
table_open_cache = 400
myisam_sort_buffer_size
= 512M
concurrent_insert = 2
read_buffer_size = 2M
read_rnd_buffer_size = 1M
#
# * Query Cache Configuration
#
# Cache only tiny result sets, so we can fit more in the query cache.
query_cache_limit = 128K
query_cache_size = 64M
# for more write intensive setups, set to DEMAND or OFF
#query_cache_type = DEMAND
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file = /var/log/mysql/mysql.log
#general_log = 1
#
# Error logging goes to syslog due to
/etc/mysql/conf.d/mysqld_safe_syslog.cnf.
#
# we do want to know about network errors and such
#log_warnings = 2
#
# Enable the slow query log to see queries with especially long duration
#slow_query_log[={0|1}]
slow_query_log_file = /var/log/mysql/mariadb-slow.log
long_query_time = 10
#log_slow_rate_limit = 1000
#log_slow_verbosity = query_plan
#log-queries-not-using-indexes
#log_slow_admin_statements
#
# The following can be used as easy to replay backup logs or for
replication.
# note: if you are setting up a replication slave, see README.Debian about
# other settings you may need to change.
#server-id = 1
#report_host = master1
#auto_increment_increment = 2
#auto_increment_offset = 1
#log_bin = /var/log/mysql/mariadb-bin
#log_bin_index = /var/log/mysql/mariadb-bin.index
# not fab for performance, but safer
#sync_binlog = 1
expire_logs_days = 10
max_binlog_size = 100M
# slaves
#relay_log = /var/log/mysql/relay-bin
#relay_log_index = /var/log/mysql/relay-bin.index
#relay_log_info_file = /var/log/mysql/relay-bin.info
#log_slave_updates
#read_only
#
# If applications support it, this stricter sql_mode prevents some
# mistakes like inserting invalid dates etc.
#sql_mode = NO_ENGINE_SUBSTITUTION,TRADITIONAL
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
default_storage_engine = InnoDB
# you can't just change log file size, requires special procedure
#innodb_log_file_size = 50M
innodb_buffer_pool_size
= 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 400
innodb_io_capacity = 400
innodb_flush_method = O_DIRECT
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
#
# * Galera-related settings
#
[galera]
# Mandatory settings
#wsrep_on=ON
#wsrep_provider=
#wsrep_cluster_address=
#binlog_format=row
#default_storage_engine=InnoDB
#innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
#bind-address=0.0.0.0
#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[mysql]
#no-auto-rehash # faster start of mysql but no tab completion
[isamchk]
key_buffer = 16M
#
# * IMPORTANT: Additional settings that can override those from this file!
# The files must end with '.cnf', otherwise they'll be ignored.
#
!includedir /etc/mysql/conf.d/
On Wed, Jan 4, 2017 at 6:31 PM, Reindl Harald <h.reindl@xxxxxxxxxxxxx>
wrote:
> the stuff below only shows innodb settings which is not all
>
> how large is your *dataset*
> how many connections do you have at peak times
> how much memory does your system have for the workload
> how is query cache configured
> why don't you just use MyISAM if you don't have suiteable ressources
> what is the content of your "my.cnf"
>
> if your container is oom killed you likely don't have enough physical
> ressources for your dataset and should fix that properly instead cripple
> down services - we are at 2017 and not in the 1990's where 64 MB was common
>
>
> Am 04.01.2017 um 22:21 schrieb l vic:
>
>> I didn't suggest it as "solution", just wondering if huge default value
>> can cause problems.
>> The real problem is that memory consumption of mysqld is growing in time
>> until the container oom and is killed;
>> What would be my "read config": list of system variables?
>> +---------------------------------------------+------------------------+
>> | Variable_name | Value |
>> +---------------------------------------------+------------------------+
>> | innodb_adaptive_flushing | ON |
>> | innodb_adaptive_flushing_lwm | 10.000000 |
>> | innodb_adaptive_hash_index | ON |
>> | innodb_adaptive_hash_index_partitions | 1 |
>> | innodb_adaptive_max_sleep_delay | 150000 |
>> | innodb_additional_mem_pool_size | 8388608 |
>> | innodb_api_bk_commit_interval | 5 |
>> | innodb_api_disable_rowlock | OFF |
>> | innodb_api_enable_binlog | OFF |
>> | innodb_api_enable_mdl | OFF |
>> | innodb_api_trx_level | 0 |
>> | innodb_autoextend_increment | 64 |
>> | innodb_autoinc_lock_mode | 2 |
>> | innodb_background_scrub_data_check_interval | 3600 |
>> | innodb_background_scrub_data_compressed | OFF |
>> | innodb_background_scrub_data_interval | 604800 |
>> | innodb_background_scrub_data_uncompressed | OFF |
>> | innodb_buf_dump_status_frequency | 0 |
>> | innodb_buffer_pool_dump_at_shutdown | OFF |
>> | innodb_buffer_pool_dump_now | OFF |
>> | innodb_buffer_pool_dump_pct | 100 |
>> | innodb_buffer_pool_filename | ib_buffer_pool |
>> | innodb_buffer_pool_instances | 8 |
>> | innodb_buffer_pool_load_abort | OFF |
>> | innodb_buffer_pool_load_at_startup | OFF |
>> | innodb_buffer_pool_load_now | OFF |
>> | innodb_buffer_pool_populate | OFF |
>> | innodb_buffer_pool_size | 2147483648 |
>> | innodb_change_buffer_max_size | 25 |
>> | innodb_change_buffering | all |
>> | innodb_checksum_algorithm | INNODB |
>> | innodb_checksums | ON |
>> | innodb_cleaner_lsn_age_factor | HIGH_CHECKPOINT |
>> | innodb_cmp_per_index_enabled | OFF |
>> | innodb_commit_concurrency | 0 |
>> | innodb_compression_algorithm | none |
>> | innodb_compression_failure_threshold_pct | 5 |
>> | innodb_compression_level | 6 |
>> | innodb_compression_pad_pct_max | 50 |
>> | innodb_concurrency_tickets | 5000 |
>> | innodb_corrupt_table_action | assert |
>> | innodb_data_file_path | ibdata1:12M:autoextend |
>> | innodb_data_home_dir | |
>> | innodb_default_encryption_key_id | 1 |
>> | innodb_defragment | OFF |
>> | innodb_defragment_fill_factor | 0.900000 |
>> | innodb_defragment_fill_factor_n_recs | 20 |
>> | innodb_defragment_frequency | 40 |
>> | innodb_defragment_n_pages | 7 |
>> | innodb_defragment_stats_accuracy | 0 |
>> | innodb_disable_sort_file_cache | OFF |
>> | innodb_disallow_writes | OFF |
>> | innodb_doublewrite | ON |
>> | innodb_empty_free_list_algorithm | BACKOFF |
>> | innodb_encrypt_log | OFF |
>> | innodb_encrypt_tables | OFF |
>> | innodb_encryption_rotate_key_age | 1 |
>> | innodb_encryption_rotation_iops | 100 |
>> | innodb_encryption_threads | 0 |
>> | innodb_fake_changes | OFF |
>> | innodb_fast_shutdown | 1 |
>> | innodb_fatal_semaphore_wait_threshold | 600 |
>> | innodb_file_format | Antelope |
>> | innodb_file_format_check | ON |
>> | innodb_file_format_max | Antelope |
>> | innodb_file_per_table | ON |
>> | innodb_flush_log_at_timeout | 1 |
>> | innodb_flush_log_at_trx_commit | 1 |
>> | innodb_flush_method | O_DIRECT |
>> | innodb_flush_neighbors | 1 |
>> | innodb_flushing_avg_loops | 30 |
>> | innodb_force_load_corrupted | OFF |
>> | innodb_force_primary_key | OFF |
>> | innodb_force_recovery | 0 |
>> | innodb_foreground_preflush | EXPONENTIAL_BACKOFF |
>> | innodb_ft_aux_table | |
>> | innodb_ft_cache_size | 8000000 |
>> | innodb_ft_enable_diag_print | OFF |
>> | innodb_ft_enable_stopword | ON |
>> | innodb_ft_max_token_size | 84 |
>> | innodb_ft_min_token_size | 3 |
>> | innodb_ft_num_word_optimize | 2000 |
>> | innodb_ft_result_cache_limit | 2000000000 |
>> | innodb_ft_server_stopword_table | |
>> | innodb_ft_sort_pll_degree | 2 |
>> | innodb_ft_total_cache_size | 640000000 |
>> | innodb_ft_user_stopword_table | |
>> | innodb_idle_flush_pct | 100 |
>> | innodb_immediate_scrub_data_uncompressed | OFF |
>> | innodb_instrument_semaphores | OFF |
>> | innodb_io_capacity | 400 |
>> | innodb_io_capacity_max | 2000 |
>> | innodb_kill_idle_transaction | 0 |
>> | innodb_large_prefix | OFF |
>> | innodb_lock_wait_timeout | 50 |
>> | innodb_locking_fake_changes | ON |
>> | innodb_locks_unsafe_for_binlog | ON |
>> | innodb_log_arch_dir | ./ |
>> | innodb_log_arch_expire_sec | 0 |
>> | innodb_log_archive | OFF |
>> | innodb_log_block_size | 512 |
>> | innodb_log_buffer_size | 8388608 |
>> | innodb_log_checksum_algorithm | INNODB |
>> | innodb_log_compressed_pages | OFF |
>> | innodb_log_file_size | 50331648 |
>> | innodb_log_files_in_group | 2 |
>> | innodb_log_group_home_dir | ./ |
>> | innodb_lru_scan_depth | 1024 |
>> | innodb_max_bitmap_file_size | 104857600 |
>> | innodb_max_changed_pages | 1000000 |
>> | innodb_max_dirty_pages_pct | 75.000000 |
>> | innodb_max_dirty_pages_pct_lwm | 0.001000 |
>> | innodb_max_purge_lag | 0 |
>> | innodb_max_purge_lag_delay | 0 |
>> | innodb_mirrored_log_groups | 1 |
>> | innodb_monitor_disable | |
>> | innodb_monitor_enable | |
>> | innodb_monitor_reset | |
>> | innodb_monitor_reset_all | |
>> | innodb_mtflush_threads | 8 |
>> | innodb_old_blocks_pct | 37 |
>> | innodb_old_blocks_time | 1000 |
>> | innodb_online_alter_log_max_size | 134217728 |
>> | innodb_open_files | 400 |
>> | innodb_optimize_fulltext_only | OFF |
>> | innodb_page_size | 16384 |
>> | innodb_prefix_index_cluster_optimization | OFF |
>> | innodb_print_all_deadlocks | OFF |
>> | innodb_purge_batch_size | 300 |
>> | innodb_purge_threads | 1 |
>> | innodb_random_read_ahead | OFF |
>> | innodb_read_ahead_threshold | 56 |
>> | innodb_read_io_threads | 4 |
>> | innodb_read_only | OFF |
>> | innodb_replication_delay | 0 |
>> | innodb_rollback_on_timeout | OFF |
>> | innodb_rollback_segments | 128 |
>> | innodb_sched_priority_cleaner | 19 |
>> | innodb_scrub_log | OFF |
>> | innodb_scrub_log_speed | 256 |
>> | innodb_show_locks_held | 10 |
>> | innodb_show_verbose_locks | 0 |
>> | innodb_simulate_comp_failures | 0 |
>> | innodb_sort_buffer_size | 1048576 |
>> | innodb_spin_wait_delay | 6 |
>> | innodb_stats_auto_recalc | ON |
>> | innodb_stats_method | nulls_equal |
>> | innodb_stats_modified_counter | 0 |
>> | innodb_stats_on_metadata | OFF |
>> | innodb_stats_persistent | ON |
>> | innodb_stats_persistent_sample_pages | 20 |
>> | innodb_stats_sample_pages | 8 |
>> | innodb_stats_traditional | ON |
>> | innodb_stats_transient_sample_pages | 8 |
>> | innodb_status_output | OFF |
>> | innodb_status_output_locks | OFF |
>> | innodb_strict_mode | OFF |
>> | innodb_support_xa | ON |
>> | innodb_sync_array_size | 1 |
>> | innodb_sync_spin_loops | 30 |
>> | innodb_table_locks | ON |
>> | innodb_thread_concurrency | 0 |
>> | innodb_thread_sleep_delay | 10000 |
>> | innodb_track_changed_pages | OFF |
>> | innodb_undo_directory | . |
>> | innodb_undo_logs | 128 |
>> | innodb_undo_tablespaces | 0 |
>> | innodb_use_atomic_writes | OFF |
>> | innodb_use_fallocate | OFF |
>> | innodb_use_global_flush_log_at_trx_commit | ON |
>> | innodb_use_mtflush | OFF |
>> | innodb_use_native_aio | ON |
>> | innodb_use_stacktrace | OFF |
>> | innodb_use_sys_malloc | ON |
>> | innodb_use_trim | OFF |
>> | innodb_version | 5.6.28-76.1 |
>> | innodb_write_io_threads | 4 |
>>
>>
>> On Tue, Jan 3, 2017 at 2:39 PM, Reindl Harald <h.reindl@xxxxxxxxxxxxx
>> <mailto:h.reindl@xxxxxxxxxxxxx>> wrote:
>>
>> don't come with random solutions instead describe your problem
>> properly and show your real config - the subject of this thread is
>> nonsense and the options below are *not* your problem in real life
>>
>> Am 03.01.2017 um 20:13 schrieb l vic:
>>
>> How's about |max_binlog_cache_size or max_binlog_stmt_cache_size ?
>> ||
>> ||max_binlog_cache_size defaults to 18446744073709547520
>> |
>> |Can it be more than available RAM ?
>> |
>>
>> On Fri, Dec 30, 2016 at 9:55 AM, Reindl Harald
>> <h.reindl@xxxxxxxxxxxxx <mailto:h.reindl@xxxxxxxxxxxxx>
>> <mailto:h.reindl@xxxxxxxxxxxxx <mailto:h.reindl@xxxxxxxxxxxxx>>>
>> wrote:
>>
>>
>>
>> Am 30.12.2016 um 14:20 schrieb l vic:
>>
>> and what should happen after reach it?
>> a hard stop?
>> Should use virt. memory/swap
>> why don't you just configure your server to match your
>> needs?
>> I don't have control over the cluster and MariaDB
>> instances share
>> hardware with other services
>>
>>
>> jesus christ fix your mail-client
>>
>> using virtual memory / swap is not the job of the
>> application, it's
>> the job of the underlying operating system to swap out
>> unused pages
>> by LRU algorithms - in most cases when your OS starts heavily
>> swapping the game is over and so you have to adjust the
>> ressource
>> consumation of your services or just add physical memory
>>
>> "I don't have control over the cluster and MariaDB instances
>> share"
>> - when you have no control at all you can't change anything,
>> as long
>> as you have control of your "my.cnf" adjust caches and
>> buffers as i
>> told you that we have mariadb instances which are consuming
>> between
>> 35 MB and many GB RAM - depending on the size of the dataset,
>> buffer-pool settings and so on
>>
>>
>>
>> On Wed, Dec 28, 2016 at 6:42 PM, Reindl Harald
>> <h.reindl@xxxxxxxxxxxxx <mailto:h.reindl@xxxxxxxxxxxxx>
>> <mailto:h.reindl@xxxxxxxxxxxxx <mailto:h.reindl@xxxxxxxxxxxxx>>
>> <mailto:h.reindl@xxxxxxxxxxxxx
>> <mailto:h.reindl@xxxxxxxxxxxxx> <mailto:h.reindl@xxxxxxxxxxxxx
>> <mailto:h.reindl@xxxxxxxxxxxxx>>>>
>> wrote:
>>
>> Am 28.12.2016 um 19:47 schrieb l vic:
>>
>> Is there some way to set hard limit for use of
>> RAM by
>> mariaDB
>> server?
>>
>>
>> and what should happen after reach it?
>> a hard stop?
>>
>> why don't you just configure your server to match
>> your needs?
>>
>> we have MariaDB instances between 35 MB and 10 GB RAM
>> depending on
>> the usecase - small instances with a small dataset
>> don't
>> need much
>> caches and buffers, machine srunning dbmail need
>> really
>> large buffers
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~maria-discuss
> Post to : maria-discuss@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~maria-discuss
> More help : https://help.launchpad.net/ListHelp
>
Follow ups
References
