maria-discuss team mailing list archive

[Jonas Krauss <jkrauss82@xxxxxxxxx>]

Hi all,

we had a database setup running in production for quite some time (> 5
months) and it crashed during last weekend without obvious reason. MariaDB
error log stops with (seemingly most useful output set to bold):

2019-01-14  9:07:47 139766003611904 [Note] RocksDB: 2 column families found
2019-01-14  9:07:47 139766003611904 [Note] RocksDB: Column Families at
start:
2019-01-14  9:07:47 139766003611904 [Note]   cf=default
2019-01-14  9:07:47 139766003611904 [Note]     write_buffer_size=134217728
2019-01-14  9:07:47 139766003611904 [Note]
 target_file_size_base=268435456
2019-01-14  9:07:47 139766003611904 [Note]   cf=__system__
2019-01-14  9:07:47 139766003611904 [Note]     write_buffer_size=134217728
2019-01-14  9:07:47 139766003611904 [Note]
 target_file_size_base=268435456
mysqld:
/home/buildbot/buildbot/build/mariadb-10.2.18/storage/rocksdb/rocksdb/db/column_family.cc:1238:
*rocksdb::ColumnFamilySet::~ColumnFamilySet(): Assertion `last_ref' failed.*
190114  9:07:58 [ERROR] mysqld got signal 6 ;
...

The rest is a memory dump seemingly not too helpful(?). We tried the
various recovery modes of rocksdb_wal_recovery_mode (0,2,3) but none seems
to have any effect, the output in the log is exactly the same.

*Version and OS infos*:
MariaDB version: 10.2.18-MariaDB-10.2.18+maria~stretch-log
Running on Debian Linux version 4.9.0-8-amd64 (
debian-kernel@xxxxxxxxxxxxxxxx) (gcc version 6.3.0 20170516 (Debian
6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08)
We use *jemalloc* as memory allocator.
64GB RAM machine with RAID1 SSD 256GB for logs and 10TB HDD for tables

Here is the *rocksdb config part of the my.cnf*:
plugin-load-add = ha_rocksdb.so
default-storage-engine = rocksdb
rocksdb_unsafe_for_binlog = 1 # enables statement based replication
rocksdb_datadir = /var/local/mysql/rocksdb
rocksdb_wal_dir = /var/mysql_logs/rocksdb
rocksdb_tmpdir = /var/mysql_logs/rocksdb
rocksdb_flush_log_at_trx_commit = 0
rocksdb_use_direct_io_for_flush_and_compaction = 0
rocksdb_use_direct_reads = 0
rocksdb_max_open_files = -1
rocksdb_max_background_jobs = 8
rocksdb_max_total_wal_size = 4G
rocksdb_block_size = 64K
rocksdb_block_cache_size = 16G
rocksdb_table_cache_numshardbits = 6
rocksdb_new_table_reader_for_compaction_inputs = 1
rocksdb_compaction_readahead_size = 4M
#rocksdb_db_write_buffer_size = 0 # max write buffer across all column
families, zero = disabled
#rocksdb_wal_recovery_mode = 1
rocksdb_bytes_per_sync = 4M
rocksdb_wal_bytes_per_sync = 4M
rocksdb_rate_limiter_bytes_per_sec = 80M # MB/s. Increase if you're running
on higher spec machines
rocksdb_compaction_sequential_deletes_count_sd = 1
rocksdb_compaction_sequential_deletes = 199999
rocksdb_compaction_sequential_deletes_window = 200000
#rocksdb_rpl_lookup_rows=0
rocksdb_default_cf_options=write_buffer_size=128m;max_write_buffer_number=4;target_file_size_base=256m;max_bytes_for_level_base=2560m;target_file_size_multiplier=2;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;compression=kSnappyCompression;bottommost_compression=kZlibCompression;compression_opts=-14:1:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio

Does anyone have a idea what could cause this crash? Google did not return
helpful results, most similar thing anyone had is this in my search:
https://github.com/facebook/mysql-5.6/issues/391

Any help is greatly appreciated.

Best regards

Jonas