pbxt-discuss team mailing list archive

[Paul McCullagh <paul.mccullagh@xxxxxxxxx>]

Hi Mark,

What has happened here is the index cache is full of dirty pages, and  
needs to be flushed.

When a thread hits this situation it starts a background thread to  
flush the indexes. That is why we have xt_wait_for_async_tasks() in  
the top-most stack trace.

The background tasks are all busy flushing the indexes. One of the  
problems here is that indexes are actually synced to disk before  
anything can continue (although this is unnecessary for an import -  
but there is no way to tell PBXT that this is the case).

Also, writing an index is "all-or-nothing" so since about 20 GB of  
index data is dirty it probably all has to be written.

Of course, as soon as one thread runs into this problem so do they all.

Normally this situation should be avoided...

That is because PBXT starts to flush the index before it is full of  
dirty pages. However, if it starts too late, then we have this problem.

Since the PBXT does not always know when to start flushing, PBXT 1.1  
has the pbxt_index_dirty_threshold system variable. Try setting this  
down to 50 (the default value is 80%, I think. Then PBXT will start  
flushing when half of the cache is full of dirty pages.

Hopefully it will then finish before the foreground threads fill the  
index cache with dirty pages.

On Oct 30, 2010, at 1:45 AM, MARK CALLAGHAN wrote:

> I am reloading many tables into pbxt tables using 8 concurrent
> sessions. Using poor man's profiler output most threads are waiting.
> The first thread stack shows where all of the user threads spend most
> of their time. These stacks are from PBXT 1.1 and I am using the
> default settings except for:
> pbxt_index_cache_size=20G
> pbxt_record_cache_size=40G
> pbxt_flush_log_at_trx_commit=2
>
> Can you explain this?
>
>      8  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,xt_wait_for_async_tasks 
> ,xt_wait_for_async_task_results 
> ,idx_out_of_memory_failure 
> ,xt_idx_insert,xt_tab_new_record,ha_pbxt::write_row,h
> andler 
> ::ha_write_row 
> ,write_record 
> ,mysql_load 
> ,mysql_execute_command 
> ,mysql_parse 
> ,dispatch_command,do_command,handle_one_connection,start_thread,clone
>      7  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,XTDatabaseLog 
> ::xlog_append 
> ,XTDatabaseLog 
> ::xlog_flush 
> ,xt_flush_indices 
> ,XTFlushIndexTask::tk_task,db_thread_pool_main,db_thread_pool_ru
> n_thread,xt_thread_main,start_thread,clone
>      6  
> pthread_cond_wait 
> @@GLIBC_2.3.2 
> ,MYSQL_BIN_LOG 
> ::wait_for_update 
> ,mysql_binlog_send 
> ,dispatch_command,do_command,handle_one_connection,start_thread,clone
>      2  
> pthread_cond_wait 
> @@GLIBC_2.3.2 
> ,one_thread_per_connection_end 
> ,handle_one_connection,start_thread,clone
>      2  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,db_thread_pool_main 
> ,db_thread_pool_run_thread,xt_thread_main,start_thread,clone
>      2
>      1  
> read 
> ,my_real_read 
> ,my_net_read,do_command,handle_one_connection,start_thread,clone
>      1  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,xt_wait_for_async_tasks 
> ,xt_wait_for_async_task_results 
> ,xres_cp_async_checkpoint 
> ,xres_cp_main,xres_cp_run_thread,xt_thread_main,start_
> thread,clone
>      1  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,xt_wait_for_async_tasks 
> ,xt_wait_for_async_task_results 
> ,idx_out_of_memory_failure 
> ,xt_idx_update_row_id,xn_sw_clean_indices,xn_sw_clean
> up_variation 
> ,xn_sw_main,xn_sw_run_thread,xt_thread_main,start_thread,clone
>      1  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,xlog_wr_wait_for_write_condition 
> ,xlog_wr_main,xlog_wr_run_thread,xt_thread_main,start_thread,clone
>      1  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,tabc_fr_main,tabc_fr_run_thread,xt_thread_main,start_thread,clone
>      1  
> pthread_cond_timedwait 
> @@GLIBC_2.3.2 
> ,xt_timed_wait_cond 
> ,dl_co_wait 
> ,dl_co_main,dl_run_co_thread,xt_thread_main,start_thread,clone
>      1 poll,handle_connections_sockets,main,poll
>      1  
> fsync 
> ,xt_flush_file 
> ,XTDatabaseLog 
> ::xlog_append 
> ,XTDatabaseLog 
> ::xlog_flush 
> ,xt_flush_indices 
> ,XTFlushIndexTask 
> ::tk_task 
> ,db_thread_pool_main 
> ,db_thread_pool_run_thread,xt_thread_main,start_threa
> d,clone
>      1 do_sigwait,sigwait,signal_hand,start_thread,clone
>
>
> --
> Mark Callaghan
> mdcallag@xxxxxxxxx
>
> _______________________________________________
> Mailing list: https://launchpad.net/~pbxt-discuss
> Post to     : pbxt-discuss@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~pbxt-discuss
> More help   : https://help.launchpad.net/ListHelp