percona-discussion team mailing list archive

[MARK CALLAGHAN <mdcallag@xxxxxxxxx>]

On Fri, May 22, 2009 at 12:43 AM, Jeremy Kerr <jk@xxxxxxxxxx> wrote:
> Just a thought here - perhaps we're not actually deadlocked, just
> running really slowly due to the one-second wakeups issued by the
> sync_arr_wake_threads_if_sema_free() thread. If the wakeups aren't
> occurring properly due to the weak memory ordering model on Power6, it
> may be possible that lock acquires are taking up to a second to
> complete.
>
> I'd suggest testing this by putting a breakpoint on line 908 of
> sync0arr.c and seeing how often it's hit.

One of the earlier versions of the Google faster rw-mutex patch had a
performance bug found by Percona testing where wakeups were frequently
missed so we know that such a problem is possible. I am not sure how
that can happen for the Innodb mutex. But debugging this would be
easier if there were a counter for threads that are made runnable by
sync_arr_wake_threads_if_sema_free() and this counter were displayed
in SHOW INNODB STATUS output. The v3 Google patch does that.

Until we get that, getting several stacktraces a few seconds apart
might help to determine whether there is no or slow progress when the
server is stuck. But first I want to see the contents of the buffer
pool mutex when the server is in this state. And we also need to know
whether the binary uses gcc atomics.

-- 
Mark Callaghan
mdcallag@xxxxxxxxx