mysql-proxy-discuss team mailing list archive

[nick loeve <trickie@xxxxxxxxx>]

Hey

On Wed, Feb 18, 2009 at 2:18 AM, Kay Röpke <Kay.Roepke@xxxxxxx> wrote:
> Hi!
>
> On Feb 17, 2009, at 6:57 PM, nick loeve wrote:
>
>> I reset the funnel branch on launchpad, which is now using the changes
>> from Jan's event threads branch.
>
> Great to hear!
>
>> ATM there is some memory corruption when backend connections are idle,
>> a wait_timeout occurs, mysql kills the conneciton, and the connection
>> is removed from the connection pool. Im currently debugging this,
>> which seems to propogate as a segfault during libevent
>> dispatching/loop-exiting in the event threads themselves.
>
> Interesting. It sounds like a missing event_del to me.
> Looking at the code in network_connection_pool_entry_free() the most obvious
> thing
> would be to track the free_sock boolean to see what's going on.
> I've just gone over the call sites for network_connection_pool_entry_free
> and all three
> seem to correctly remove the event handler (one is non-obvious at first, at
> least to me at this hour).
> The next best thing I can come up with is:
> going from network_mysqld_con_idle_handle() to
> network_connection_pool_remove() to
> network_connection_pool_entry_free() will attempt to remove the event
> handler.
> Maybe the problem lies in there somehow.
>

Well I fixed a couple of issues. Im a bit stuck with the following issue:

If I set event-threads > 0 then we get memory corruption like the
following backtrace.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x4294c950 (LWP 20747)]
0x00007ff9321b33c5 in *__GI_raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007ff9321b33c5 in *__GI_raise (sig=<value optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ff9321b473e in *__GI_abort () at abort.c:88
#2  0x00007ff9321efacf in malloc_printerr (action=2,
str=0x7ff93228af45 "malloc(): memory corruption", ptr=0x5108) at
malloc.c:5892
#3  0x00007ff9321f1d36 in _int_malloc (av=0x7ff9324bd960, bytes=1025)
at malloc.c:4185
#4  0x00007ff9321f3c82 in malloc_check (sz=1024, caller=<value
optimized out>) at hooks.c:265
#5  0x00007ff9321f417f in realloc_check (oldmem=0x0, bytes=1024,
caller=0x6) at hooks.c:329
#6  0x00007ff93250d3b6 in IA__g_realloc (mem=0x0, n_bytes=1024) at gmem.c:170
#7  0x00007ff93252d337 in g_string_maybe_expand (string=0x65a720,
len=607) at gstring.c:359
#8  0x00007ff93252d39b in IA__g_string_sized_new (dfl_size=607) at gstring.c:384
#9  0x00007ff933d07a8b in network_socket_read (sock=0x631d90) at
network-socket.c:507
#10 0x00007ff933cfa89f in network_mysqld_read (srv=0x6098f0,
con=0x631d90) at network-mysqld.c:499
#11 0x00007ff933cfc3d4 in network_mysqld_con_handle (event_fd=25,
events=2, user_data=0x655ce0) at network-mysqld.c:1348
#12 0x00007ff933ad7d9b in event_process_active (base=0x61fc50) at event.c:387
#13 0x00007ff933ad80df in event_base_loop (base=0x61fc50, flags=0) at
event.c:536
#14 0x00007ff933ad7e00 in event_base_dispatch (event_base=0x61fc50) at
event.c:407
#15 0x00007ff933f19958 in chassis_event_thread_loop
(event_thread=0x61fba0) at chassis-event-thread.c:257
#16 0x00007ff932534e71 in g_thread_create_proxy (data=0x62e9c0) at gthread.c:635
#17 0x00007ff9329d7047 in start_thread (arg=<value optimized out>) at
pthread_create.c:296
#18 0x00007ff93224328d in clone () from /lib/libc.so.6
#19 0x0000000000000000 in ?? ()

As each connection should have it's state machine running in exactly
one thread at any given time, So I am not how the network_socket
queues are becoming an issue.

I assume though that although the crash consistently happens while
processing the network_socket queues, it may have been caused by my
stupidity elsewhere.

Running with '--event-threads=0' shows no issue.

This has pretty stopped me flat at the moment... any help/pointers
would be appreciated.

Has anyone had a play with the funnel plugin?

Cheers

-- 
Nick Loeve