mysql-proxy-discuss team mailing list archive

[nick loeve <trickie@xxxxxxxxx>]

Hey!

On Mon, Feb 9, 2009 at 1:20 PM, nick loeve <trickie@xxxxxxxxx> wrote:
>> that.
>>
>>>>> [...]
>>>
>>> Step one sounds similar to what we tried to do within our plugin, but
>>> we more and more had to re-implement parts of the core engine within
>>> our plugin to accommodate multiple threads using libevent. I would be
>>> interested in helping out where possible to achieve what you described
>>> above.
>>
>> I just briefly talked to Jan and it seems he still has a somewhat clean
>> patch
>> for our multithreading efforts. IIUC he's pushing that to Launchpad today,
>> so maybe
>> that helps to see what's involved. Of course it's totally experimental and
>> might
>> prove disastrous if used in production etc. *disclaim* ;)
>
> Excellent! We are going to give it run in our benchmarks.

I reset the funnel branch on launchpad, which is now using the changes
from Jan's event threads branch.

ATM there is some memory corruption when backend connections are idle,
a wait_timeout occurs, mysql kills the conneciton, and the connection
is removed from the connection pool. Im currently debugging this,
which seems to propogate as a segfault during libevent
dispatching/loop-exiting in the event threads themselves.

Pending the fix of that issue, we will do some more benchmarking, but
some quick tests we did show that (at least in our funnel use case)
there is alot of fine tuning to be had between no. of backend
connections and no. of event threads. Should be interesting.

>
>>
>> Implementing a backlog mostly in a plugin is probably rather painful in the
>> long run,
>> so I'd like to see this functionality to go into the core proper.
>
> Yes i agree. If you do not need a backlog, most of the implementation
> can be fairly transparent IMHO.

I changed a few things in backlog handling so that we can start moving
it into the core. Im not sure of the best way to do it in a
plugin-generic way, but it works for us so far.

>
>>
>> Last night I've had a brief diff-look on the funnel plugin, here are some
>> observations in no particular order:
>> * in funnel_read_auth I would refrain from doing `do { ... } while(0);` just
>> because there's a break statement in there (i guess it was just the fastest
>> way to implement...)
>> * we are generally using G_STRLOC instead of __FILE__, __LINE__ now, this is
>> not consistently in the proxy plugin either ATM, just noting (and less to
>> type :))
>> * the diff is showing what i've suspect for a long time:
>>  we need to refactor the proxy plugin into some more manageable parts to
>> avoid copying around this much code. I've noticed this before in a different
>> plugin (for a completely different purpose, though) but somehow time was
>> always to short to actually do it.
>> * that you removed all Lua code gave me an interesting idea:
>>  it looks like we can factor that out completely as well, to make it
>> optional, and of course pluggable, for plugin writers. I have some nifty
>> ideas on this already.
>>  in general the amount of necessary code duplication right now bothers me
>> greatly
>> * the limitations you mentioned in your initial mail regarding the number of
>> backends and different users look relatively easy to lift.
>>  backends: since you always get the first backend replacing the constant
>> with a function call that picks a backend based on some algorithm should be
>> all that's necessary
>>  users: if i'm not mistaken (and there's no bug in it) the connection pool
>> code should provide this already.
>> * moving to a multithreaded network io implementation, the code is obviously
>> not 100% safe because of the backends. for now i'd take a global lock when
>> actually handling them (as long as they aren't modified at runtime that
>> should be "safe", barring the fact that UP/DOWN information has a race
>> condition). this is something we have to properly fix in the core for sure,
>> but clients should be prepared to handle failed queries by reexecuting them
>> anyway.

I fixed up a couple of the style things you mentioned also.

Cheers

-- 
Nick Loeve