mysql-proxy-discuss team mailing list archive

[Kay Röpke <Kay.Roepke@xxxxxxx>]

Hi Nick!

On Jun 8, 2009, at 2:08 PM, nick loeve wrote:
> I am going to outline the future direction of the funnel plugin,
> partly to kickstart discussions for the hackathon in Hamburg (in the
> architecture areas I am looking at anyway).

yay :)

> The code in my branch on launchpad is kind of a 'feature complete
> prototype'. We are using it internally at Hyves (where I work) for a
> variety of purposes. Apart from any major failings we have here, the
> current branch will probably not get any more functionality or
> changes. I don't assume anyone else is actively using the plugin, but
> it is usable for the original use case outlined in the blueprint in
> launchpad (a proxy instance per mysqld in a load balanced replication
> slave farm).
>
> I have started a new branch (not yet on launchpad) that will implement
> the feature set of the current plugin, plus some longer term goals,
> such as multiple backends, dynamic add/remove of backends, query load
> balancing and support for multiple users/databases. I would also like
> a cleaner code structure than I have now and I intend to get the code
> more closely aligned with the rest of the project's code, for instance
> creating patches for the connection pool used by the proxy plugin
> instead of implementing a completely separate pool (as it is now in
> the funnel).
>
> The goals (in no particular order here) include:
>
> * backlog support in the chassis core
> * changes to the protocol state machine to make the backlog usable

yes, that's a requirement for sure if we are to make your life easier :)

> * enforceable connection size limits to a connection pool, per user/db
> * dynamically add/remove a backend

We are not sure yet if this will make Proxy 1.0, but it sure would  
make us
happy.
AFAICS we will probably not have the resources to promise that this  
gets done
in time for 1.0, but if you are interested to work on it, I'm sure we  
can
find a way to assist!

> * optional scripting layer (we do not use LUA in the funnel plugin,
> and it would be nice to make LUA support a compile time option)

Once upon a time Lua was an optional compile time thing, but not  
anymore.
Back then, we found that we relied on it for the two things that we  
are working
on: Proxy and the Enterprise Tools agent.
However, we have loose plans to make this layer entirely pluggable in  
the future
and most of the code dealing with Lua interfaces to the internals is  
separated
out into the *-lua.[ch] files. Only a few places remain that probably  
do not have
the necessary #ifdefs today.
Therefore I imagine that we can separate those parts out into a  
separate shared library
which may or may not get compiled/loaded.
Naturally you couldn't compile the proxy plugin without Lua, and even  
if you could
it would make zero sense :)

This is an area I definitely want to tackle at the hackathon, even  
though it is
not planned to do actually introduce the pluggability for Proxy 1.0  
(too intrusive work).
It would be nice to get a feel for the amount of work necessary, though.

> * advanced statistics on the backends, connection pools, chassis as  
> a whole etc

The rough infrastructure is there, although it is not currently used  
in any of the plugins.
Actually that's not entirely true, since the Enterprise Tools agent  
uses it.
The current limitation is that you can only increment/decrement  
integers, using the
glib atomic ops functions (so we don't have to use mutexes all over  
the place).
I would really really really (no really) get timing information in  
there, though.

Another area I definitely want to explore when we meet.

> Some of these things (backlog, state machine changes) are already
> present in the current prototype, but they are probably not done
> 'correctly' and are mixed into my branch with other changes. I would
> like to get feedback on whether the above types of things would be
> wanted/needed/acceptable in the chassis/mysql-proto core, and if
> anyone has ideas on how they are best implemented.
>
> My immediate goals:
>
> * backlog in the chassis core

+1

> * hooks for the backend 'objects' and connection pool  'objects'
> whereby a plugin can influence their behavior. This may be something
> like callbacks when adding or removing or requesting current state. I
> see this as essential for the goal of dynamically adding/removing
> backends and limiting connection pool size.

Internally we have talked about moving to a more service oriented  
architecture,
especially concerning cross-plugin communication and services, mainly  
because
there are times when someone develops a plugin and another group wants  
to use it
for entirely different purposes.
Obviously the existing callback infrastructure will not give us that,  
simply because
its focus is the protocol lifecycle. And I think that's a good thing.
Currently, plugins tend to be monolithic blobs piggybacking on the  
protocol lifecycle
callbacks, which I think will bite us in the long run. There's also a  
fair amount of
code duplication going on that makes me itch.
I would be interested in what kind of callbacks we would need and for  
which objects.

Looking forward to Proxy 0.9 (the scripting layer multithreading) this  
will become
important anyway, and it would be nice to come up with a reasonably  
simple yet flexible
way of dealing with it - before diving into release plans,  
committments and dates :)
OTOH we might not be able to go all the way in that release, again  
given that we are
somewhat resource constrained at the moment.

> Obviously the above changes should have no impact, performance or
> otherwise on plugins that do not use them.

In principle I agree, although certain changes will have to be made  
anyway, at least in 0.9.
But since it's still labelled alpha we are allowed to break stuff  
(within reason) :)
As long as the regular lifecycle callbacks stay intact I think we are  
good to change pretty
much everything else at this point.

cheers,
-k
--
Kay Roepke
Software Engineer, MySQL Enterprise Tools

Sun Microsystems GmbH    Sonnenallee 1, DE-85551 Kirchheim-Heimstetten
Geschaeftsfuehrer:    Thomas Schroeder,  Wolfang Engels,  Wolf Frenkel
Vorsitz d. Aufs.rat.: Martin Haering                    HRB MUC 161028