maria-developers team mailing list archive

[Zardosht Kasheff <zardosht@xxxxxxxxx>]

I've worked on the TokuDB storage engine for quite a while now. I have
had many experiences over the years, so I guess it's hard to know
where to begin. I guess I will start small, and if the conversation
evolves, I can contribute more thoughts. I think the current API is
really good, as evidenced by the fact that many storage engines have
used it to plug into MySQL. The two areas that I see we can really
benefit from are the following:
 - documentation of the API, discussing clearly what the contracts for
the functions are. There are still several functions I am not fully
clear on, but am glad that they "just work". This includes
documentation on the locking mechanisms. For instance, clear
documentation on what external_lock, store_lock, and other related
functions do would be great.
 - general API cleanup. In too many places, the functions just assume
things, such as the buffer passed into write_row is the same as
table->record[0]. In other instances, the well defined way to answer
certain questions is not obvious. For instance:
  - If I want information on the fields, do I use the table object or
table_share?
  - There seem to be several ways to determine if a field is NULL
  - There is redundant information in the KEY* structure and the
fields that make up the KEY. I recall when working on MySQL 5.1, this
information was inconsistent and it led to bugs, such as MySQL bug
37292
  - I am sure there are plenty of other instances of this that I can find.
 - Transactions. The API seems to not be designed with transactions
taken into account. Transactions are started in some cases of
external_lock, or in start_stmt. Transactions are committed either in
the handlerton's commit function or in some cases in external_lock. A
clear API that takes transactions into consideration would be quite
helpful.

Those are my early thoughts. I am happy to share more if people are interested.

-Zardosht

On Tue, Aug 13, 2013 at 6:41 PM, MARK CALLAGHAN <mdcallag@xxxxxxxxx> wrote:
> The storage engine API has never been my favorite part of MySQL. I haven't
> written a storage engine but I used to maintain a custom storage engine and
> migrated that from 4.0 to 5.0 so I had some experience with the API.
>
> Is there a different API that would make it easier for innovation to come to
> MySQL/MariaDB? For example there is a lot going on in the write-optimized
> database space. Until recently we had nothing for MySQL open-source. Now we
> have open source TokuDB (great news). I haven't read their source but I have
> been told they use something that was derived from the BerkeleyDB API.
>
> Would the BerkeleyDB API make it easier to get new storage engines into
> MySQL? For example, we don't have an engine optimized for in-memory
> workloads. And while InnoDB will have great cache hit rates when all data is
> in-memory. It might write much more data to disk than required.
>
> Do we need a better API or is the current code good enough? I hope to hear
> from people who have done serious storage engine work.
>
> --
> Mark Callaghan
> mdcallag@xxxxxxxxx
>
> _______________________________________________
> Mailing list: https://launchpad.net/~maria-developers
> Post to     : maria-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~maria-developers
> More help   : https://help.launchpad.net/ListHelp
>