← Back to team overview

pbxt-discuss team mailing list archive

Re: goal 0 of embedde pbxt reached || (was: Re: PBXT: Embedded Database (library))

 

Hi Martin,

(If possible it would be great if you could switch to text e-mails. I find quoting is strange and fonts are small in HTML e-mails.)

On Feb 12, 2010, at 12:44 AM, Martin Scholl wrote:


- In which format shall we store the table/db definitions? protobuff maybe? Afair Drizzle does so, so we could borrow some code there...

protobuf may be an overkill for the initial implementation.

How are you planning to do create table? The innodb API does it by building a create table structure with various API calls.

By submitting a CREATE TABLE statement as text, you can save a lot of API routines.

PBXT already has a parser for CREATE (and ALTER) table statements. So, you could accept the text and feed the parser.

Then, you could actually store the table definition as a CREATE TABLE statement. When the table is loaded you just invoke the parser. The CREATE TABLE text could be stored in a separate file, like the .frm file, for each table.

Alternatively the text could be stored in the header of the .xtd file, where I already store the foreign key information (the foreign key information is actually stored as SQL text).

However, this may be going too far with the integration of the embedded code and PBXT itself. I like this idea really really much -- its a nice separation and self-hosting feature also. I'd guess, the lexer code of mysql would have to be ported over then? If I remember things correctly, lexer and THD are coupled which would complicate things even further.

No, PBXT can do everything:

XTDDTable *xt_ri_create_table(XTThreadPtr self, bool convert, XTPathStrPtr tab_path, char *sql, XTDDTable *my_tab, struct XTDictionary *source_dic);

This function takes a string of SQL and converts it to a table (XTDDTable) structure.


Basically, what would be cool is if the embedded wrapper code controls the following:

1. The types of data stored.
  - We can start with a few very basic types.
2. The format of a record in RAM
- This is the same format that PBXT uses on disk, as long as the records are fixed length - For variable length records it uses a simple serialization method (as I mentioned before)
3. The format of index records
- with an interface to get and set data in a row, the engine does not need to actual format
4. The comparison of data types
  - the wrapper provides routines to compare data types.
- These are mostly methods which are part of the data dictionary in RAM
5. The format of the data dictionary on disk, and in RAM
  - the wrapper reads and writes this data.

This will give us great flexibility to add data types and other complexities later. Sounds great. Especially when having a data type registry for supplying custom data types at runtime.

Yes, that would then work.


It is also pretty much the division of work between MySQL code and PBXT today. However, the division is not so clear in the code.


- else, should table serialization / deserialization be pluggable or even be purely programmatic? I am fine with this, too, as it is an _embedded_ library and I'd guess most people will control pbx programmatically anyways

Although I spoke mainly about the textual interface above, I am really flexible on this. I think both solutions have there advantages.

Use whichever is best and easiest for you at the moment, which may be simply writing your own stuff! :) I guess hardcoding table definitions will be the most easy way for now.

OK, so the wrapper class would just know the format of the tables?

Best regards,

Paul


--
Paul McCullagh
PrimeBase Technologies
www.primebase.org
www.blobstreaming.org
pbxt.blogspot.com




Follow ups

References