pbxt-discuss team mailing list archive

[Paul McCullagh <paul.mccullagh@xxxxxxxxxxxxx>]

Hi Stewart,

On May 24, 2010, at 5:26 AM, Stewart Smith wrote:

> On Fri, 21 May 2010 11:31:20 +0200, Paul McCullagh <paul.mccullagh@xxxxxxxxxxxxx 
> > wrote:
> > My problem is that the val_ methods are hard-coded to reference
> > record[0].
>
> You can get around that by using Field::move_field_offset.
>
> something like:
>
>  ptrdiff_t row_offset= buf - table->record[0];
>  (**field).move_field_offset(row_offset);
>  (do things with field)
>  (**field).move_field_offset(-row_offset);
>
> if buf is record[1] then we'll be doing things with it.

Because I need to access row buffers internally, this does not work  
for me, because the Table object is shared amongst threads, so I  
cannot set the ptr inside the fields.

I have also now switched to the TableShare object, instead of the  
Table object, because the are easier to create and destroy.

TableShare has no record[], instead of record[0] you have to share- 
>use default_values.

> > So I have a problem when the row buffer is an engine internal buffer
> > (not record[0]). In this case I need the offset to calculate where  
> > the
> > field data starts in the internal row buffer.
>
> the above should work... although it would be way better if we fix  
> it up
> to be an absolute pointer instead of doing (faulty) pointer math.

As I say, this does not work if your table object is shared amongst  
threads. I don't really want to have to create one of these objects  
per thread.

> > From this point of view, expanding the dependency on record[] would
> > be the wrong way to go.
> >
> > Instead, I suggest a "Row" object, into which I can plug the start of
> > the row buffer. Then we would have:
> >
> > doInsertRecord(Row *row_object)
> >
> > The Row object knows everything about the row and can be used to
> > access the fields.
>
> I've been thinking of a Tuple object instead, so we don't always  
> have to
> have the entire row buffer around if only a subset of rows are  
> operated
> on (this could be more advantageous for column based engines).

I guess you mean a "subset of columns".

I guess by "tuple" you just mean an object where I can get and set  
field values but don't know how they are stored. Or do you mean  
something else?

> I'm also thinking that Tuple would just be an interface, and engines
> could provide their own implementation.
>
> With accessor methods, then the main server code could examine rows in
> their native format instead of always having to convert from engine to
> server formats. We would then only convert rows to over-the-wire
> (server) format when they go over the wire.

This would mean the engine would supply the storage for the row. This  
may indeed be more efficient.

The engine already provide storage for the BLOBs, so I guess it would  
not be a problem if the tuple has the same scope as BLOBs today (which  
means the tuple is valid until the next row is retrieved on the cursor).

But, how would you then handle doInsertRow()?

The natural way to handle this would be for the front-end to provide  
the tuple to be inserted.

Would you add a call: getTuple(), which returns an empty tuple, and  
the call setField() for each column on the tuple and then call  
doInsertRow(tuple)?

Seems rather awkward to me. Or do you have a different idea?

> > So the Row object is a "per thread" object. You call row_object-
> > > setBuffer(u_char *buf) to set the row buffer and then you use an
> > array of fields to get and set the data in the buffer.
>
> We could do RowBuffer(row_buffer, ptr) instead, so that it's  
> impossible
> to ever forget to set the buffer back to something.

I am not sure what you mean here. Is RowBuffer() a constructor? What  
is row_buffer, and what is ptr in this case?

> > So then the record[] array would become an array of Row objects. And
> > an engine could create Row object to manipulate row buffers  
> > internally
> > (which is exactly what I am doing with the internal Table object - or
> > TableShare object).
>
> I think a Row object is just formalising the existing interface a bit
> more... which could be a good intermediate step.

Yes, this is true.

> I like the Tuple idea a
> bit better for a number of reasons, most of which are just different
> ways of expressing "things more compact in memory" and "save  
> converting
> row/tuple formats".
>
> One thing to think about is that Field is not part of record[], and it
> would sort of have to be part of Row to be sane and be able to have
> different Row objects accessed concurrently (or wrap Field calls to
> get/set the offset).

Actually I see Field as part of record[] today by the fact that 'ptr'  
references the record[0] buffer.

Getting a field to reference any other buffer is a fudge at the moment.

But otherwise, you are right, my suggestion would mean that this  
binding between row buffer and Field becomes even more close.

Although, with setBuffer() as I propose above, all fields can be set  
to reference a different buffer.

This is the same as move_field_offset(), but instead of changing one  
field, you change all fields in the row at once.

So there are 2 possibilities:

1. Keep one set of fields, as it is at the moment, and switch from  
record[0] to record[1] as required, or
2. Turn record[] into an array of Row object where each row reference  
a different row buffer (then no switching is required).

There is a 3rd possibility, which will probably require a lot of code  
change:

- Provide the row buffer, on each call to a Field method. In this  
case, the Row object would be buffer independent, and could be used by  
multiple threads.

> > The record[0] object would be passed to doInsertRecord(buf), as it is
> > today.
> >
> > But now, the engine will not have to know that: buf == record[0]!
>
> mostly :)
>
> That's the annoying bit - it's *mostly* true :)

Yup, and it is therefore just as well that I have never relied on  
this. I have always used the field offset, and the buffer pointer to  
get the field data.

> > Stewart: would something like this also work for your implementation
> > of the Embedded InnoDB?
>
> I'd prefer the Tuple way of doing things, as when operating on a  
> subset
> of columns, it would save a bunch of effort.

This is fine, as long as Drizzle provides some sort of interface to do  
the following:

- Get the collation sequence of a column.
- Do string comparison operations with field data, and the collation  
sequence
- Do comparison of decimal encoded values, and other special data types


--
Paul McCullagh
PrimeBase Technologies
www.primebase.org
www.blobstreaming.org
pbxt.blogspot.com