maria-developers team mailing list archive
-
maria-developers team
-
Mailing list archive
-
Message #08104
Re: custom storage backend rnd_next() implementation and caching questions
Hi Sergei,
Thanks for the quick reply.
1. I have tried your suggestion (which is a cool simplification to my bit-messing… ;). Unfortunately the code is crashing now:
mariadb’s crash report shows the following stack trace:
stack_bottom = 0x9e5de314 thread_stack 0x30000
mysys/stacktrace.c:246(my_print_stacktrace)[0x89912c5]
sql/signal_handler.cc:153(handle_fatal_signal)[0x841593a]
[0xb76f5c58]
[0xb76f5c7c]
/lib/i386-linux-gnu/libc.so.6(gsignal+0x47)[0xb70e6607]
/lib/i386-linux-gnu/libc.so.6(abort+0x143)[0xb70e7d83]
/lib/i386-linux-gnu/libc.so.6(+0x27757)[0xb70df757]
/lib/i386-linux-gnu/libc.so.6(+0x27807)[0xb70df807]
sql/field.cc:6876(Field_varstring::store(char const*, unsigned int, charset_info_st const*))[0x840032c]
/usr/local/mysql/lib/plugin/ha_fsview.so(_ZN9ha_fsview8rnd_nextEPh+0xec)[0xb70adb00]
sql/handler.cc:2553(handler::ha_rnd_next(unsigned char*))[0x841b003]
sql/records.cc:465(rr_sequential(READ_RECORD*))[0x852cf50]
sql/sql_select.cc:18532(join_init_read_record(st_join_table*))[0x82adff6]
sql/sql_select.cc:17639(sub_select(JOIN*, st_join_table*, bool))[0x82ac256]
sql/sql_select.cc:17304(do_select)[0x82abb38]
sql/sql_select.cc:3079(JOIN::exec_inner())[0x82895c3]
sql/sql_select.cc:2369(JOIN::exec())[0x8286b3a]
sql/sql_select.cc:3307(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x8289e70]
sql/sql_select.cc:372(handle_select(THD*, LEX*, select_result*, unsigned long))[0x8280acc]
sql/sql_parse.cc:5269(execute_sqlcom_select)[0x8258632]
sql/sql_parse.cc:2552(mysql_execute_command(THD*))[0x8250ee9]
sql/sql_parse.cc:6415(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x825aa1f]
sql/sql_parse.cc:1307(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x824e3db]
sql/sql_parse.cc:1004(do_command(THD*))[0x824d7fd]
sql/sql_connect.cc:1379(do_handle_one_connection(THD*))[0x8358b06]
sql/sql_connect.cc:1293(handle_one_connection)[0x835887c]
/lib/i386-linux-gnu/libpthread.so.0(+0x6f16)[0xb75fff16]
/lib/i386-linux-gnu/libc.so.6(clone+0x5e)[0xb71a3a3e]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x9891f840): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
Apparently it seems that Field_varchar::store is somehow unhappy.
I am using 10.0.14 sources, the given function is:
int Field_varstring::store(const char *from,uint length,CHARSET_INFO *cs)
{
ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED;
uint copy_length;
const char *well_formed_error_pos;
const char *cannot_convert_error_pos;
const char *from_end_pos;
copy_length= well_formed_copy_nchars(field_charset,
(char*) ptr + length_bytes,
field_length,
cs, from, length,
field_length / field_charset->mbmaxlen,
&well_formed_error_pos,
&cannot_convert_error_pos,
&from_end_pos);
if (length_bytes == 1)
*ptr= (uchar) copy_length;
else
int2store(ptr, copy_length);
if (check_string_copy_error(this, well_formed_error_pos,
cannot_convert_error_pos, from + length, cs))
return 2;
return report_if_important_data(from_end_pos, from + length, TRUE);
}
(More specificallly the problematic line seems to be the well_formed_copy_nchars() invocation in this function).
I am a bit puzzled about how this stuff is working. I mean i haven’t told the table struct buf pointer value from rnd_next(). How can the field class compute, where he should put the data?
Thanks again for the quick help
Andras
--
Andras Szabo
Sent with Airmail
On 30 Jan 2015 at 17:19:21, Sergei Golubchik (serg@xxxxxxxxxxx) wrote:
Hi, Andras!
On Jan 30, Andras Szabo wrote:
> Hi Everyone,
>
> I am developing a custom mariadb storage backend which is intended to
> give filesystem related information via mariadb tables.
> For example one could tell:
> CREATE TEMPORARY TABLE IF NOT EXISTS `/var/log` ENGINE=fsview;
> which would create a table (using the assisted discovery method) with the following schema:
Cool!
> 1. I am having trouble implementing rnd_next() in the backend:
>
> int ha_fsview::rnd_next(uchar *buf) {
> DBUG_ENTER(__PRETTY_FUNCTION__);
> if (!dirp) {
> DBUG_RETURN(HA_ERR_INTERNAL_ERROR);
> }
> if (struct dirent* entry = readdir(dirp)) {
> uchar *pos = buf;
> *pos = 0;
> ++pos;
> // file name (1bytes size + str)
> *pos = strlen(entry->d_name);
> ++pos;
> size_t bytes_to_copy = strlen(entry->d_name);
> memcpy(pos, entry->d_name, bytes_to_copy);
> pos+=bytes_to_copy;
> DBUG_PRINT("info", ("fsview filename: %s", entry->d_name));
> // content (4 bytes size + content ptr)
> static const std::string fake_content = "hello world";
> int s = fake_content.size();
> memcpy(pos, &s, sizeof(s));
> pos+=sizeof(s);
> const char* c = fake_content.c_str();
> memcpy(pos, &c, sizeof(c));
> pos+=sizeof(c);
> DBUG_RETURN(0);
> } else {
> DBUG_RETURN(HA_ERR_END_OF_FILE);
> }
> }
>
> My problem is that filenames (first field) is ok, but i cannot really
> see the expected content “hello world” in the second field. I am
> assuming that i am doing something wrong, but i cannot really find the
> reason. Could somebody help me, please?
This looks correct to me. May be, perhaps. std::string->c_str() isn't
returning the pointer to your static string (but to a some temporary
copy that gets invalidated on return)? I don't know how std::string is
supposed to work in this case.
Anyway it might've been easier to use Field methods than assembling the
row directly. Like
table->field[0]->store(entry->d_name, strlen(entry->d_name), system_charset_info);
table->field[1]->store(STRING_WITH_LEN("hello world"), system_charset_info);
> 2. If i am issuing:
>
> SELECT * FROM `/var`;
>
> twice, then the server is not initiating a table scan again on my
> backend (most likely results from the first run are stored in some
> cache). So if filesystem contents are changed between the two
> statements, then it won’t be reflected in the resultset.
>
> Of course if i do a:
>
> SELECT SQL_NO_CACHE * FROM `/var`;
>
> then everything is fine, however i would not like to put the burden on
> the user to specify the option to not cache the results.
>
> My question is whether it’s possible to specify in the storage backend
> that the results from the tables should not be cached.
Yes. See handler::register_query_cache_table().
Regards,
Sergei
Follow ups
References