randgen team mailing list archive

["Philip Stoev" <philip@xxxxxxxxx>]

Everyone,

I will be spending some time trying to make the RQG more user friendly and 
less prone to silent failures that render entire tests useless without much 
warning. To this end, here is what is on my mind:

- adopt Bernt's MTR --start-and-exit replacement and remove all use of MTR 
altogether forever and ever;
- make the RQG print out textual descriptions for all error codes (already 
implemented);
- properly report errors during the data loading phase. In particular, 
properly report crashes and any errors that prevented tables from being 
created and data from being inserted;
- properly report errors that caused the test to become meaningless, such as 
zero valid queries being generated in the entire test;
- report better statistics for the test while it is running:
* report grammar rules that are broken and produced no valid queries 
(already implemented);
* report final table count and table status (a distilled version of SHOW 
TABLE STATUS showing row count and data size)
* a concise report for the % of successfull queries, % of queries that 
modified rows and % that returned 1 or more rows, to provide a single-line 
information on the health of the grammar
- enable easy creation of lcov reports (already implemented)

The goal would be to have a framework that is not significantly more 
"dangerous" to use than MTR, that is, to have no increased risk of creating 
a meaningless test that reports a false negative sucess for every run.

I have not reviewed Matthias' suggestions from the past, but if you have 
previously suggested an improvement in the direction of usability that has 
been forgotten in the meantime, please remind me.

Thank you.

Philip Stoev