maria-developers team mailing list archive

[Kristian Nielsen <knielsen@xxxxxxxxxxxxxxx>]

Michael Widenius <michael.widenius@xxxxxxxxx> writes:

> To go trough the things that are wrong with mysqltest:

In my opinion, the single biggest problem with mysqltest is that it is a
seperate program from the test suite driver mysql-test-run.pl. It's even
written in a different language (C vs. Perl), and test cases themselves are
written in a third language (the mysqltest language) + a mixture of SQL and
even some shell scripts. That's _5_ different languages in a single
application :-(

This is a disaster for the structure of mysql-test-run. Such a large part of
the code (and complexity) is just trying to handle communication between the
various parts, which is really hard to do well due to different languages.

The first step in any major improvement (in my opinion) should be to
re-implement mysqltest in Perl inside mysql-test-run.pl. But that would be a
project of several months, not something I'm eager to start just now ...

> I like the philosophy behind it:  Write a test in pure SQL and then
> ensure that the generated result doesn't change.  It's easy enough to
> allow any developers to generate a test case.

Yes. This is a very strong point for testing that correct results are produced
by a query or similar. Keeping something like this is very important if
thinking about extending or replacing the test framework.

The weakness with mysqltest / mysql-test-run is not that it does this, it is
that it does not provide additional/separate functinality for other kinds of
tests that do not fit this simple framework well.

> In particular, it's good that tests and results are different files as
> it makes it easier to understand what are wrong when you get a failure.

I am trying to understand what precisely you have in mind here...

One big problem with other test suites I have seen is that if you want to
debug a failure, you first have to spend a lot of time debugging and
understanding the test case code just to understand what the problem actually
is. Getting a diff of queries run and results produced like mysqltest does
makes this step trivial (if the problem is wrong results from a query).

Also the --record functionality for generating .result files can be really
good to ease the production of test case and also make it more likely the
developer will not be tempted to insufficiently check the results (everything
is checked by default). It does have the big risk of not properly checking
that the committed .result is in fact correct, something I have seen more than
once.

This does not really have to mean using separate files. I have in mind
something like a Perl-based suite where I could do:

    query <<SQL, <<RESULT
    SELECT a, b
      FROM t1 JOIN t2 ON (a = b)
     ORDER BY a, b;
    SQL
    RECORDME
    RESULT

and `mysql-test-run --record` would rewrite this as

    query <<SQL, <<RESULT
    SELECT a, b
      FROM t1 JOIN t2 ON (a = b)
     ORDER BY a, b;
    SQL
     a  b
     1  1
     2  2
     4  4
    RESULT

And in case of different result, it could output a diff of just the RESULT
section of the failing query, along with the query. It could also give the
correct line number in the .test file, avoiding the need to search the .test
file for the query that produced the diff. I think that might make it even
easier than separate result file.

Something like that would of course allow putting arbitrary Perl code
in-between running queries (with lots of useful modules available) where this
is more appropriate. This would eliminate the need for the mysqltest language
for new test cases, which is one less language again.

And it would be quite easy to have a backwards-compatible emulation of the old
.test and .result files (just a small wrapper that reads the .test and .result
files and passes the contents to the query() function).

>   One benefit is that you can usually run a test by just piping it
>   trough the normal MySQL client, which simplifies testing and
>   debugging a lot!

Yes, this should be kept for simple stuff, very important.

 - Kristian.