maria-developers team mailing list archive

[Michael Widenius <monty@xxxxxxxxxxxx>]

Hi!

>>>>> "Patrick" == Patrick Crews <gleebix@xxxxxxxxx> writes:

<cut>

>> This does not really have to mean using separate files. I have in mind
>> something like a Perl-based suite where I could do:
>> 
>> query <<SQL, <<RESULT
>> SELECT a, b
>> FROM t1 JOIN t2 ON (a = b)
>> ORDER BY a, b;
>> SQL
>> RECORDME
>> RESULT
>> 
>> and `mysql-test-run --record` would rewrite this as
>> 
>> query <<SQL, <<RESULT
>> SELECT a, b
>> FROM t1 JOIN t2 ON (a = b)
>> ORDER BY a, b;
>> SQL
>> a  b
>> 1  1
>> 2  2
>> 4  4
>> RESULT
>> 
>> And in case of different result, it could output a diff of just the RESULT
>> section of the failing query, along with the query. It could also give the
>> correct line number in the .test file, avoiding the need to search the
>> .test
>> file for the query that produced the diff. I think that might make it even
>> easier than separate result file.

The reason I don't like the above, compared to have different files
are:

- We may want to run the test with different engines and different
  configure options which would produce different results.
  In this case, having different files, one for each combination that
  gives a different result, is much easier than having the result in
  the test case.

- Most of the current test cases you can pipe through 'mysql' and then
  compare the result files;  A very good way when you are trying to
  find out what's wrong.  If everything is in the same file, things
  like this gets to be cumbersome.

What I *would* like to change ASAP is that the test file and result
file is in the same directory so that in 'bzr gcommit' and in 'emacs
directory' the test and result files comes directly after eachother!
This has been on the mysqltest TODO for a long time but for some
reason never get done, even if it's a trivial change!

>> Something like that would of course allow putting arbitrary Perl code
>> in-between running queries (with lots of useful modules available) where
>> this
>> is more appropriate. This would eliminate the need for the mysqltest
>> language
>> for new test cases, which is one less language again.

We can alread do this.

>> And it would be quite easy to have a backwards-compatible emulation of the
>> old
>> .test and .result files (just a small wrapper that reads the .test and
>> .result
>> files and passes the contents to the query() function).

Patrick> I just gave a MySQL University session on some of these problems.  One of
Patrick> the issues I have seen is that people don't expend the extra energy to make
Patrick> a test informative (through comments explaining the test, defining what a
Patrick> 'real' <not in setup / teardown, but in the behavior being tested> failure
Patrick> would look like / what 'next-steps' seem logical for diagnosing a failure,
Patrick> etc.
Patrick> http://forge.mysql.com/wiki/How_to_Create_a_Test_Case

Agree this is a common problem.

<cut>

Patrick> I think a properly revised test-suite would help if it presented the ability
Patrick> to capture much more information:

I don't see a big reason to do 'totally revised' (ie, rewritten) test-suite.
There is however a great need to simplify the current architecture and
have another test system for performance and 'heavy duty' testing.

Patrick> Test purpose
Patrick> Defining key portions of the test - maybe defining code blocks of a test as
Patrick> setup / teardown / and testcode.  Each testcode portion could have things
Patrick> like  "on_failure - If the test fails here, it is a problem with xyz."  Each
Patrick> 'real' portion of the test could define this.  Failures in setup / teardown
Patrick> are still bad, but they might point to a less-serious failure and could be a
Patrick> Build host issue (full filesystem, etc).

I see the point of this, but fear that simple tests would be made to
complex.

In practice we already have this parts in most tests, but it's not
properly formalized or clear in all cases.


Patrick> Probably the biggest wishes for a testing tool I have are:
Patrick> Power
Patrick> Simplicity / Elegance (make it easy to do anything, and allow complex
Patrick> 'thoughts' to be spelled out in an easily understood way)

The phrase 'If we would smart enough to understand our brains, our
brains would not be smart enough to understand it'.
In other words, it's hard to do a complex problem simple.
Do you think we can come up with one tool to do this?

Patrick> Ease of comprehending a test (opening it up and understanding what is being
Patrick> tested, how it is being done, and why a method has been chosen)

Most of the tests are done to:

- Test that a feature works according to specification
- Test that a bug is fixed
- To get more code covered

For most of the above, it would be hard to get people to specify why a
method was choosen.

How do you envision this part?
(Not beeing negative, just trying to understand)

Patrick> Speed of failure analysis. (seeing a failure and being able to quickly
Patrick> discern what failed and what kind of a problem it is pointing to).

Which is not an easy task in itself.

Patrick> Some extra wishlist items:
Patrick> Ease of cataloging / cross-referencing tests (ie run all 'Innodb' related
Patrick> tests and not having this rely solely on suite organization / naming
Patrick> patterns)

Agree.

Patrick> Execution speed (we are likely to expand the test suite, so ensuring a lean
Patrick> run-time is always nice)

That's one of the bigest challanges.

Patrick> Anyway, I saw the topic and couldn't resist chiming in.

You are welcome!
It's a very important topic.

Regards,
Monty