maria-developers team mailing list archive

[Patrick Crews <gleebix@xxxxxxxxx>]

Hi everyone,
As a QA engineer, I had a few thoughts I would put into the discussion as
well.  Please feel free to take them or leave them (or to throw them a
parade and give them the key to the city) ;-)

On Mon, Apr 6, 2009 at 3:17 PM, Kristian Nielsen
<knielsen@xxxxxxxxxxxxxxx>wrote:

> Michael Widenius <michael.widenius@xxxxxxxxx> writes:
>
> > To go trough the things that are wrong with mysqltest:
>
> In my opinion, the single biggest problem with mysqltest is that it is a
> seperate program from the test suite driver mysql-test-run.pl. It's even
> written in a different language (C vs. Perl), and test cases themselves are
> written in a third language (the mysqltest language) + a mixture of SQL and
> even some shell scripts. That's _5_ different languages in a single
> application :-(
>
> This is a disaster for the structure of mysql-test-run. Such a large part
> of
> the code (and complexity) is just trying to handle communication between
> the
> various parts, which is really hard to do well due to different languages.
>
> The first step in any major improvement (in my opinion) should be to
> re-implement mysqltest in Perl inside mysql-test-run.pl. But that would be
> a
> project of several months, not something I'm eager to start just now ...


I agree that more 'moving parts' = more overhead.  I can't say that I agree
with re-implementing in this fashion, but now would be a good time to see
how we can make things leaner / more efficient / more to-the-point (ie we
can focus on using the tool to test and not 'keeping the parts in harmony')

>
>
> > I like the philosophy behind it:  Write a test in pure SQL and then
> > ensure that the generated result doesn't change.  It's easy enough to
> > allow any developers to generate a test case.
>
> Yes. This is a very strong point for testing that correct results are
> produced
> by a query or similar. Keeping something like this is very important if
> thinking about extending or replacing the test framework.
>
> The weakness with mysqltest / mysql-test-run is not that it does this, it
> is
> that it does not provide additional/separate functinality for other kinds
> of
> tests that do not fit this simple framework well.
>

I very much like the straight-up SQL-> result paradigm *for certain tests*.
 This is GREAT for a basic, functional suite (does SELECT work.  Are INSERT
DEFAULT and INSERT () interchangeable <in certain scenarios>).  However,
getting into more 'scenario-based' testing, there are problems -
coordination between 'users' in a test, tools for greybox-manipulation
(tweaking the software mechanics beyond just INPUT/OUTPUT testing), etc.

It would be great to keep the simplicity for certain scenarios, but to have
extensible power for more complicated tests when needed.  One issue I have
seen with certain proposed replacements is that they allow complex testing,
but the overhead of that power results in even the most simple tests
becoming a monster (the overhead of defining a test).

> In particular, it's good that tests and results are different files as
> > it makes it easier to understand what are wrong when you get a failure.
>
> I am trying to understand what precisely you have in mind here...
>
> One big problem with other test suites I have seen is that if you want to
> debug a failure, you first have to spend a lot of time debugging and
> understanding the test case code just to understand what the problem
> actually
> is. Getting a diff of queries run and results produced like mysqltest does
> makes this step trivial (if the problem is wrong results from a query).
>
> Also the --record functionality for generating .result files can be really
> good to ease the production of test case and also make it more likely the
> developer will not be tempted to insufficiently check the results
> (everything
> is checked by default). It does have the big risk of not properly checking
> that the committed .result is in fact correct, something I have seen more
> than
> once.
>
> This does not really have to mean using separate files. I have in mind
> something like a Perl-based suite where I could do:
>
>    query <<SQL, <<RESULT
>    SELECT a, b
>      FROM t1 JOIN t2 ON (a = b)
>     ORDER BY a, b;
>    SQL
>    RECORDME
>    RESULT
>
> and `mysql-test-run --record` would rewrite this as
>
>    query <<SQL, <<RESULT
>    SELECT a, b
>      FROM t1 JOIN t2 ON (a = b)
>     ORDER BY a, b;
>    SQL
>     a  b
>     1  1
>     2  2
>     4  4
>    RESULT
>
> And in case of different result, it could output a diff of just the RESULT
> section of the failing query, along with the query. It could also give the
> correct line number in the .test file, avoiding the need to search the
> .test
> file for the query that produced the diff. I think that might make it even
> easier than separate result file.
>
> Something like that would of course allow putting arbitrary Perl code
> in-between running queries (with lots of useful modules available) where
> this
> is more appropriate. This would eliminate the need for the mysqltest
> language
> for new test cases, which is one less language again.
>
> And it would be quite easy to have a backwards-compatible emulation of the
> old
> .test and .result files (just a small wrapper that reads the .test and
> .result
> files and passes the contents to the query() function).


I just gave a MySQL University session on some of these problems.  One of
the issues I have seen is that people don't expend the extra energy to make
a test informative (through comments explaining the test, defining what a
'real' <not in setup / teardown, but in the behavior being tested> failure
would look like / what 'next-steps' seem logical for diagnosing a failure,
etc.
http://forge.mysql.com/wiki/How_to_Create_a_Test_Case

I do like the thought of defining query / result portions a LOT.  One of the
biggest time-sinks we have is sifting through more complicated tests and
piecing together what is going on.

I think a properly revised test-suite would help if it presented the ability
to capture much more information:
Test purpose
Defining key portions of the test - maybe defining code blocks of a test as
setup / teardown / and testcode.  Each testcode portion could have things
like  "on_failure - If the test fails here, it is a problem with xyz."  Each
'real' portion of the test could define this.  Failures in setup / teardown
are still bad, but they might point to a less-serious failure and could be a
Build host issue (full filesystem, etc).

Probably the biggest wishes for a testing tool I have are:
Power
Simplicity / Elegance (make it easy to do anything, and allow complex
'thoughts' to be spelled out in an easily understood way)
Ease of comprehending a test (opening it up and understanding what is being
tested, how it is being done, and why a method has been chosen)
Speed of failure analysis. (seeing a failure and being able to quickly
discern what failed and what kind of a problem it is pointing to).

Some extra wishlist items:
Ease of cataloging / cross-referencing tests (ie run all 'Innodb' related
tests and not having this rely solely on suite organization / naming
patterns)
Execution speed (we are likely to expand the test suite, so ensuring a lean
run-time is always nice)

Anyway, I saw the topic and couldn't resist chiming in.

Thanks,
Patrick.


>
> >   One benefit is that you can usually run a test by just piping it
> >   trough the normal MySQL client, which simplifies testing and
> >   debugging a lot!
>
> Yes, this should be kept for simple stuff, very important.
>
>  - Kristian.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~maria-developers
> Post to     : maria-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~maria-developers
> More help   : https://help.launchpad.net/ListHelp
>