randgen team mailing list archive

[Matthias Leich <matthias.leich@xxxxxxxxxx>]

Hi Roel,

Am 31.12.2010 00:37, schrieb Roel Van de Paar:
>  Hi!
> 
> I am interested to know if there was ever(/is) an "all-inclusive" test
> (grammar/config file) created for testing MySQL with RQG, or something
> that came close to it.

I do not know of such a grammar.

> 
> With "all-inclusive" I mean a test (grammar/config file) that test every
> area/aspect of the server (or at least many/several), but from one test
> instead of many different tests spread across many files.

IMHO my WL5004 grammar monster could be called the most complete SQL
syntax/SQL scenario check from all RQG grammars we have.
But this is also far away of what you are thinking of because
- generated syntax + scenarios is huge but nevertheless incomplete
  example of missing stuff: ALTER TABLE for partitioned tables
                            some replication related commands ...
                            whatever extreme complex WHERE/JOIN/...
- mostly coarse grained checking = search for assert+crash+deadlock
  but no checks of
  - result sets
  - optimizer strategies ...
  - correct behavior around concurrent transactions
    Do we get exact the modifications we ordered?
    Has every concurrent session seen what it was allowed to see,
    not committed stuff etc.?

I have huge doubts if the development and use of such an "all-inclusive" grammar
is effective. The different areas cause different more or less contradicting
requirements which leads to complexity and more or less nice compromises.
With effective I refer to the
- one time value of a tests
  Find a bunch of bugs during test development.
- ongoing value of the test
  Avoid that new or old bugs show up by applying this test frequent to permanent
  modified source trees.
in relation to the
- one time development expenses
- ongoing usage expenses (resource consumption on test box farm) + more or less
  manual result analysis
- ongoing maintenance expenses
You usually end up with
a) test a bit super thorough and effective (expenses in development and maintenance/execution)
b) test a lot only coarse grained but effective (expenses ....)
c) any attempt to inflate tests like type a) or b) above with more statements/scenarios
   or more thorough checking leads to exponential expense growth.
   This is no problem at all as long as the number of areas checked within one grammar is small.
IMHO test development starts with hopefully some "try to avoid to create obstacles which
clear prevent a SUPER c in mind".
Than the first version of the test is something between a) and b).
Than you walk some time into direction c).
And the *art* is to stop this walk/give up before you become inefficient on your way
into direction c).

If you just look for inspiration:
- WL5004_* grammars try to avoid the generation of
  - plain wrong SQL syntax
  - wrong statement for wrong object like CREATE INDEX on a view
- performance_schema.yy reduces "table does not exist" to a significant extend
- WL5092_*, replication-ddl_ tries to avoid to generate scenarios which does not
  fit to the current binlog_format  (tests most probably broken since some time)
- some grammars created by Philip
  Check thorough ...
  Tricky solutions for ...
- grammars by Patrick for optimizer checking
  I had only a short view into these tests. He seems to have his own battle which is
  mostly outside of the focus I had within my RQG tasks.
  But I am sure that he had to figure out some great solutions.
I apologize in advance if I maybe characterize any solution found by anybody not
per se the best possible solution though it is reality the best one.
Most probably many of the solutions above are the product of the first attempt to get
some frequent and annoying problem under control.
So they
- are maybe not the best thinkable solutions
- maybe cannot coexist with solutions for other important problems
but they show
- that something isn't impossible, a solution could be found at all
- some solution which was sufficient good enough under some specific scenario
Just use all this as quarry. Reuse whatever code or idea and maybe detect that
it is unusable, could be drastic improved ....


Regards

Matthias


> 
> Kind Regards,
> God Bless,
> -- 
> Roel Van de Paar, Senior Technical Support Engineer
> MySQL @ Oracle Corporation, Asia-Pacific
> http://www.mysql.com
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~randgen
> Post to     : randgen@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~randgen
> More help   : https://help.launchpad.net/ListHelp


-- 
Matthias Leich
Senior QA Developer
Office: +49 30 4764614

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Rijnzathe 6, 3454PV De Meern, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Jürgen Kunz, Marcel van de Molen, Alexander van der Ven