maria-developers team mailing list archive

[Sergei Golubchik <serg@xxxxxxxxxxx>]

Hi, Pablo!

First: please always cc: maria-developers@xxxxxxxxxxxxxxxxxxx,
don't send emails to me only.

Now:

On Mar 11, Pablo Estrada wrote:
> Hello!
> I am getting ready to submit my proposal. Is there any feedback for my
> application document? Anything that would be good to add, remove, rephrase,
> etc?

Go on and submit it. There will be a comment form under your proposal,
if I will want you to change something, I will leave my comments there.

Please mention what tools/languages you think you'll be using.

> Also, if I could see the data available (the history of test results), I
> could write more in detail : )

Hm. Okay. Attached is the preprocessed table that I used two years ago.
It lists test failures in buildbot in a given period of time.
Columns are (as far as I remember):
* time of the test run start
* code branch that was tested
* revision id that was tested
 - it's fake, our buildbot tables don't store revision id. I've
   constructed it from the set of changed files, revision number, etc.
   hopefully it's as unique as a real revision id
* build number
* platform where the tests were run
* test type
 - basically, the set of tests to run and what protocol to use. one test
   may be run with different protocols
* test name
* test variant
 - or combination. certain tests may be run in many combinations

Using tis file I tried to rearrange test execution order for every
specific build number (using failure data from previous builds in the
same table) to detect test failures as early as possible. I managed to
have 90% of all test failures happening within first 10% of the tests.

The goal of this project is to have something we can put back into
buildbot, so that it could actually run only 10% of the tests, still
having 90% probability of discovering a failure (or 20%, or 95%,
whaveter).

See also http://buildbot.askmonty.org/buildbot/
you can click on a link for any branch, for example,
http://buildbot.askmonty.org/buildbot/grid?branch=10.0
then on any particular build, for example,
http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297
and on a stdio log for a particular test run:
http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/steps/test_6/logs/stdio

on these pages you'll see what a platform (builder) is, build number, test
types, combinations, etc.

Regards,
Sergei

Attachment: test_failures.in.bz2
Description: BZip2 compressed data