maria-developers team mailing list archive

[Pablo Estrada <polecito.em@xxxxxxxxx>]

Hello Sergei and all,
I have made a last update to my proposal, and submitted it. I added more
specifics regarding the scope of the project (running as part of the
buildbot), as well as data to leverage for the project and potential
learning algorithms to use depending on the data that the project will have
access to in the end (only test result history, or also code change
history, etc.).

If anyone has any changes to suggest before the deadline, let me know. I'll
be glad to look into them.
All the best, all : )

Pablo


On Tue, Mar 11, 2014 at 6:53 PM, Sergei Golubchik <serg@xxxxxxxxxxx> wrote:

> Hi, Pablo!
>
> First: please always cc: maria-developers@xxxxxxxxxxxxxxxxxxx,
> don't send emails to me only.
>
> Now:
>
> On Mar 11, Pablo Estrada wrote:
> > Hello!
> > I am getting ready to submit my proposal. Is there any feedback for my
> > application document? Anything that would be good to add, remove,
> rephrase,
> > etc?
>
> Go on and submit it. There will be a comment form under your proposal,
> if I will want you to change something, I will leave my comments there.
>
> Please mention what tools/languages you think you'll be using.
>
> > Also, if I could see the data available (the history of test results), I
> > could write more in detail : )
>
> Hm. Okay. Attached is the preprocessed table that I used two years ago.
> It lists test failures in buildbot in a given period of time.
> Columns are (as far as I remember):
> * time of the test run start
> * code branch that was tested
> * revision id that was tested
>  - it's fake, our buildbot tables don't store revision id. I've
>    constructed it from the set of changed files, revision number, etc.
>    hopefully it's as unique as a real revision id
> * build number
> * platform where the tests were run
> * test type
>  - basically, the set of tests to run and what protocol to use. one test
>    may be run with different protocols
> * test name
> * test variant
>  - or combination. certain tests may be run in many combinations
>
> Using tis file I tried to rearrange test execution order for every
> specific build number (using failure data from previous builds in the
> same table) to detect test failures as early as possible. I managed to
> have 90% of all test failures happening within first 10% of the tests.
>
> The goal of this project is to have something we can put back into
> buildbot, so that it could actually run only 10% of the tests, still
> having 90% probability of discovering a failure (or 20%, or 95%,
> whaveter).
>
> See also http://buildbot.askmonty.org/buildbot/
> you can click on a link for any branch, for example,
> http://buildbot.askmonty.org/buildbot/grid?branch=10.0
> then on any particular build, for example,
> http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297
> and on a stdio log for a particular test run:
>
> http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/2297/steps/test_6/logs/stdio
>
> on these pages you'll see what a platform (builder) is, build number, test
> types, combinations, etc.
>
> Regards,
> Sergei
>
>