maria-developers team mailing list archive

[Pablo Estrada <polecito.em@xxxxxxxxx>]

Hello Sergei,

as for (2) - I'd say the most important part is to figure out how to
> select the subset of tests to run, not how to integrate it in buildbot.
>

Definitely! I was just reading into buildbot to spend time while I got
access to the data! : )


> I have a bzr repository with the test data (text files, basically
> database dumps) and a pretty hairy perl script that processes them and
> outputs result files. Then a gnuplot script that reads these result
> files and shows graphs.



There were 9 script runs, that is 9 result files and 9 graphs.
> I'll attach them too.
>
> The model I used was:
>
> 1. For every new revision buildbot runs all tests on all platforms
>
> 2. But they are sorted specially. No matter in what order tests are
> executed, this doesn't change the result - the set of failed tests.  So,
> my script was emulating that, changing the order in which tests were
> run. With the goal to have failures happen as early as possible.
>
> That's how to interpret the graphs. For example, on v5 you can see that
> more than 90% of all test failures happen within the first 20K tests
>

Yup, I understand what you mean here... I can grasp the concepts, but I am
still having some trouble understanding some of the terms that you use in
the graphs. I can see that recall is related to the percentage of failures
that have been encountered, and I guess that cutoff has to do with how many
files you analyze before starting reordering... Also, I can see by your
comments a bit of your thought process, but I have a few questions.

I also have some questions regarding what you did with some of the data, to
get some ideas on how to do it myself. Also regarding how the buildbot
organizes builds and how they correspond to code changes and to test runs.

If it's not too inconvenient, do you think we could set up a Google Hangout
or a Skype call on Monday to go over a few questions quickly?

If you are busy, we can do it through email or on IRC. I can also take a
dive into the bzr repo, and look at what you did, as well as read up all
the info regarding the data; but it would be real helpful if you could lend
me a hand : )

Thank you very much.
Best
Pablo