maria-developers team mailing list archive

[Pablo Estrada <polecito.em@xxxxxxxxx>]

Hello everyone,
Here's a small report on the news that I have so far:


   1. I had a slow couple of weeks because of a quick holiday that I took.
   I will make up for that.
   2. I added the metric that considers number of test_runs since a
   test_case ran for the last time. I graphed it, and it does not affect the
   results much at all. -I still think this is useful to uncover hidden bugs
   that might lurk in the code for a long time; but testing this condition is
   difficult with our data. I would like to keep this measure, specially since
   it doesn't seem to affect results negatively. Opinions?
   3. I liked Sergei's idea about using the changes on the test files to
   calculate the relevancy index. If a test has been changed recently, its
   relevancy index should be high. This is also more realistic, and uses
   information that it's easy for us to figure out.
   - I am trying to match the change_files table or the mysql-test
      directory with the test_failure table. I was confused about the name of
      tests and test suites, but I am making progress on it. Once I am able to
      match at least 90% of the test_names in test_failure with the filename in
      the change_files table, I will incorporate this data into the
code and see
      how it works out.
      - *Question*: Looking at the change_files table, there are files that
      have been *ADDED* several times. Why would this be? Maybe when a new
      branch is created, all files are ADDED to it? Any ideas? : ) (If no one
      knows, I'll figure it out, but maybe you do know ; ))
   4. I uploaded inline comments for my code last week, let me know if it's
   clear enough. You can start by run_basic_simulations.py, where the most
   important functions are called... and after, you can dive into
   basic_simulator.py, where the simulation is actually done. The repository
   is a bit messy, I admit. I'll clean it up in the following commits.

This is all I have to report for now. Any advice on the way I'm proceeding
is welcome : )
Have a nice week, everyone.

Pablo


On Sun, May 25, 2014 at 11:43 PM, Sergei Golubchik <serg@xxxxxxxxxxx> wrote:

> Hi, Pablo!
>
> On May 25, Pablo Estrada wrote:
> > On Thu, May 22, 2014 at 5:39 PM, Sergei Golubchik <serg@xxxxxxxxxxx>
> wrote:
> > > I don't think you should introduce artificial limitations that make
> > > the recall worse, because they "look realistic".
> > >
> > > You can do it realistic instead, not look realistic - simply pretend
> > > that your code is already running on buildbot and limits the number
> > > of tests to run. So, if the test didn't run - you don't have any
> > > failure information about it.
> > >
> > > And then you only need to do what improves recall, nothing else :)
> > >
> > > (of course, to calculate the recall you need to use all failures,
> > > even for tests that you didn't run)
> >
> > Yes, my code *already works this way*. It doesn't consider failure
> > information from tests that were not supposed to run.  The graphs that
> > I sent are from scripts that ran like this.
>
> Good. I hoped that'll be the case (but didn't check your scripts on
> github yet, sorry).
>
> > Of course, the recall is just the number of spotted failures from the
> 100%
> > of known failures : )
> >
> > Anyway, with all this, I will get to work on adapting the simulation a
> > little bit:
> >
> >    - Time since last run will also affect the relevancy of a test
> >    - I will try to use the list of changed files from commits to make
> sure
> >    new tests start running right away
> >
> > Any other comments are welcome.
>
> Getting back to your "potential fallacy" at how you start taking tests
> into account only when they fail for the first time...
>
> I agree, in real life we cannot do that. Instead we start from a
> complete list of tests, that is known in advance. And you don't have it,
> unfortunately.
>
> An option would be to create a complete list of all tests that have
> ever failed (and perhaps remove tests that were added in some revision
> present in the history). And use that as a "starting set" of tests.
>
> Alternatively, we can generate a list of all tests currently present in
> the 10.0 tree - everything that you have in the history tables should be
> a subset of that.
>
> Regards,
> Sergei
>
>