maria-developers team mailing list archive

[Pablo Estrada <polecito.em@xxxxxxxxx>]

Hi Elena,
A small progress report: I was able to quickly make the changes related to
selecting code changes to measure correlations with test failures. Recall
is still around 80% with running set of 300 and short prediction stages. I
can focus now on the input file list, since I believe this will make
results more realistic, and (I expect)  help push recall a further up.

Can you please upload the report files from MTR, so that I can include the
logic of an input test list?

Also, since I am going to incorporate this logic, it might be good to
define (even if just roughly) the "core module" and the "wrapper module"
that you had mentioned earlier, rather than just incorporating the list,
and making the code that I have now even more bloated with mixed up
functionalities. What do you think?

Regards
Pablo


On Tue, Jul 15, 2014 at 2:18 PM, Pablo Estrada <polecito.em@xxxxxxxxx>
wrote:

> Hello Elena,
>
> Can you give a raw estimation of a ratio of failures missed due to being
>> low in the priority queue vs those that were not in the queue at all?
>>
>
> I sent this information in a previous email, here:
> https://lists.launchpad.net/maria-developers/msg07482.html
>
> Also, once again, I would like you to start using an incoming test list as
>> an initial point of your test set generation. It must be done sooner or
>> later, I already explained earlier why; and while it's not difficult to
>> implement even after the end of your project, it might affect the result
>> considerably, so we need to know if it makes it better or worse, and adjust
>> the algorithm accordingly.
>>
>> You are right. I understand that this information is not fully available
> for all the test_runs, so can you upload the information going back as much
> as possible? I can parse these files and adjust the program to work with
> this. I will get on to work with this, I think this should significantly
> improve results. I think, it might even push my current strategy from
> promising results into attractive ones.
>
>
>> There are several options which change the way the tests are executed;
>> e.g. tests can be run in a "normal" mode, or in PS protocol mode, or with
>> valgrind, or with embedded server. And it might well be that some tests
>> always fail e.g. with valgrind, but almost never fail otherwise.
>> Information about these options is partially available in test_run.info,
>> but it would require some parsing. It would be perfect if you could analyze
>> the existing data to understand whether using it can affect your results
>> before spending time on actual code changes.
>>
>
> I will keep this in consideration, but for now I will focus on these two
> main things:
>
>    - Improving precision of selecting code changes to estimate
>    correlation with test failures
>    - Adding the use of an incoming test list
>
>
>
>> When we are trying to watch all code changes and find correlation with
>> test failures, if it's done well, it should actually provide immediate
>> gain; however, it's very difficult to do it right, there is way too much
>> noise in the statistical data to get a reliable picture. So, while it will
>> be nice if you get it work (since you already started doing it), don't take
>> it as a defeat if you eventually find out that it doesn't work very well.
>>
>
> Well, actually, this is the only big difference between the original
> strategy using just a weighted average of failures; and the new strategy,
> which performs *significantly better* in longer testing settings. It has
> been working for a few weeks, and is up on github.
>
> Either way, as I said before, I will, from today, focus on improving
> precision of selecting code changes to estimate correlation with test
> failures.
>
> Regards
> Pablo
>