maria-developers team mailing list archive

[Pablo Estrada <polecito.em@xxxxxxxxx>]

Hello everyone,
I ran the tests with randomization on Standard and Mixed mode, and here are
the results.
1. Standard does not experience variation - The queue is always long enough.
2. Mixed does experience some variation - Actually, the number of tests run
changes dramatically, but I forgot to add the data in the chart. I can
report it too, but yes, the difference is large.
3. In any case, the results are still not quite satisfactory, so we can
think back to what I had mentioned earlier: How should we change our
paradigm to try to improve our chances?

Regards
Pablo


On Fri, Jun 20, 2014 at 7:45 PM, Pablo Estrada <polecito.em@xxxxxxxxx>
wrote:

> I have pushed my latest version of the code, and here is a test run that
> ran on this version of the code. It is quite different from the original
> expectation; so I'm taking a close look at the code for bugs, and will run
> another simulation ASAP  (I'll use less data to make it faster).
>
>
> On Thu, Jun 19, 2014 at 5:16 PM, Elena Stepanova <elenst@xxxxxxxxxxxxxxxx>
> wrote:
>
>> Hi Pablo,
>>
>> I'll send a more detailed reply later, just a couple of quick
>> comments/questions now.
>>
>> To your question
>>
>>
>>
>>> I'm just not quite sure what you mean with this example:
>>> mysql-test/plugin/example/mtr/t
>>>
>>> In this example, what is the test name? And what is exactly the path?
>>> (./mysql-test/...) or (./something/mysql-test/...)? I tried to look at
>>> some
>>> of the test result files but I couldn't find one certain example of this
>>> pattern (Meaning that I'm not sure what would be a real instance of it).
>>> Can you be more specific please?
>>>
>>>
>> I meant that if you look into the folder <tree>/mysql-test/suite/mtr/t/ ,
>> you'll see an example of what I described as "The result file can live not
>> only in /r dir, but also in /t dir, together with the test file":
>>
>> ls mysql-test/suite/mtr/t/
>> combs.combinations
>> combs.inc
>> inc.inc
>> newcomb.result
>> newcomb.test
>> proxy.inc
>> self.result
>> self.test
>> simple,c2,s1.rdiff
>> simple.combinations
>> simple.result
>> simple,s2,c2.rdiff
>> simple,s2.result
>> simple.test
>> single.result
>> single.test
>> source.result
>> source.test
>> test2.result
>> test2.test
>> testsh.result
>> testsh.test
>>
>> As far as I remember, your matching algorithm didn't cover that.
>>
>>
>>
>>  Here are the results. They are both a bit counterintuitive, and a bit
>>> strange
>>>
>>
>> Have you already done anything regarding (not) populating the queue
>> completely? I did expect that with the current logic, after adding full
>> cleanup between simulations, the more restrictive configuration would have
>> lower recall, because it generally runs much fewer tests.
>>
>> It would be interesting to somehow indicate in the results how many tests
>> were *actually* run. But if you don't have this information, please don't
>> re-run the full set just for the sake of it, maybe run only one running set
>> for standard/platform/branch/mixed, and let us see the results. No need
>> to spend time on graphs for that, a text form will be ok.
>>
>> Either way, please push the current code, I'd like to see it before I
>> come up with any suggestions about the next big moves.
>>
>> Regards,
>> Elena
>>
>
>

Attachment: randomized_2.png
Description: PNG image