← Back to team overview

kicad-developers team mailing list archive

Re: Regression Testing

 

On 4/29/2013 12:38 PM, Adam Wolf wrote:
> If we evaluate cppunit, and run into issues when we try to make it
> cross platform, I suggest looking at Google Test Framework, which is
> like cppunit, but works better with regards to weird compilers.
> 
> Adam Wolf
> Wayne and Layne, LLC

If it helps, here is a nice test framework feature comparison chart
(assuming it's correct and reasonably current) for just about every
language.

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

We could always create branch to try different ideas to see what works
and what doesn't before making any decisions.  Honestly, I don't have
much experience in this area so I'm going to have to depend on someone
else but I'm willing to do my part as best I can.

Wayne

> 
> On Mon, Apr 29, 2013 at 11:33 AM, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>> Miguel,
>>
>> Sorry, I also was evaluating.  So I apologize for being hypocritical on that.
>>
>> But you'll notice that I was very careful to remain neutral still at this point, and I was
>> analyzing mostly my own ideas.
>>
>>
>> Still, my bad.  You see, brainstorming does take some practice  :)
>>
>> I think we'll probably end up with a good solution anyways.
>>
>>
>> Dick
>>
>>
>>
>>  for sounding On 04/29/2013 11:27 AM, Dick Hollenbeck wrote:
>>> Technically, during the brainstorming process, it is best to gather ideas, and postpone
>>> passing judgement on those you have already heard, until the evaluation process, which
>>> comes later.  There is a theoretical reason why this is not to be done, and it supposedly
>>> thwarts creativity.
>>>
>>> Creativity is the genesis of good ideas.
>>>
>>> After the list of ideas is made, evaluation eventually begins.  When it is agreed that
>>> there are enough ideas to warrant the transition.
>>>
>>> So your idea is CPPUnit.
>>>
>>> Brainstorming is a fantastic tool, but admittedly it takes some understanding and
>>> agreement to follow it.
>>>
>>> The notion of thwarting creativity is specifically one of the things brainstorming tries
>>> to avoid, since sometimes the best ideas end up being out of the norm.  Improvements come
>>> by challenging old ways.  If they fall into the waste bucket during the evaluation
>>> process, there is no harm at that point.
>>>
>>>
>>> Dick
>>>
>>>
>>>
>>>
>>> On 04/29/2013 10:40 AM, Miguel Angel Ajo wrote:
>>>> I have the feeling that setting entry points from Python to C, when not needed will be an
>>>> extra work and harder to maintain. In that
>>>> case why not just using C++ standard testing frameworks?, like CPPUnit, or something that
>>>> plays well with wx for UI testing?
>>>>
>>>> http://wiki.wxwidgets.org/Tools <-- Testing
>>>>
>>>>
>>>> We have two things to test:
>>>>
>>>> 1) UI/behaviour , this is usually done launching dialogs manually, and calling click
>>>> methods, reading form entries, checking output, etc…,
>>>> Ideally it must be something like (warning, pseudocode)
>>>>
>>>> dialog = MyDialog()
>>>>
>>>> n = dialog->getListElements( 'list1' )
>>>> dialog->fillField( 'Field1','Content1' )
>>>> dialog->clickButton( 'Add' )
>>>> assert dialog->getListElements('list1')==(n+1)
>>>>
>>>> 2) Model testing: We call the back models with loads of tests to check their correct behavior
>>>>
>>>>    a) Normal ones, just model behaviour, correct DRC checks,
>>>>
>>>>    b) Models that generate outputs: In webkit, they just have pngs of the outputs
>>>> renderings, and they xor/compare. In our case, we can make a diff of
>>>>        the output contents with expected ones, or try to render them to PNG & test theme.
>>>> (warning: needs extra maintenance when outputs are enhanced or change, but it will catch
>>>> unexpected changes)
>>>>
>>>>    c) Model testing from python, everything that's swigged out can be tested from there,
>>>> and then we get a contract on what the available methods
>>>>    are to stay as fixed as possible. Even we get loads of examples out of this.
>>>>
>>>>
>>>> The automated tests I've seen so far (in the last two years) generate an xml standardized
>>>> files (junit xml format) that's used for reports (I use them in bamboo at nbee, and also
>>>> with other client for web ui testing, I've even used it to generate VHDL testing outputs
>>>> from ghdl) http://projects.nbee.es:8085/browse/S12-DEF the bad thing of bamboo is that it
>>>> doesn't play well with bzr (AFAIK)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Miguel Angel Ajo
>>>> http://www.nbee.es <http://www.nbee.es/>
>>>> +34911407752
>>>> skype: ajoajoajo
>>>>
>>>> On Monday, 29 de April de 2013 at 17:06, Dick Hollenbeck wrote:
>>>>
>>>>>>
>>>>>> We could address a) and b) by having a single DLL/DSO specific entrypoint for all tests,
>>>>>> with a big switch in there to route to the actual test. This scaffolding has some
>>>>>> maintenance cost, and begins to wash out the ease of using python on top, but perhaps not
>>>>>> fully.
>>>>>
>>>>>
>>>>> In the python C implementation code you find instances of the paradigm of va_args (tuple)
>>>>> with a C string telling which argument types are in the tuple. The C string is like a
>>>>> printf format string.
>>>>>
>>>>> So it would be possible to make this callable from python and pass all required arguments
>>>>> for any worker function. You would have to instantiate the C++ instance before calling
>>>>> any member function which is your worker function. But that could be done in the switch
>>>>> within the test entrypoint.
>>>>>
>>>>>
>>>>> test_result TestEntryPoint( aTestNum, formatStringTellingArgumentTypesInTuple, Tuple )
>>>>> {
>>>>> switch( aTestNum )
>>>>> {
>>>>> case XYZ:
>>>>> {
>>>>> // instantiate what is needed, say obj
>>>>>
>>>>> // gleen into C++ arguments, those from Tuple by parsing
>>>>> // formatStringTellingArgumentTypesInTuple
>>>>> va_args...
>>>>>
>>>>> // call object member func with C++ form of arguments in tuple.
>>>>> result = obj->WorkerFunction( arg1, arg2, arg3 );
>>>>>
>>>>> // destroy what we instantiated.
>>>>> }
>>>>> }
>>>>>
>>>>> return test_result( result ) ; // telling what just happened.
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> The above could be a seed for better ideas. It is a means of testing any "worker
>>>>> function" from python.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~kicad-developers
>>>>> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>>>> More help : https://help.launchpad.net/ListHelp
>>>>
>>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 




Follow ups

References