← Back to team overview

kicad-developers team mailing list archive

Re: Regression Testing

 

On 5/7/2013 9:01 AM, Brian Sidebotham wrote:
> On 7 May 2013 13:51, Dick Hollenbeck <dick@xxxxxxxxxxx
> <mailto:dick@xxxxxxxxxxx>> wrote:
> 
>     On 05/06/2013 04:36 PM, Brian Sidebotham wrote:
>     > This turned into just a set of notes about CTest and Boost.Test:
>     >
>     > CTest integrates well with CMake [1] and allows us to run
>     executables that perform tests.
>     > Each test can be set as required (to be passed) or not. The result
>     of each test is based
>     > on a rule that is defined in the each tests properties. For
>     example, the integer return
>     > code from a process can be tested, or a regex match can be
>     performed on stdout or stderr
>     > for the test process.
>     >
>     > Simple tests that depend on the integer return code of a process
>     can be done via CMake's
>     > execute_process by checking RESULT_VARIABLE.
>     >
>     > Using the process return code, both CTest and CMake can test
>     python API changes by
>     > executing python scripts and checking python's return value.
>     >
>     > The Boost.Test framework provides easy methods for writing C++
>     test cases as executables.
>     > A single executable can contain as many tests as required. Each
>     test is called
>     > individually as a command line argument. The Boost.Test framework
>     can work by returning a
>     > different integer return code from the executable depending on the
>     result of a test. It is
>     > has some additional features which aid automatic regression
>     testing (for example on build
>     > servers and the like) [2]
>     >
>     > Writing test cases using the Boost.Test framework provides an easy
>     way to test
>     > WorkerFunctions within the codebase as we can write test cases and
>     link them to the
>     > classes we're testing. This really requires code to be more
>     modular so that we can link
>     > test executables against libraries.
>     >
>     > CTest and Boost.Test can allow memory leak testing as CTest
>     integrates well with Purify or
>     > Valgrind.
>     >
>     > CTest allows subsets of tests to be run, so if you know you've
>     only been working on a
>     > certain part of the codebase, you can quickly run just the tests
>     that should be affected
>     > by your changes before running the whole test suite.
>     >
>     > The Python API requires test cases for stability so we know that
>     our changes to the C++
>     > code do not break other peoples IP based on our Python API.

You pretty much came to the same conclusion I did when I looked at this.
 Use CTest to build, run, and manage the tests and use the Boost unit
test framework to write the tests.  It might be a good idea to set up a
separate branch on Launchpad until we have a workable testing framework
in place and then merge that into the testing branch.  I also want to
make sure we have a reasonable set of expectations that we can either
add to the coding policy or create a separate test policy before we make
it available for general consumption.

> 
> 
>     Thanks very much.  I probably will not have much input into this
>     decision moving forward.
> 
>     How is the wxPython build script coming for Windows?
> 
> 
> Hi Dick,
> 
> I am going to cmake-ify wxPython to build it (I will maintain a patch,
> and hope wxPython will accept it), as I want to get rid of distutils
> which is really very broken for MinGW. Building wxPython is actually
> relatively straight forward, but there are a lot of options which
> translate to wxWidgets build options which need covering in cmake. This
> work isn't yet off the ground.

I apologize for not being able to help with CMake-ifying wxPython.  I've
been busy and the footprint library table code is proving to be way more
work than I had planned.  Thank you for taking the lead on this.

Best Wishes

Wayne

> 
> I'm currently working on the NSIS packaging of python-a-mingw-us. At the
> moment it looks like we'll have to use an 8-bit colour BMP for the logo
> which doesn't look great. There seems to be some peculiarities with the
> BMP format used by makensis because if it is anything other than an
> 8-bit BMP the logo doesn't get displayed.
> 
> I have read a lot of posts that sort-of relate to this, but most end up
> building the windows installer on a windows machine and this limitation
> doesn't then appear to exist.
> 
> Anyway, I have the installer pretty much up and running. I'd like to
> email you a diff to review before committing, just to make sure you're
> happy with what I've done. I'll email a patch tonight. I had to rename
> one of the installable components because the name is directly used as a
> variable name in the nsis script, and a few other things too in order to
> get the packaging to work.
> 
> I think everything except the headers component is a required
> installation component.
> 
> Best Regards, Brian.
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 



Follow ups

References