← Back to team overview

kicad-developers team mailing list archive

Re: Regression Testing

 

On 05/07/2013 08:01 AM, Brian Sidebotham wrote:
> On 7 May 2013 13:51, Dick Hollenbeck <dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>> wrote:
> 
>     On 05/06/2013 04:36 PM, Brian Sidebotham wrote:
>     > This turned into just a set of notes about CTest and Boost.Test:
>     >
>     > CTest integrates well with CMake [1] and allows us to run executables that perform
>     tests.
>     > Each test can be set as required (to be passed) or not. The result of each test is based
>     > on a rule that is defined in the each tests properties. For example, the integer return
>     > code from a process can be tested, or a regex match can be performed on stdout or stderr
>     > for the test process.
>     >
>     > Simple tests that depend on the integer return code of a process can be done via CMake's
>     > execute_process by checking RESULT_VARIABLE.
>     >
>     > Using the process return code, both CTest and CMake can test python API changes by
>     > executing python scripts and checking python's return value.
>     >
>     > The Boost.Test framework provides easy methods for writing C++ test cases as
>     executables.
>     > A single executable can contain as many tests as required. Each test is called
>     > individually as a command line argument. The Boost.Test framework can work by
>     returning a
>     > different integer return code from the executable depending on the result of a test.
>     It is
>     > has some additional features which aid automatic regression testing (for example on
>     build
>     > servers and the like) [2]
>     >
>     > Writing test cases using the Boost.Test framework provides an easy way to test
>     > WorkerFunctions within the codebase as we can write test cases and link them to the
>     > classes we're testing. This really requires code to be more modular so that we can link
>     > test executables against libraries.
>     >
>     > CTest and Boost.Test can allow memory leak testing as CTest integrates well with
>     Purify or
>     > Valgrind.
>     >
>     > CTest allows subsets of tests to be run, so if you know you've only been working on a
>     > certain part of the codebase, you can quickly run just the tests that should be affected
>     > by your changes before running the whole test suite.
>     >
>     > The Python API requires test cases for stability so we know that our changes to the C++
>     > code do not break other peoples IP based on our Python API.
> 
> 
>     Thanks very much.  I probably will not have much input into this decision moving forward.
> 
>     How is the wxPython build script coming for Windows?
> 
> 
> Hi Dick,
> 
> I am going to cmake-ify wxPython to build it (I will maintain a patch, and hope wxPython
> will accept it), as I want to get rid of distutils which is really very broken for MinGW.
> Building wxPython is actually relatively straight forward, but there are a lot of options
> which translate to wxWidgets build options which need covering in cmake. This work isn't
> yet off the ground.

Very cool.  Is there any starting point for this work, anybody ever tried it before.  This
does not sound like Windows specific work then, since a CMake script could be sketched out
on Linux also.  Is it mostly collecting the source files from each directory, and then
putting them into statements in CMakeLists.txt files?

something like, per directory:

$ ls *.cpp | sort > files.txt

Then copying file.txt into CMakeLists.txt, etc.

set( DIR_SRCS
  files go here
  )

Obviously some degree of automation might be available, perhaps as a script on the
internet also.

Thanks for doing this.  CMake is a wonderful tool.  And it is wonderful that you recognize
that and are willing to help.



> 
> I'm currently working on the NSIS packaging of python-a-mingw-us. At the moment it looks
> like we'll have to use an 8-bit colour BMP for the logo which doesn't look great. There
> seems to be some peculiarities with the BMP format used by makensis because if it is
> anything other than an 8-bit BMP the logo doesn't get displayed.
> 
> I have read a lot of posts that sort-of relate to this, but most end up building the
> windows installer on a windows machine and this limitation doesn't then appear to exist.
> 
> Anyway, I have the installer pretty much up and running. I'd like to email you a diff to
> review before committing, just to make sure you're happy with what I've done. I'll email a
> patch tonight. I had to rename one of the installable components because the name is
> directly used as a variable name in the nsis script, and a few other things too in order
> to get the packaging to work.

Great stuff.

> 
> I think everything except the headers component is a required installation component.
> 
> Best Regards, Brian.



References