← Back to team overview

kicad-developers team mailing list archive

Re: Regression Testing

 

On May 17, 2013 5:00 PM, "Brian Sidebotham" <brian.sidebotham@xxxxxxxxx>
wrote:
>
> On 17 May 2013 22:08, Dick Hollenbeck <dick@xxxxxxxxxxx> wrote:
>>
>> On 05/07/2013 08:01 AM, Brian Sidebotham wrote:
>> > On 7 May 2013 13:51, Dick Hollenbeck <dick@xxxxxxxxxxx <mailto:
dick@xxxxxxxxxxx>> wrote:
>> >
>> >     On 05/06/2013 04:36 PM, Brian Sidebotham wrote:
>> >     > This turned into just a set of notes about CTest and Boost.Test:
>> >     >
>> >     > CTest integrates well with CMake [1] and allows us to run
executables that perform
>> >     tests.
>> >     > Each test can be set as required (to be passed) or not. The
result of each test is based
>> >     > on a rule that is defined in the each tests properties. For
example, the integer return
>> >     > code from a process can be tested, or a regex match can be
performed on stdout or stderr
>> >     > for the test process.
>> >     >
>> >     > Simple tests that depend on the integer return code of a process
can be done via CMake's
>> >     > execute_process by checking RESULT_VARIABLE.
>> >     >
>> >     > Using the process return code, both CTest and CMake can test
python API changes by
>> >     > executing python scripts and checking python's return value.
>> >     >
>> >     > The Boost.Test framework provides easy methods for writing C++
test cases as
>> >     executables.
>> >     > A single executable can contain as many tests as required. Each
test is called
>> >     > individually as a command line argument. The Boost.Test
framework can work by
>> >     returning a
>> >     > different integer return code from the executable depending on
the result of a test.
>> >     It is
>> >     > has some additional features which aid automatic regression
testing (for example on
>> >     build
>> >     > servers and the like) [2]
>> >     >
>> >     > Writing test cases using the Boost.Test framework provides an
easy way to test
>> >     > WorkerFunctions within the codebase as we can write test cases
and link them to the
>> >     > classes we're testing. This really requires code to be more
modular so that we can link
>> >     > test executables against libraries.
>> >     >
>> >     > CTest and Boost.Test can allow memory leak testing as CTest
integrates well with
>> >     Purify or
>> >     > Valgrind.
>> >     >
>> >     > CTest allows subsets of tests to be run, so if you know you've
only been working on a
>> >     > certain part of the codebase, you can quickly run just the tests
that should be affected
>> >     > by your changes before running the whole test suite.
>> >     >
>> >     > The Python API requires test cases for stability so we know that
our changes to the C++
>> >     > code do not break other peoples IP based on our Python API.
>> >
>> >
>> >     Thanks very much.  I probably will not have much input into this
decision moving forward.
>> >
>> >     How is the wxPython build script coming for Windows?
>> >
>> >
>> > Hi Dick,
>> >
>> > I am going to cmake-ify wxPython to build it (I will maintain a patch,
and hope wxPython
>> > will accept it), as I want to get rid of distutils which is really
very broken for MinGW.
>> > Building wxPython is actually relatively straight forward, but there
are a lot of options
>> > which translate to wxWidgets build options which need covering in
cmake. This work isn't
>> > yet off the ground.
>> >
>> > I'm currently working on the NSIS packaging of python-a-mingw-us. At
the moment it looks
>> > like we'll have to use an 8-bit colour BMP for the logo which doesn't
look great. There
>> > seems to be some peculiarities with the BMP format used by makensis
because if it is
>> > anything other than an 8-bit BMP the logo doesn't get displayed.
>> >
>> > I have read a lot of posts that sort-of relate to this, but most end
up building the
>> > windows installer on a windows machine and this limitation doesn't
then appear to exist.
>> >
>> > Anyway, I have the installer pretty much up and running. I'd like to
email you a diff to
>> > review before committing, just to make sure you're happy with what
I've done. I'll email a
>> > patch tonight. I had to rename one of the installable components
because the name is
>> > directly used as a variable name in the nsis script, and a few other
things too in order
>> > to get the packaging to work.
>>
>> Its been weeks now.  I did not receive it.  I don't see it checked in
either.
>>
>> >
>> > I think everything except the headers component is a required
installation component.
>>
>> The headers are required for folks to stack C/C++ code on top, such as
wxPython.  So
>> unless we want to do a second "development" package, I don't think
there's harm in leaving
>> them in a single package.
>>
>> Thanks,
>>
>> Dick
>
>
> Sorry Dick! Please find the diff attached. It took me ages to sort the
branding out. It turns out that NSIS is only happy with an 8-bit windows
bitmap with no colourspace information or run-length-encoding. Once I found
that out, I could generate some initial bitmaps for the nsis installer.
I've attached a few images of what the installer steps look like on my
machine.

Looks as good as it can.  Thanks for sorting that out.


>
> Let me know if you're happy and I'll commit the changes. It re-enables
the NSIS installer for the package target.
>

Can we have a variable to test which reenables the zip packaging?  So
either can be built by cmake.

I assume your higher level cmake scripts, say in winbuilder, will be easier
if python is installable as a zip.

I will put both types of packaging at google.

> I've marked all components as "required to be installed"
>
> Is pointing the the installer licence page to the LICENCE file correct?
Or do we want to do a separate licence file for Python-a-mingw-us?

It is still python.  Python is plagued with an evolutionary license.   What
about simply pointing via url to python's license at python site?  That way
any confusion is pinned there.

On a-mingus itself, for extension building on windows, maybe a cmake
template, which has knowledge of header and lib locs?  Since no distutils
for mingw?  Possibly a python wrapper to the cmake creation from template
process?  Could code the cmake script tempate in the python wrapper file.
Customize it, write out cmakelists.txt, with proper dirs and libs and
targets in it.  Runs cmake, then make, then make install.


>
> Best Regards, Brian.
>

Follow ups

References