kicad-developers team mailing list archive
-
kicad-developers team
-
Mailing list archive
-
Message #37912
Re: [PATCH] Fuzzable PCB parsing test harness
Hi,
This is an updated path sets that includes some extras on top of the
previously mentioned:
* TOC for the testing docs
* Update the io_benchmark tool to work with full paths.
* Add an in-memory io_benchmark case, which gives an approximate
baseline for how fast it could be without any IO overhead
Cheers,
John
On Thu, Oct 11, 2018 at 11:43 AM John Beard <john.j.beard@xxxxxxxxx> wrote:
>
> Hi,
>
> Here's a quick patch to add some documentation about print-debugging
> and trace. It's a follow-up to the fuzz tool, as there would be a
> conflict in the MD file, as one section follows the other, so it needs
> to go in afterwards, or otherwise be manually resolved.
>
> Also includes a handy list of WXTRACE masks, as they aren't easily
> found (you have to grep for wxLogTrace, then grep for the mask
> variable to find the actual string to use). I didn't document the
> functions, as they're pretty self-explanatory ("KICAD_FIND_ITEM: info
> about the "Find Item" tool" is not that helpful!).
>
> Cheers,
>
> John
> On Tue, Oct 9, 2018 at 2:53 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> >
> > Hi,
> >
> > Here is an update patch that rebases over the commenting out of
> > pcb_test_window, polygon_triangulation and polygon_generator and fixes
> > a link error to do with base_screen.cpp.
> >
> > Cheers,
> >
> > John
> > On Mon, Oct 8, 2018 at 5:27 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> > >
> > > Sorry,
> > >
> > > I wrote "ms", I meant "us" - the times are in the handful-of-millsecond range.
> > >
> > > Cheers,
> > >
> > > John
> > > On Mon, Oct 8, 2018 at 5:24 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> > > >
> > > > Hi,
> > > >
> > > > This is a patch to add a test program that allows to parse a Pcbnew
> > > > file from command line params or stdin. This means you can use it for
> > > > fuzz testing.
> > > >
> > > > I have done a little bit of fuzz testing so far (8 million execs,
> > > > about 70% of a cycle), and have not found any crashes, but I can make
> > > > it hang in a few ways. These all seem to be in streams which contain
> > > > nul's. This is actually not reachable from the UI due to reading files
> > > > into wxStrings first (nut quite sure why), whereas this program uses
> > > > the parser directly. Thus, the bug is probably not very critical.
> > > > Example hanging input attached (note there's a nul in it, so your
> > > > editor may or may not like that).
> > > >
> > > > This program can also be fed a number of files, which means it could
> > > > be used for automated testing that all files in a batch can be parsed
> > > > successfully, and also provides a handy way to put GDB on a program
> > > > when debugging the parser against specific input.
> > > >
> > > > There is timing on the parsing too, mostly for interest (use the -v
> > > > flag). It takes about 150-3000ms per FP on my machine for the FPs in
> > > > Connector_PinSocket_2.54mm.pretty.
> > > >
> > > > There's also some centralisation of some QA-related utils into a
> > > > qa_utils library.
> > > >
> > > > Cheers,
> > > >
> > > > John
From be14a545f313d01e1048668525fec67a01aa3d75 Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Mon, 8 Oct 2018 19:11:46 +0100
Subject: [PATCH 2/3] Add an in-memory STREAM_LINE_READER benchmark
This adds an io-benchmark case of the STRING_LINE_READER
class, which reads a file into a std::string, *then*
reads it line by line.
As expected, due to it all being in memory, this is very fast.
Also fixes an issue in io_benchmark where the input file
must be in the current dir.
---
qa/qa_utils/stdstream_line_reader.cpp | 4 +-
tools/io_benchmark/io_benchmark.cpp | 75 ++++++++++++++++++++++-----
2 files changed, 64 insertions(+), 15 deletions(-)
diff --git a/qa/qa_utils/stdstream_line_reader.cpp b/qa/qa_utils/stdstream_line_reader.cpp
index 3f6596fdb..d473f3442 100644
--- a/qa/qa_utils/stdstream_line_reader.cpp
+++ b/qa/qa_utils/stdstream_line_reader.cpp
@@ -68,7 +68,7 @@ void STDISTREAM_LINE_READER::SetStream( std::istream& aStream )
IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName ) :
- m_fStream( aFileName.GetFullName().ToUTF8() )
+ m_fStream( aFileName.GetFullPath().ToUTF8() )
{
if( !m_fStream.is_open() )
{
@@ -79,7 +79,7 @@ IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName ) :
SetStream( m_fStream );
- m_source = aFileName.GetFullName();
+ m_source = aFileName.GetFullPath();
}
diff --git a/tools/io_benchmark/io_benchmark.cpp b/tools/io_benchmark/io_benchmark.cpp
index 9442d9317..7236884d6 100644
--- a/tools/io_benchmark/io_benchmark.cpp
+++ b/tools/io_benchmark/io_benchmark.cpp
@@ -123,7 +123,7 @@ static void bench_line_reader( const wxFileName& aFile, int aReps, BENCH_REPORT&
{
for( int i = 0; i < aReps; ++i)
{
- LR fstr( aFile.GetFullName() );
+ LR fstr( aFile.GetFullPath() );
while( fstr.ReadLine() )
{
report.linesRead++;
@@ -140,7 +140,7 @@ static void bench_line_reader( const wxFileName& aFile, int aReps, BENCH_REPORT&
template<typename LR>
static void bench_line_reader_reuse( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
{
- LR fstr( aFile.GetFullName() );
+ LR fstr( aFile.GetFullPath() );
for( int i = 0; i < aReps; ++i)
{
@@ -155,6 +155,53 @@ static void bench_line_reader_reuse( const wxFileName& aFile, int aReps, BENCH_R
}
+/**
+ * Benchmark using STRING_LINE_READER on string data read into memory from a file
+ * using std::ifstream, but read the data fresh from the file each time
+ */
+static void bench_string_lr( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
+{
+ for( int i = 0; i < aReps; ++i)
+ {
+ std::ifstream ifs( aFile.GetFullPath() );
+ std::string content((std::istreambuf_iterator<char>(ifs)),
+ std::istreambuf_iterator<char>());
+
+ STRING_LINE_READER fstr( content, aFile.GetFullPath() );
+ while( fstr.ReadLine() )
+ {
+ report.linesRead++;
+ report.charAcc += (unsigned char) fstr.Line()[0];
+ }
+ }
+}
+
+
+/**
+ * Benchmark using STRING_LINE_READER on string data read into memory from a file
+ * using std::ifstream
+ *
+ * The STRING_LINE_READER is not reused (it cannot be rewound),
+ * but the file is read only once
+ */
+static void bench_string_lr_reuse( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
+{
+ std::ifstream ifs( aFile.GetFullPath() );
+ std::string content((std::istreambuf_iterator<char>(ifs)),
+ std::istreambuf_iterator<char>());
+
+ for( int i = 0; i < aReps; ++i)
+ {
+ STRING_LINE_READER fstr( content, aFile.GetFullPath() );
+ while( fstr.ReadLine() )
+ {
+ report.linesRead++;
+ report.charAcc += (unsigned char) fstr.Line()[0];
+ }
+ }
+}
+
+
/**
* Benchmark using an INPUTSTREAM_LINE_READER with a given
* wxInputStream implementation.
@@ -163,11 +210,11 @@ static void bench_line_reader_reuse( const wxFileName& aFile, int aReps, BENCH_R
template<typename S>
static void bench_wxis( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
{
- S fileStream( aFile.GetFullName() );
+ S fileStream( aFile.GetFullPath() );
for( int i = 0; i < aReps; ++i)
{
- INPUTSTREAM_LINE_READER istr( &fileStream, aFile.GetFullName() );
+ INPUTSTREAM_LINE_READER istr( &fileStream, aFile.GetFullPath() );
while( istr.ReadLine() )
{
@@ -188,8 +235,8 @@ static void bench_wxis( const wxFileName& aFile, int aReps, BENCH_REPORT& report
template<typename S>
static void bench_wxis_reuse( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
{
- S fileStream( aFile.GetFullName() );
- INPUTSTREAM_LINE_READER istr( &fileStream, aFile.GetFullName() );
+ S fileStream( aFile.GetFullPath() );
+ INPUTSTREAM_LINE_READER istr( &fileStream, aFile.GetFullPath() );
for( int i = 0; i < aReps; ++i)
{
@@ -212,12 +259,12 @@ static void bench_wxis_reuse( const wxFileName& aFile, int aReps, BENCH_REPORT&
template<typename WXIS>
static void bench_wxbis( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
{
- WXIS fileStream( aFile.GetFullName() );
+ WXIS fileStream( aFile.GetFullPath() );
wxBufferedInputStream bufferedStream( fileStream );
for( int i = 0; i < aReps; ++i)
{
- INPUTSTREAM_LINE_READER istr( &bufferedStream, aFile.GetFullName() );
+ INPUTSTREAM_LINE_READER istr( &bufferedStream, aFile.GetFullPath() );
while( istr.ReadLine() )
{
@@ -238,10 +285,10 @@ static void bench_wxbis( const wxFileName& aFile, int aReps, BENCH_REPORT& repor
template<typename WXIS>
static void bench_wxbis_reuse( const wxFileName& aFile, int aReps, BENCH_REPORT& report )
{
- WXIS fileStream( aFile.GetFullName() );
+ WXIS fileStream( aFile.GetFullPath() );
wxBufferedInputStream bufferedStream( fileStream );
- INPUTSTREAM_LINE_READER istr( &bufferedStream, aFile.GetFullName() );
+ INPUTSTREAM_LINE_READER istr( &bufferedStream, aFile.GetFullPath() );
for( int i = 0; i < aReps; ++i)
{
@@ -262,10 +309,12 @@ static std::vector<BENCHMARK> benchmarkList =
{
{ 'f', bench_fstream, "std::fstream" },
{ 'F', bench_fstream_reuse, "std::fstream, reused" },
- { 'r', bench_line_reader<FILE_LINE_READER>, "RICHIO" },
- { 'R', bench_line_reader_reuse<FILE_LINE_READER>, "RICHIO, reused" },
+ { 'r', bench_line_reader<FILE_LINE_READER>, "RichIO FILE_L_R" },
+ { 'R', bench_line_reader_reuse<FILE_LINE_READER>, "RichIO FILE_L_R, reused" },
{ 'n', bench_line_reader<IFSTREAM_LINE_READER>, "std::ifstream L_R" },
{ 'N', bench_line_reader_reuse<IFSTREAM_LINE_READER>, "std::ifstream L_R, reused" },
+ { 's', bench_string_lr, "RichIO STRING_L_R"},
+ { 'S', bench_string_lr_reuse, "RichIO STRING_L_R, reused"},
{ 'w', bench_wxis<wxFileInputStream>, "wxFileIStream" },
{ 'W', bench_wxis<wxFileInputStream>, "wxFileIStream, reused" },
{ 'g', bench_wxis<wxFFileInputStream>, "wxFFileIStream" },
@@ -358,7 +407,7 @@ int main( int argc, char* argv[] )
os << "IO Bench Mark Util" << std::endl;
- os << " Benchmark file: " << inFile.GetFullName() << std::endl;
+ os << " Benchmark file: " << inFile.GetFullPath() << std::endl;
os << " Repetitions: " << (int) reps << std::endl;
os << std::endl;
--
2.18.0
From dd3b56b5f1711952bba1bb1d79253c3cc371d9ac Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Sat, 6 Oct 2018 15:35:17 +0100
Subject: [PATCH 1/3] QA: PCB file input parse test program (fuzzable)
This adds a test program which can be used to test the
parsing of a given KiCad PCB file. This interface is
useful for both manual or automated debugging of given
files, as well as providing an interface suitable for
fuzz-testing tools.
Also adds to the testing docs to detail how fuzzing can
be used.
Also moves some useful re-usable code from io-benchmark
to a new library qa_utils, which can contain code that
isn't need in the actual KiCad libs.
---
Documentation/development/testing.md | 52 +++++-
qa/CMakeLists.txt | 4 +
qa/pcb_parse_input/CMakeLists.txt | 66 ++++++++
qa/pcb_parse_input/main.cpp | 158 ++++++++++++++++++
qa/qa_utils/CMakeLists.txt | 40 +++++
qa/qa_utils/scoped_timer.h | 62 +++++++
.../qa_utils}/stdstream_line_reader.cpp | 8 +-
.../qa_utils}/stdstream_line_reader.h | 8 +-
tools/io_benchmark/CMakeLists.txt | 2 +-
9 files changed, 391 insertions(+), 9 deletions(-)
create mode 100644 qa/pcb_parse_input/CMakeLists.txt
create mode 100644 qa/pcb_parse_input/main.cpp
create mode 100644 qa/qa_utils/CMakeLists.txt
create mode 100644 qa/qa_utils/scoped_timer.h
rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.cpp (91%)
rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.h (92%)
diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md
index 4726ca536..86c1e2dde 100644
--- a/Documentation/development/testing.md
+++ b/Documentation/development/testing.md
@@ -103,6 +103,56 @@ You can run the tests in GDB to trace this:
If the test segfaults, you will get a familiar backtrace, just like
if you were running pcbnew under GDB.
+## Fuzz testing ##
+
+It is possible to run fuzz testing on some parts of KiCad. To do this for a
+generic function, you need to be able to pass some kind of input from the fuzz
+testing tool to the function under test.
+
+For example, to use the [AFL fuzzing tool][], you will need:
+
+* A test executable that can:
+ * Receive input from `stdin` to be run by `afl-fuzz`.
+ * Optional: process input from a filename to allow `afl-tmin` to minimise the
+ input files.
+* To compile this executable with an AFL compiler, to enable the instrumentation
+ that allows the fuzzer to detect the fuzzing state.
+
+For example, the `qa_pcb_parse_input` executable can be compiled like this:
+
+ mkdir build
+ cd build
+ cmake -DCMAKE_CXX_COMPILER=/usr/bin/afl-clang-fast++ -DCMAKE_C_COMPILER=/usr/bin/afl-clang-fast ../kicad_src
+ make qa_pcb_parse_input
+
+You may need to disable core dumps and CPU frequency scaling on your system (AFL
+will warn you if you should do this). For example, as root:
+
+ # echo core >/proc/sys/kernel/core_pattern
+ # echo performance | tee cpu*/cpufreq/scaling_governor
+
+To fuzz:
+
+ $ afl-fuzz -i fuzzin -o fuzzout -m500 qa/pcb_parse_input/qa_pcb_parse_input
+
+where:
+
+* `-i` is a directory of files to use as fuzz input "seeds"
+* `-o` is a directory to write the results (including inputs that provoke crashes
+ or hangs)
+* `-t` is the maximum time that a run is allowed to take before being declared a "hang"
+* `-m` is the memory allowed to use (this often needs to be bumped, as KiCad code
+ tends to use a lot of memory to initialise)
+
+The AFL TUI will then display the fuzzing progress. Hang- or
+crash-provoking inputs will be saved under the output directory specified.
+You can use these inputs to debug the program using the filename parameter
+to the executable, using your debugger of choice, for example GDB:
+
+ $ gdb --arg qa/pcb_parse_input/qa_pcb_parse_input ./fuzzout/hangs/<name>
+
+
[CTest]: https://cmake.org/cmake/help/latest/module/CTest.html
[Boost Unit Test framework]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/index.html
-[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
\ No newline at end of file
+[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
+[AFL fuzzing tool]: http://lcamtuf.coredump.cx/afl/
\ No newline at end of file
diff --git a/qa/CMakeLists.txt b/qa/CMakeLists.txt
index 36aa7b601..ae30e1a38 100644
--- a/qa/CMakeLists.txt
+++ b/qa/CMakeLists.txt
@@ -12,8 +12,12 @@ if( KICAD_SCRIPTING_MODULES )
endif()
+# common QA helpers
+add_subdirectory( qa_utils )
+
add_subdirectory( common )
add_subdirectory( shape_poly_set_refactor )
+add_subdirectory( pcb_parse_input )
# add_subdirectory( pcb_test_window )
# add_subdirectory( polygon_triangulation )
# add_subdirectory( polygon_generator )
\ No newline at end of file
diff --git a/qa/pcb_parse_input/CMakeLists.txt b/qa/pcb_parse_input/CMakeLists.txt
new file mode 100644
index 000000000..5f8e2f350
--- /dev/null
+++ b/qa/pcb_parse_input/CMakeLists.txt
@@ -0,0 +1,66 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+
+if( BUILD_GITHUB_PLUGIN )
+ set( GITHUB_PLUGIN_LIBRARIES github_plugin )
+endif()
+
+add_executable( qa_pcb_parse_input
+ # This is needed for the global mock objects
+ ../qa_utils/mocks.cpp
+
+ main.cpp
+
+ ../../common/base_units.cpp
+ ../../common/xnode.cpp
+ ../../common/base_screen.cpp
+)
+
+include_directories(
+ ${CMAKE_SOURCE_DIR}
+ ${CMAKE_SOURCE_DIR}/include
+ ${CMAKE_SOURCE_DIR}/polygon
+ ${CMAKE_SOURCE_DIR}/pcbnew
+ ${CMAKE_SOURCE_DIR}/common
+ ${CMAKE_SOURCE_DIR}/pcbnew/router
+ ${CMAKE_SOURCE_DIR}/pcbnew/tools
+ ${CMAKE_SOURCE_DIR}/pcbnew/dialogs
+ ${Boost_INCLUDE_DIR}
+ ${INC_AFTER}
+)
+
+target_link_libraries( qa_pcb_parse_input
+ legacy_wx
+ pcbcommon
+ common
+ bitmaps
+ polygon
+ pcad2kicadpcb
+ ${GITHUB_PLUGIN_LIBRARIES}
+ qa_utils
+ ${wxWidgets_LIBRARIES}
+)
+
+# we need to pretend to be something to appease the units code
+target_compile_definitions( qa_pcb_parse_input
+ PRIVATE PCBNEW
+)
\ No newline at end of file
diff --git a/qa/pcb_parse_input/main.cpp b/qa/pcb_parse_input/main.cpp
new file mode 100644
index 000000000..fc79f932b
--- /dev/null
+++ b/qa/pcb_parse_input/main.cpp
@@ -0,0 +1,158 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+
+#include <kicad_plugin.h>
+#include <pcb_parser.h>
+#include <richio.h>
+#include <class_board_item.h>
+
+#include <wx/cmdline.h>
+
+#include <stdstream_line_reader.h>
+#include <scoped_timer.h>
+
+using PARSE_DURATION = std::chrono::microseconds;
+
+/**
+ * Parse a PCB or footprint file from the given input stream
+ *
+ * @param aStream the input stream to read from
+ * @return success, duration (in us)
+ */
+bool parse(std::istream& aStream, bool aVerbose )
+{
+ // Take input from stdin
+ STDISTREAM_LINE_READER reader;
+ reader.SetStream( aStream );
+
+ PCB_PARSER parser;
+
+ parser.SetLineReader( &reader );
+
+ BOARD_ITEM* board = nullptr;
+
+ PARSE_DURATION duration {};
+
+ try
+ {
+ SCOPED_TIMER<PARSE_DURATION> timer( duration );
+ board = parser.Parse();
+ }
+ catch( const IO_ERROR& parse_error )
+ {
+ std::cerr << parse_error.Problem() << std::endl;
+ std::cerr << parse_error.Where() << std::endl;
+ }
+
+ if( aVerbose )
+ {
+ std::cout << "Took: " << duration.count() << "us" << std::endl;
+ }
+
+ return board != nullptr;
+}
+
+
+static const wxCmdLineEntryDesc g_cmdLineDesc [] =
+{
+ { wxCMD_LINE_SWITCH, "h", "help",
+ _( "displays help on the command line parameters" ),
+ wxCMD_LINE_VAL_NONE, wxCMD_LINE_OPTION_HELP },
+ { wxCMD_LINE_SWITCH, "v", "verbose",
+ _( "print parsing information") },
+ { wxCMD_LINE_PARAM, nullptr, nullptr,
+ _( "input file" ),
+ wxCMD_LINE_VAL_STRING, wxCMD_LINE_PARAM_OPTIONAL | wxCMD_LINE_PARAM_MULTIPLE },
+ { wxCMD_LINE_NONE }
+};
+
+
+enum RET_CODES
+{
+ OK = 0,
+ BAD_CMDLINE = 1,
+ PARSE_FAILED = 2,
+};
+
+
+int main(int argc, char** argv)
+{
+#ifdef __AFL_COMPILER
+ __AFL_INIT();
+#endif
+
+ wxMessageOutput::Set(new wxMessageOutputStderr);
+ wxCmdLineParser cl_parser( argc, argv );
+ cl_parser.SetDesc( g_cmdLineDesc );
+ cl_parser.AddUsageText( _("This program parses PCB files, either from the "
+ "stdin stream or from the given filenames. This can be used either for "
+ "standalone testing of the parser or for fuzz testing." ) );
+
+ int cmd_parsed_ok = cl_parser.Parse();
+ if( cmd_parsed_ok != 0 )
+ {
+ // Help and invalid input both stop here
+ return ( cmd_parsed_ok == -1 ) ? RET_CODES::OK : RET_CODES::BAD_CMDLINE;
+ }
+
+ const bool verbose = cl_parser.Found( "verbose" );
+
+ bool ok = true;
+ PARSE_DURATION duration;
+
+ const auto file_count = cl_parser.GetParamCount();
+
+ if ( file_count == 0 )
+ {
+ // Parse the file provided on stdin - used by AFL to drive the
+ // program
+ // while (__AFL_LOOP(2))
+ {
+ ok = parse( std::cin, verbose );
+ }
+ }
+ else
+ {
+ // Parse 'n' files given on the command line
+ // (this is useful for input minimisation (e.g. afl-tmin) as
+ // well as manual testing
+ for( unsigned i = 0; i < file_count; i++ )
+ {
+ const auto filename = cl_parser.GetParam( i );
+
+ if( verbose )
+ std::cout << "Parsing: " << filename << std::endl;
+
+ std::ifstream fin;
+ fin.open( filename );
+
+ ok = ok && parse( fin, verbose );
+ }
+ }
+
+ if( !ok )
+ return RET_CODES::PARSE_FAILED;
+
+ return RET_CODES::OK;
+}
\ No newline at end of file
diff --git a/qa/qa_utils/CMakeLists.txt b/qa/qa_utils/CMakeLists.txt
new file mode 100644
index 000000000..b5befaeee
--- /dev/null
+++ b/qa/qa_utils/CMakeLists.txt
@@ -0,0 +1,40 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+set( QA_UTIL_COMMON_SRC
+ stdstream_line_reader.cpp
+)
+
+# A generic library of useful functions for various testing purposes
+add_library( qa_utils
+ ${QA_UTIL_COMMON_SRC}
+)
+
+include_directories( BEFORE ${INC_BEFORE} )
+
+target_link_libraries( qa_utils
+ common
+ ${wxWidgets_LIBRARIES}
+)
+
+target_include_directories( qa_utils PUBLIC
+ ${CMAKE_CURRENT_SOURCE_DIR}
+)
\ No newline at end of file
diff --git a/qa/qa_utils/scoped_timer.h b/qa/qa_utils/scoped_timer.h
new file mode 100644
index 000000000..028bfef46
--- /dev/null
+++ b/qa/qa_utils/scoped_timer.h
@@ -0,0 +1,62 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+#ifndef SCOPED_TIMER_H
+#define SCOPED_TIMER_H
+
+#include <chrono>
+
+/**
+ * A simple RAII class to measure the time of an operation.
+ *
+ * ON construction, a timer is started, and on destruction, the timer is
+ * ended, and the time difference is written into the given duration
+ */
+template<typename DURATION>
+class SCOPED_TIMER
+{
+ using CLOCK = std::chrono::steady_clock;
+ using TIME_PT = std::chrono::time_point<CLOCK>;
+
+public:
+ SCOPED_TIMER( DURATION& aDuration ):
+ m_duration( aDuration )
+ {
+ m_start = CLOCK::now();
+ }
+
+ ~SCOPED_TIMER()
+ {
+ const auto end = CLOCK::now();
+
+ // update the output
+ m_duration = std::chrono::duration_cast<DURATION>( end - m_start );
+ }
+
+private:
+
+ DURATION& m_duration;
+ TIME_PT m_start;
+};
+
+#endif // SCOPED_TIMER_h
\ No newline at end of file
diff --git a/tools/io_benchmark/stdstream_line_reader.cpp b/qa/qa_utils/stdstream_line_reader.cpp
similarity index 91%
rename from tools/io_benchmark/stdstream_line_reader.cpp
rename to qa/qa_utils/stdstream_line_reader.cpp
index cb8ddabdc..3f6596fdb 100644
--- a/tools/io_benchmark/stdstream_line_reader.cpp
+++ b/qa/qa_utils/stdstream_line_reader.cpp
@@ -1,7 +1,7 @@
/*
* This program source code file is part of KiCad, a free EDA CAD application.
*
- * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
@@ -58,10 +58,10 @@ char* STDISTREAM_LINE_READER::ReadLine()
}
-void STDISTREAM_LINE_READER::setStream( std::istream& aStream )
+void STDISTREAM_LINE_READER::SetStream( std::istream& aStream )
{
// Could be done with a virtual getStream function, but the
- // virtual function call is a noticable (but minor) penalty within
+ // virtual function call is a noticeable (but minor) penalty within
// ReadLine() in tight loops
m_stream = &aStream;
}
@@ -77,7 +77,7 @@ IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName ) :
THROW_IO_ERROR( msg );
}
- setStream( m_fStream );
+ SetStream( m_fStream );
m_source = aFileName.GetFullName();
}
diff --git a/tools/io_benchmark/stdstream_line_reader.h b/qa/qa_utils/stdstream_line_reader.h
similarity index 92%
rename from tools/io_benchmark/stdstream_line_reader.h
rename to qa/qa_utils/stdstream_line_reader.h
index 5ccf86f3c..81c238dc6 100644
--- a/tools/io_benchmark/stdstream_line_reader.h
+++ b/qa/qa_utils/stdstream_line_reader.h
@@ -44,9 +44,11 @@ public:
char* ReadLine() override;
-protected:
-
- void setStream( std::istream& aStream );
+ /**
+ * Set the stream for this line reader.
+ * @param aStream a stream to read
+ */
+ void SetStream( std::istream& aStream );
private:
std::string m_buffer;
diff --git a/tools/io_benchmark/CMakeLists.txt b/tools/io_benchmark/CMakeLists.txt
index 2e50d5901..bbab2a9c6 100644
--- a/tools/io_benchmark/CMakeLists.txt
+++ b/tools/io_benchmark/CMakeLists.txt
@@ -3,7 +3,6 @@ include_directories( BEFORE ${INC_BEFORE} )
set( IOBENCHMARK_SRCS
io_benchmark.cpp
- stdstream_line_reader.cpp
)
add_executable( io_benchmark
@@ -12,6 +11,7 @@ add_executable( io_benchmark
target_link_libraries( io_benchmark
common
+ qa_utils
${wxWidgets_LIBRARIES}
)
--
2.18.0
From d4ba423132ca754122c63b5a823add38b5478a14 Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Wed, 10 Oct 2018 17:53:00 +0100
Subject: [PATCH 3/3] Docs: printing and trace
Add a quick outline of some of the ways you can dump debug during
debugging. Also include a list of known trace mask strings.
---
Documentation/development/testing.md | 70 ++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md
index 86c1e2dde..240034bbc 100644
--- a/Documentation/development/testing.md
+++ b/Documentation/development/testing.md
@@ -152,6 +152,76 @@ to the executable, using your debugger of choice, for example GDB:
$ gdb --arg qa/pcb_parse_input/qa_pcb_parse_input ./fuzzout/hangs/<name>
+# Run-time debugging #
+
+KiCad can be debugged at run-time, either under a full debugger
+such as GDB, or using simple methods like logging debug to the
+console.
+
+## Printing debug ##
+
+If you are compiling KiCad yourself, you can simply add debugging statements to
+relevant places in the code, for example:
+
+ wxLogDebug( "Value of variable: %d", my_int );
+
+This produces debug output that can only be seen when compiling
+in Debug mode.
+
+You can also use `std::cout` and `printf`.
+
+Ensure you do not leave this kind of debugging in place when
+submitting code.
+
+## Printing trace ##
+
+Some parts of the code have "trace" that can be enabled selectively according to
+a "mask", for example:
+
+ wxLogTrace( "TRACEMASK", "My trace, value: %d", my_int );
+
+This will not be printed by default. To show it, set the `WXTRACE` environment
+variable when you run KiCad to include the masks you wish to enable:
+
+ $ WXTRACE="TRACEKEY,OTHERKEY" kicad
+
+When printed, the debug will be prefixed with a timestamp and the trace mask:
+
+ 11:22:33: Trace: (KICAD_FIND_ITEM) item Symbol GNDPWR, #PWR020
+
+Some available masks:
+
+* Core KiCad functions:
+ * `KICAD_KEY_EVENTS`
+ * `KicadScrollSettings`
+ * `KICAD_FIND_ITEM`
+ * `KICAD_FIND_REPLACE`
+ * `KICAD_NGSPICE`
+ * `KICAD_PLUGINLOADER`
+ * `GAL_PROFILE`
+ * `GAL_CACHED_CONTAINER`
+ * `PNS`
+ * `CN`
+* Plugin-specific (including "standard" KiCad formats):
+ * `3D_CACHE`
+ * `3D_SG`
+ * `3D_RESOLVER`
+ * `3D_PLUGIN_MANAGER`
+ * `KI_TRACE_CCAMERA`
+ * `PLUGIN_IDF`
+ * `PLUGIN_VRML`
+ * `KICAD_SCH_LEGACY_PLUGIN`
+ * `KICAD_GEDA_PLUGIN`
+ * `KICAD_PCB_PLUGIN`
+
+
+
+
+
+
+
+
+
[CTest]: https://cmake.org/cmake/help/latest/module/CTest.html
[Boost Unit Test framework]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/index.html
[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
--
2.18.0
From 78fab837be999ed98410b67e566944734dc7859f Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Fri, 12 Oct 2018 10:58:44 +0100
Subject: [PATCH 4/4] Dev docs: add TOC to testing page
---
Documentation/development/testing.md | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md
index 240034bbc..1cb827218 100644
--- a/Documentation/development/testing.md
+++ b/Documentation/development/testing.md
@@ -1,6 +1,8 @@
# Testing KiCad #
-# Unit tests #
+[TOC]
+
+# Unit tests {#unit-tests}
KiCad has a limited number of unit tests, which can be used to
check that certain functionality works.
@@ -13,13 +15,13 @@ required to add a test to the testing suite.
The test CMake targets generally start with `qa_`, the names of the tests
within CTest are the same but without the `qa_` prefix.
-## Running tests ##
+## Running tests {#running-tests}
You can run all tests after building with `make test` or `ctest`. The latter
option allows many CTest options which can be useful, especially in automated
or CI environments.
-### Running specific tests ##
+### Running specific tests {#running-specific-tests}
To run a specific test executable, you can just run with `ctest` or run
the executable directly. Running directly is often the simplest way when
@@ -42,7 +44,7 @@ Common useful patterns:
You can rebuild just a specific test with CMake to avoid rebuilding
everything when working on a small area, e.g. `make qa_common`.
-### Writing Boost tests ###
+## Writing Boost tests {#writing-boost-tests}
Boost unit tests are straightforward to write. Individual test cases can be
registered with:
@@ -68,7 +70,7 @@ messages inside tested functions (i.e. where you don't have access to the Boost
unit test headers). These will always be printed, so take care
to remove them before committing, or they'll show up when KiCad runs normally!
-## Python modules ##
+## Python modules {#python-tests}
The Pcbnew Python modules have some test programs in the `qa` directory.
You must have the `KICAD_SCRIPTING_MODULES` option on in CMake to
@@ -87,7 +89,7 @@ from the source tree:
cd /path/to/kicad/source/qa
python2 testcase/test_001_pcb_load.py
-### Diagnosing segfaults ###
+### Diagnosing segfaults {#python-segfaults}
Although the module is Python, it links against a C++ library
(the same one used by KiCad Pcbnew), so it can segfault if the library
@@ -103,7 +105,7 @@ You can run the tests in GDB to trace this:
If the test segfaults, you will get a familiar backtrace, just like
if you were running pcbnew under GDB.
-## Fuzz testing ##
+# Fuzz testing {#fuzz-testing}
It is possible to run fuzz testing on some parts of KiCad. To do this for a
generic function, you need to be able to pass some kind of input from the fuzz
@@ -152,13 +154,13 @@ to the executable, using your debugger of choice, for example GDB:
$ gdb --arg qa/pcb_parse_input/qa_pcb_parse_input ./fuzzout/hangs/<name>
-# Run-time debugging #
+# Run-time debugging {#run-time}
KiCad can be debugged at run-time, either under a full debugger
such as GDB, or using simple methods like logging debug to the
console.
-## Printing debug ##
+## Printing debug {#print-debug}
If you are compiling KiCad yourself, you can simply add debugging statements to
relevant places in the code, for example:
@@ -173,7 +175,7 @@ You can also use `std::cout` and `printf`.
Ensure you do not leave this kind of debugging in place when
submitting code.
-## Printing trace ##
+## Printing trace {#trace-debug}
Some parts of the code have "trace" that can be enabled selectively according to
a "mask", for example:
--
2.18.0
References
