kicad-developers team mailing list archive
-
kicad-developers team
-
Mailing list archive
-
Message #37970
Re: [PATCH] Fuzzable PCB parsing test harness
Hi Wayne,
I think I might have fixed the ordering in the link libraries (at
least, it now builds on Jenkins).
This is just the first patch to focus on the build error, I'll rebase
the other docs stuff later if/when it works.
Cheers,
John
On Fri, Oct 12, 2018 at 7:58 PM Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
>
> John,
>
> This patch fails to link on windows. I've attached the build error.
>
> Wayne
>
> On 10/9/2018 9:53 AM, John Beard wrote:
> > Hi,
> >
> > Here is an update patch that rebases over the commenting out of
> > pcb_test_window, polygon_triangulation and polygon_generator and fixes
> > a link error to do with base_screen.cpp.
> >
> > Cheers,
> >
> > John
> > On Mon, Oct 8, 2018 at 5:27 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> >>
> >> Sorry,
> >>
> >> I wrote "ms", I meant "us" - the times are in the handful-of-millsecond range.
> >>
> >> Cheers,
> >>
> >> John
> >> On Mon, Oct 8, 2018 at 5:24 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> >>>
> >>> Hi,
> >>>
> >>> This is a patch to add a test program that allows to parse a Pcbnew
> >>> file from command line params or stdin. This means you can use it for
> >>> fuzz testing.
> >>>
> >>> I have done a little bit of fuzz testing so far (8 million execs,
> >>> about 70% of a cycle), and have not found any crashes, but I can make
> >>> it hang in a few ways. These all seem to be in streams which contain
> >>> nul's. This is actually not reachable from the UI due to reading files
> >>> into wxStrings first (nut quite sure why), whereas this program uses
> >>> the parser directly. Thus, the bug is probably not very critical.
> >>> Example hanging input attached (note there's a nul in it, so your
> >>> editor may or may not like that).
> >>>
> >>> This program can also be fed a number of files, which means it could
> >>> be used for automated testing that all files in a batch can be parsed
> >>> successfully, and also provides a handy way to put GDB on a program
> >>> when debugging the parser against specific input.
> >>>
> >>> There is timing on the parsing too, mostly for interest (use the -v
> >>> flag). It takes about 150-3000ms per FP on my machine for the FPs in
> >>> Connector_PinSocket_2.54mm.pretty.
> >>>
> >>> There's also some centralisation of some QA-related utils into a
> >>> qa_utils library.
> >>>
> >>> Cheers,
> >>>
> >>> John
> >>>
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~kicad-developers
> >>> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~kicad-developers
> >>> More help : https://help.launchpad.net/ListHelp
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help : https://help.launchpad.net/ListHelp
From 985a3e5e0007d2a7b17bdfd5173a65fb3de0eb60 Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Sat, 6 Oct 2018 15:35:17 +0100
Subject: [PATCH 1/2] QA: PCB file input parse test program (fuzzable)
This adds a test program which can be used to test the
parsing of a given KiCad PCB file. This interface is
useful for both manual or automated debugging of given
files, as well as providing an interface suitable for
fuzz-testing tools.
Also adds to the testing docs to detail how fuzzing can
be used.
Also moves some useful re-usable code from io-benchmark
to a new library qa_utils, which can contain code that
isn't need in the actual KiCad libs.
---
Documentation/development/testing.md | 47 +++++-
qa/CMakeLists.txt | 4 +
qa/pcb_parse_input/CMakeLists.txt | 67 ++++++++
qa/pcb_parse_input/main.cpp | 158 ++++++++++++++++++
qa/qa_utils/CMakeLists.txt | 40 +++++
qa/qa_utils/scoped_timer.h | 62 +++++++
.../qa_utils}/stdstream_line_reader.cpp | 8 +-
.../qa_utils}/stdstream_line_reader.h | 8 +-
tools/io_benchmark/CMakeLists.txt | 2 +-
9 files changed, 387 insertions(+), 9 deletions(-)
create mode 100644 qa/pcb_parse_input/CMakeLists.txt
create mode 100644 qa/pcb_parse_input/main.cpp
create mode 100644 qa/qa_utils/CMakeLists.txt
create mode 100644 qa/qa_utils/scoped_timer.h
rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.cpp (91%)
rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.h (92%)
diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md
index 4726ca536..cf0d4c129 100644
--- a/Documentation/development/testing.md
+++ b/Documentation/development/testing.md
@@ -103,6 +103,51 @@ You can run the tests in GDB to trace this:
If the test segfaults, you will get a familiar backtrace, just like
if you were running pcbnew under GDB.
+## Fuzz testing ##
+
+It is possible to run fuzz testing on some parts of KiCad. To do this for a
+generic function, you need to be able to pass some kind of input from the fuzz
+testing tool to the function under test.
+
+For example, to use the [AFL fuzzing tool][], you will need:
+
+* A test executable that can:
+** Receive input from `stdin` to be run by `afl-fuzz`.
+** Optional: process input from a filename to allow `afl-tmin` to minimise the
+ input files.
+* To compile this executable with an AFL compiler, to enable the instrumentation
+ that allows the fuzzer to detect the fuzzing state.
+
+For example, the `qa_pcb_parse_input` executable can be compiled like this:
+
+ mkdir build
+ cd build
+ cmake -DCMAKE_CXX_COMPILER=/usr/bin/afl-clang-fast++ -DCMAKE_C_COMPILER=/usr/bin/afl-clang-fast ../kicad_src
+ make qa_pcb_parse_input
+
+You may need to disable core dumps and CPU frequency scaling on your system (AFL
+will warn you if you should do this). For example, as root:
+
+ # echo core >/proc/sys/kernel/core_pattern
+ # echo performance | tee cpu*/cpufreq/scaling_governor
+
+To fuzz:
+
+ afl-fuzz -i fuzzin -o fuzzout -m500 qa/pcb_parse_input/qa_pcb_parse_input
+
+where:
+
+* `-i` is a directory of files to use as fuzz input "seeds"
+* `-o` is a directory to write the results (including inputs that provoke crashes
+ or hangs)
+* `-t` is the maximum time that a run is allowed to take before being declared a "hang"
+* `-m` is the memory allowed to use (this often needs to be bumped, as KiCad code
+ tends to use a lot of memory to initialise)
+
+The AFL TUI will then display the fuzzing progress, and you can use the hang- or
+crash-provoking inputs to debug code as needed.
+
[CTest]: https://cmake.org/cmake/help/latest/module/CTest.html
[Boost Unit Test framework]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/index.html
-[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
\ No newline at end of file
+[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
+[AFL fuzzing tool]: http://lcamtuf.coredump.cx/afl/
\ No newline at end of file
diff --git a/qa/CMakeLists.txt b/qa/CMakeLists.txt
index 36aa7b601..ae30e1a38 100644
--- a/qa/CMakeLists.txt
+++ b/qa/CMakeLists.txt
@@ -12,8 +12,12 @@ if( KICAD_SCRIPTING_MODULES )
endif()
+# common QA helpers
+add_subdirectory( qa_utils )
+
add_subdirectory( common )
add_subdirectory( shape_poly_set_refactor )
+add_subdirectory( pcb_parse_input )
# add_subdirectory( pcb_test_window )
# add_subdirectory( polygon_triangulation )
# add_subdirectory( polygon_generator )
\ No newline at end of file
diff --git a/qa/pcb_parse_input/CMakeLists.txt b/qa/pcb_parse_input/CMakeLists.txt
new file mode 100644
index 000000000..d45bf3b16
--- /dev/null
+++ b/qa/pcb_parse_input/CMakeLists.txt
@@ -0,0 +1,67 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+
+if( BUILD_GITHUB_PLUGIN )
+ set( GITHUB_PLUGIN_LIBRARIES github_plugin )
+endif()
+
+add_executable( qa_pcb_parse_input
+ # This is needed for the global mock objects
+ ../qa_utils/mocks.cpp
+
+ main.cpp
+
+ ../../common/base_units.cpp
+ ../../common/xnode.cpp
+ ../../common/base_screen.cpp
+)
+
+include_directories(
+ ${CMAKE_SOURCE_DIR}
+ ${CMAKE_SOURCE_DIR}/include
+ ${CMAKE_SOURCE_DIR}/polygon
+ ${CMAKE_SOURCE_DIR}/pcbnew
+ ${CMAKE_SOURCE_DIR}/common
+ ${CMAKE_SOURCE_DIR}/pcbnew/router
+ ${CMAKE_SOURCE_DIR}/pcbnew/tools
+ ${CMAKE_SOURCE_DIR}/pcbnew/dialogs
+ ${Boost_INCLUDE_DIR}
+ ${INC_AFTER}
+)
+
+target_link_libraries( qa_pcb_parse_input
+ pcbcommon
+ legacy_wx
+ common
+ bitmaps
+ polygon
+ gal
+ pcad2kicadpcb
+ ${GITHUB_PLUGIN_LIBRARIES}
+ qa_utils
+ ${wxWidgets_LIBRARIES}
+)
+
+# we need to pretend to be something to appease the units code
+target_compile_definitions( qa_pcb_parse_input
+ PRIVATE PCBNEW
+)
\ No newline at end of file
diff --git a/qa/pcb_parse_input/main.cpp b/qa/pcb_parse_input/main.cpp
new file mode 100644
index 000000000..fc79f932b
--- /dev/null
+++ b/qa/pcb_parse_input/main.cpp
@@ -0,0 +1,158 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+
+#include <kicad_plugin.h>
+#include <pcb_parser.h>
+#include <richio.h>
+#include <class_board_item.h>
+
+#include <wx/cmdline.h>
+
+#include <stdstream_line_reader.h>
+#include <scoped_timer.h>
+
+using PARSE_DURATION = std::chrono::microseconds;
+
+/**
+ * Parse a PCB or footprint file from the given input stream
+ *
+ * @param aStream the input stream to read from
+ * @return success, duration (in us)
+ */
+bool parse(std::istream& aStream, bool aVerbose )
+{
+ // Take input from stdin
+ STDISTREAM_LINE_READER reader;
+ reader.SetStream( aStream );
+
+ PCB_PARSER parser;
+
+ parser.SetLineReader( &reader );
+
+ BOARD_ITEM* board = nullptr;
+
+ PARSE_DURATION duration {};
+
+ try
+ {
+ SCOPED_TIMER<PARSE_DURATION> timer( duration );
+ board = parser.Parse();
+ }
+ catch( const IO_ERROR& parse_error )
+ {
+ std::cerr << parse_error.Problem() << std::endl;
+ std::cerr << parse_error.Where() << std::endl;
+ }
+
+ if( aVerbose )
+ {
+ std::cout << "Took: " << duration.count() << "us" << std::endl;
+ }
+
+ return board != nullptr;
+}
+
+
+static const wxCmdLineEntryDesc g_cmdLineDesc [] =
+{
+ { wxCMD_LINE_SWITCH, "h", "help",
+ _( "displays help on the command line parameters" ),
+ wxCMD_LINE_VAL_NONE, wxCMD_LINE_OPTION_HELP },
+ { wxCMD_LINE_SWITCH, "v", "verbose",
+ _( "print parsing information") },
+ { wxCMD_LINE_PARAM, nullptr, nullptr,
+ _( "input file" ),
+ wxCMD_LINE_VAL_STRING, wxCMD_LINE_PARAM_OPTIONAL | wxCMD_LINE_PARAM_MULTIPLE },
+ { wxCMD_LINE_NONE }
+};
+
+
+enum RET_CODES
+{
+ OK = 0,
+ BAD_CMDLINE = 1,
+ PARSE_FAILED = 2,
+};
+
+
+int main(int argc, char** argv)
+{
+#ifdef __AFL_COMPILER
+ __AFL_INIT();
+#endif
+
+ wxMessageOutput::Set(new wxMessageOutputStderr);
+ wxCmdLineParser cl_parser( argc, argv );
+ cl_parser.SetDesc( g_cmdLineDesc );
+ cl_parser.AddUsageText( _("This program parses PCB files, either from the "
+ "stdin stream or from the given filenames. This can be used either for "
+ "standalone testing of the parser or for fuzz testing." ) );
+
+ int cmd_parsed_ok = cl_parser.Parse();
+ if( cmd_parsed_ok != 0 )
+ {
+ // Help and invalid input both stop here
+ return ( cmd_parsed_ok == -1 ) ? RET_CODES::OK : RET_CODES::BAD_CMDLINE;
+ }
+
+ const bool verbose = cl_parser.Found( "verbose" );
+
+ bool ok = true;
+ PARSE_DURATION duration;
+
+ const auto file_count = cl_parser.GetParamCount();
+
+ if ( file_count == 0 )
+ {
+ // Parse the file provided on stdin - used by AFL to drive the
+ // program
+ // while (__AFL_LOOP(2))
+ {
+ ok = parse( std::cin, verbose );
+ }
+ }
+ else
+ {
+ // Parse 'n' files given on the command line
+ // (this is useful for input minimisation (e.g. afl-tmin) as
+ // well as manual testing
+ for( unsigned i = 0; i < file_count; i++ )
+ {
+ const auto filename = cl_parser.GetParam( i );
+
+ if( verbose )
+ std::cout << "Parsing: " << filename << std::endl;
+
+ std::ifstream fin;
+ fin.open( filename );
+
+ ok = ok && parse( fin, verbose );
+ }
+ }
+
+ if( !ok )
+ return RET_CODES::PARSE_FAILED;
+
+ return RET_CODES::OK;
+}
\ No newline at end of file
diff --git a/qa/qa_utils/CMakeLists.txt b/qa/qa_utils/CMakeLists.txt
new file mode 100644
index 000000000..b5befaeee
--- /dev/null
+++ b/qa/qa_utils/CMakeLists.txt
@@ -0,0 +1,40 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+
+set( QA_UTIL_COMMON_SRC
+ stdstream_line_reader.cpp
+)
+
+# A generic library of useful functions for various testing purposes
+add_library( qa_utils
+ ${QA_UTIL_COMMON_SRC}
+)
+
+include_directories( BEFORE ${INC_BEFORE} )
+
+target_link_libraries( qa_utils
+ common
+ ${wxWidgets_LIBRARIES}
+)
+
+target_include_directories( qa_utils PUBLIC
+ ${CMAKE_CURRENT_SOURCE_DIR}
+)
\ No newline at end of file
diff --git a/qa/qa_utils/scoped_timer.h b/qa/qa_utils/scoped_timer.h
new file mode 100644
index 000000000..028bfef46
--- /dev/null
+++ b/qa/qa_utils/scoped_timer.h
@@ -0,0 +1,62 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
+ */
+
+#ifndef SCOPED_TIMER_H
+#define SCOPED_TIMER_H
+
+#include <chrono>
+
+/**
+ * A simple RAII class to measure the time of an operation.
+ *
+ * ON construction, a timer is started, and on destruction, the timer is
+ * ended, and the time difference is written into the given duration
+ */
+template<typename DURATION>
+class SCOPED_TIMER
+{
+ using CLOCK = std::chrono::steady_clock;
+ using TIME_PT = std::chrono::time_point<CLOCK>;
+
+public:
+ SCOPED_TIMER( DURATION& aDuration ):
+ m_duration( aDuration )
+ {
+ m_start = CLOCK::now();
+ }
+
+ ~SCOPED_TIMER()
+ {
+ const auto end = CLOCK::now();
+
+ // update the output
+ m_duration = std::chrono::duration_cast<DURATION>( end - m_start );
+ }
+
+private:
+
+ DURATION& m_duration;
+ TIME_PT m_start;
+};
+
+#endif // SCOPED_TIMER_h
\ No newline at end of file
diff --git a/tools/io_benchmark/stdstream_line_reader.cpp b/qa/qa_utils/stdstream_line_reader.cpp
similarity index 91%
rename from tools/io_benchmark/stdstream_line_reader.cpp
rename to qa/qa_utils/stdstream_line_reader.cpp
index cb8ddabdc..3f6596fdb 100644
--- a/tools/io_benchmark/stdstream_line_reader.cpp
+++ b/qa/qa_utils/stdstream_line_reader.cpp
@@ -1,7 +1,7 @@
/*
* This program source code file is part of KiCad, a free EDA CAD application.
*
- * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
@@ -58,10 +58,10 @@ char* STDISTREAM_LINE_READER::ReadLine()
}
-void STDISTREAM_LINE_READER::setStream( std::istream& aStream )
+void STDISTREAM_LINE_READER::SetStream( std::istream& aStream )
{
// Could be done with a virtual getStream function, but the
- // virtual function call is a noticable (but minor) penalty within
+ // virtual function call is a noticeable (but minor) penalty within
// ReadLine() in tight loops
m_stream = &aStream;
}
@@ -77,7 +77,7 @@ IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName ) :
THROW_IO_ERROR( msg );
}
- setStream( m_fStream );
+ SetStream( m_fStream );
m_source = aFileName.GetFullName();
}
diff --git a/tools/io_benchmark/stdstream_line_reader.h b/qa/qa_utils/stdstream_line_reader.h
similarity index 92%
rename from tools/io_benchmark/stdstream_line_reader.h
rename to qa/qa_utils/stdstream_line_reader.h
index 5ccf86f3c..81c238dc6 100644
--- a/tools/io_benchmark/stdstream_line_reader.h
+++ b/qa/qa_utils/stdstream_line_reader.h
@@ -44,9 +44,11 @@ public:
char* ReadLine() override;
-protected:
-
- void setStream( std::istream& aStream );
+ /**
+ * Set the stream for this line reader.
+ * @param aStream a stream to read
+ */
+ void SetStream( std::istream& aStream );
private:
std::string m_buffer;
diff --git a/tools/io_benchmark/CMakeLists.txt b/tools/io_benchmark/CMakeLists.txt
index 2e50d5901..bbab2a9c6 100644
--- a/tools/io_benchmark/CMakeLists.txt
+++ b/tools/io_benchmark/CMakeLists.txt
@@ -3,7 +3,6 @@ include_directories( BEFORE ${INC_BEFORE} )
set( IOBENCHMARK_SRCS
io_benchmark.cpp
- stdstream_line_reader.cpp
)
add_executable( io_benchmark
@@ -12,6 +11,7 @@ add_executable( io_benchmark
target_link_libraries( io_benchmark
common
+ qa_utils
${wxWidgets_LIBRARIES}
)
--
2.19.0
Follow ups
References
