kicad-developers team mailing list archive

[John Beard <john.j.beard@xxxxxxxxx>]

Hi,

Here is an update patch that rebases over the commenting out of
pcb_test_window, polygon_triangulation and polygon_generator and fixes
a link error to do with base_screen.cpp.

Cheers,

John
On Mon, Oct 8, 2018 at 5:27 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
>
> Sorry,
>
> I wrote "ms", I meant "us" - the times are in the handful-of-millsecond range.
>
> Cheers,
>
> John
> On Mon, Oct 8, 2018 at 5:24 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
> >
> > Hi,
> >
> > This is a patch to add a test program that allows to parse a Pcbnew
> > file from command line params or stdin. This means you can use it for
> > fuzz testing.
> >
> > I have done a little bit of fuzz testing so far (8 million execs,
> > about 70% of a cycle), and have not found any crashes, but I can make
> > it hang in a few ways. These all seem to be in streams which contain
> > nul's. This is actually not reachable from the UI due to reading files
> > into wxStrings first (nut quite sure why), whereas this program uses
> > the parser directly. Thus, the bug is probably not very critical.
> > Example hanging input attached (note there's a nul in it, so your
> > editor may or may not like that).
> >
> > This program can also be fed a number of files, which means it could
> > be used for automated testing that all files in a batch can be parsed
> > successfully, and also provides a handy way to put GDB on a program
> > when debugging the parser against specific input.
> >
> > There is timing on the parsing too, mostly for interest (use the -v
> > flag). It takes about 150-3000ms per FP on my machine for the FPs in
> > Connector_PinSocket_2.54mm.pretty.
> >
> > There's also some centralisation of some QA-related utils into a
> > qa_utils library.
> >
> > Cheers,
> >
> > John

From 3ff4ee626860c81e48896a1b15ecc967d74e5e39 Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Sat, 6 Oct 2018 15:35:17 +0100
Subject: [PATCH] QA: PCB file input parse test program (fuzzable)

This adds a test program which can be used to test the
parsing of a given KiCad PCB file. This interface is
useful for both manual or automated debugging of given
files, as well as providing an interface suitable for
fuzz-testing tools.

Also adds to the testing docs to detail how fuzzing can
be used.

Also moves some useful re-usable code from io-benchmark
to a new library qa_utils, which can contain code that
isn't need in the actual KiCad libs.
---
 Documentation/development/testing.md          |  47 +++++-
 qa/CMakeLists.txt                             |   4 +
 qa/pcb_parse_input/CMakeLists.txt             |  66 ++++++++
 qa/pcb_parse_input/main.cpp                   | 158 ++++++++++++++++++
 qa/qa_utils/CMakeLists.txt                    |  40 +++++
 qa/qa_utils/scoped_timer.h                    |  62 +++++++
 .../qa_utils}/stdstream_line_reader.cpp       |   8 +-
 .../qa_utils}/stdstream_line_reader.h         |   8 +-
 tools/io_benchmark/CMakeLists.txt             |   2 +-
 9 files changed, 386 insertions(+), 9 deletions(-)
 create mode 100644 qa/pcb_parse_input/CMakeLists.txt
 create mode 100644 qa/pcb_parse_input/main.cpp
 create mode 100644 qa/qa_utils/CMakeLists.txt
 create mode 100644 qa/qa_utils/scoped_timer.h
 rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.cpp (91%)
 rename {tools/io_benchmark => qa/qa_utils}/stdstream_line_reader.h (92%)

diff --git a/Documentation/development/testing.md b/Documentation/development/testing.md
index 4726ca536..cf0d4c129 100644
--- a/Documentation/development/testing.md
+++ b/Documentation/development/testing.md
@@ -103,6 +103,51 @@ You can run the tests in GDB to trace this:
 If the test segfaults, you will get a familiar backtrace, just like
 if you were running pcbnew under GDB.
 
+## Fuzz testing ##
+
+It is possible to run fuzz testing on some parts of KiCad. To do this for a
+generic function, you need to be able to pass some kind of input from the fuzz
+testing tool to the function under test.
+
+For example, to use the [AFL fuzzing tool][], you will need:
+
+* A test executable that can:
+** Receive input from `stdin` to be run by `afl-fuzz`.
+** Optional: process input from a filename to allow `afl-tmin` to minimise the
+   input files.
+* To compile this executable with an AFL compiler, to enable the instrumentation
+  that allows the fuzzer to detect the fuzzing state.
+
+For example, the `qa_pcb_parse_input` executable can be compiled like this:
+
+    mkdir build
+    cd build
+    cmake -DCMAKE_CXX_COMPILER=/usr/bin/afl-clang-fast++ -DCMAKE_C_COMPILER=/usr/bin/afl-clang-fast ../kicad_src
+    make qa_pcb_parse_input
+
+You may need to disable core dumps and CPU frequency scaling on your system (AFL
+will warn you if you should do this). For example, as root:
+
+    # echo core >/proc/sys/kernel/core_pattern
+    # echo performance | tee cpu*/cpufreq/scaling_governor
+
+To fuzz:
+
+    afl-fuzz -i fuzzin -o fuzzout -m500 qa/pcb_parse_input/qa_pcb_parse_input
+
+where:
+
+* `-i` is a directory of files to use as fuzz input "seeds"
+* `-o` is a directory to write the results (including inputs that provoke crashes
+  or hangs)
+* `-t` is the maximum time that a run is allowed to take before being declared a "hang"
+* `-m` is the memory allowed to use (this often needs to be bumped, as KiCad code
+  tends to use a lot of memory to initialise)
+
+The AFL TUI will then display the fuzzing progress, and you can use the hang- or
+crash-provoking inputs to debug code as needed.
+
 [CTest]: https://cmake.org/cmake/help/latest/module/CTest.html
 [Boost Unit Test framework]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/index.html
-[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
\ No newline at end of file
+[boost-test-functions]: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/utf_reference/testing_tool_ref.html
+[AFL fuzzing tool]: http://lcamtuf.coredump.cx/afl/
\ No newline at end of file
diff --git a/qa/CMakeLists.txt b/qa/CMakeLists.txt
index 36aa7b601..ae30e1a38 100644
--- a/qa/CMakeLists.txt
+++ b/qa/CMakeLists.txt
@@ -12,8 +12,12 @@ if( KICAD_SCRIPTING_MODULES )
 
 endif()
 
+# common QA helpers
+add_subdirectory( qa_utils )
+
 add_subdirectory( common )
 add_subdirectory( shape_poly_set_refactor )
+add_subdirectory( pcb_parse_input )
 # add_subdirectory( pcb_test_window )
 # add_subdirectory( polygon_triangulation )
 # add_subdirectory( polygon_generator )
\ No newline at end of file
diff --git a/qa/pcb_parse_input/CMakeLists.txt b/qa/pcb_parse_input/CMakeLists.txt
new file mode 100644
index 000000000..5f8e2f350
--- /dev/null
+++ b/qa/pcb_parse_input/CMakeLists.txt
@@ -0,0 +1,66 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+
+
+if( BUILD_GITHUB_PLUGIN )
+    set( GITHUB_PLUGIN_LIBRARIES github_plugin )
+endif()
+
+add_executable( qa_pcb_parse_input
+    # This is needed for the global mock objects
+    ../qa_utils/mocks.cpp
+
+    main.cpp
+
+  ../../common/base_units.cpp
+  ../../common/xnode.cpp
+  ../../common/base_screen.cpp
+)
+
+include_directories(
+    ${CMAKE_SOURCE_DIR}
+    ${CMAKE_SOURCE_DIR}/include
+    ${CMAKE_SOURCE_DIR}/polygon
+    ${CMAKE_SOURCE_DIR}/pcbnew
+    ${CMAKE_SOURCE_DIR}/common
+    ${CMAKE_SOURCE_DIR}/pcbnew/router
+    ${CMAKE_SOURCE_DIR}/pcbnew/tools
+    ${CMAKE_SOURCE_DIR}/pcbnew/dialogs
+    ${Boost_INCLUDE_DIR}
+    ${INC_AFTER}
+)
+
+target_link_libraries( qa_pcb_parse_input
+    legacy_wx
+    pcbcommon
+    common
+    bitmaps
+    polygon
+    pcad2kicadpcb
+    ${GITHUB_PLUGIN_LIBRARIES}
+    qa_utils
+    ${wxWidgets_LIBRARIES}
+)
+
+# we need to pretend to be something to appease the units code
+target_compile_definitions( qa_pcb_parse_input
+    PRIVATE PCBNEW
+)
\ No newline at end of file
diff --git a/qa/pcb_parse_input/main.cpp b/qa/pcb_parse_input/main.cpp
new file mode 100644
index 000000000..fc79f932b
--- /dev/null
+++ b/qa/pcb_parse_input/main.cpp
@@ -0,0 +1,158 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+ */
+
+
+#include <kicad_plugin.h>
+#include <pcb_parser.h>
+#include <richio.h>
+#include <class_board_item.h>
+
+#include <wx/cmdline.h>
+
+#include <stdstream_line_reader.h>
+#include <scoped_timer.h>
+
+using PARSE_DURATION = std::chrono::microseconds;
+
+/**
+ * Parse a PCB or footprint file from the given input stream
+ *
+ * @param aStream the input stream to read from
+ * @return success, duration (in us)
+ */
+bool parse(std::istream& aStream, bool aVerbose )
+{
+    // Take input from stdin
+    STDISTREAM_LINE_READER reader;
+    reader.SetStream( aStream );
+
+    PCB_PARSER parser;
+
+    parser.SetLineReader( &reader );
+
+    BOARD_ITEM* board = nullptr;
+
+    PARSE_DURATION duration {};
+
+    try
+    {
+        SCOPED_TIMER<PARSE_DURATION> timer( duration );
+        board = parser.Parse();
+    }
+    catch( const IO_ERROR& parse_error )
+    {
+        std::cerr << parse_error.Problem() << std::endl;
+        std::cerr << parse_error.Where() << std::endl;
+    }
+
+    if( aVerbose )
+    {
+        std::cout << "Took: " << duration.count() << "us" << std::endl;
+    }
+
+    return board != nullptr;
+}
+
+
+static const wxCmdLineEntryDesc g_cmdLineDesc [] =
+{
+    { wxCMD_LINE_SWITCH, "h", "help",
+        _( "displays help on the command line parameters" ),
+        wxCMD_LINE_VAL_NONE, wxCMD_LINE_OPTION_HELP },
+    { wxCMD_LINE_SWITCH, "v", "verbose",
+        _( "print parsing information") },
+    { wxCMD_LINE_PARAM, nullptr, nullptr,
+        _( "input file" ),
+        wxCMD_LINE_VAL_STRING, wxCMD_LINE_PARAM_OPTIONAL | wxCMD_LINE_PARAM_MULTIPLE },
+    { wxCMD_LINE_NONE }
+};
+
+
+enum RET_CODES
+{
+    OK = 0,
+    BAD_CMDLINE = 1,
+    PARSE_FAILED = 2,
+};
+
+
+int main(int argc, char** argv)
+{
+#ifdef __AFL_COMPILER
+    __AFL_INIT();
+#endif
+
+    wxMessageOutput::Set(new wxMessageOutputStderr);
+    wxCmdLineParser cl_parser( argc, argv );
+    cl_parser.SetDesc( g_cmdLineDesc );
+    cl_parser.AddUsageText( _("This program parses PCB files, either from the "
+        "stdin stream or from the given filenames. This can be used either for "
+        "standalone testing of the parser or for fuzz testing." ) );
+
+    int cmd_parsed_ok = cl_parser.Parse();
+    if( cmd_parsed_ok != 0 )
+    {
+        // Help and invalid input both stop here
+        return ( cmd_parsed_ok == -1 ) ? RET_CODES::OK : RET_CODES::BAD_CMDLINE;
+    }
+
+    const bool verbose = cl_parser.Found( "verbose" );
+
+    bool ok = true;
+    PARSE_DURATION duration;
+
+    const auto file_count = cl_parser.GetParamCount();
+
+    if ( file_count == 0 )
+    {
+        // Parse the file provided on stdin - used by AFL to drive the
+        // program
+        // while (__AFL_LOOP(2))
+        {
+            ok = parse( std::cin, verbose );
+        }
+    }
+    else
+    {
+        // Parse 'n' files given on the command line
+        // (this is useful for input minimisation (e.g. afl-tmin) as
+        // well as manual testing
+        for( unsigned i = 0; i < file_count; i++ )
+        {
+            const auto filename = cl_parser.GetParam( i );
+
+            if( verbose )
+                std::cout << "Parsing: " << filename << std::endl;
+
+            std::ifstream fin;
+            fin.open( filename );
+
+            ok = ok && parse( fin, verbose );
+        }
+    }
+
+    if( !ok )
+        return RET_CODES::PARSE_FAILED;
+
+    return RET_CODES::OK;
+}
\ No newline at end of file
diff --git a/qa/qa_utils/CMakeLists.txt b/qa/qa_utils/CMakeLists.txt
new file mode 100644
index 000000000..b5befaeee
--- /dev/null
+++ b/qa/qa_utils/CMakeLists.txt
@@ -0,0 +1,40 @@
+# This program source code file is part of KiCad, a free EDA CAD application.
+#
+# Copyright (C) 2018 KiCad Developers, see CHANGELOG.TXT for contributors.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, you may find one here:
+# http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+# or you may search the http://www.gnu.org website for the version 2 license,
+# or you may write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+
+set( QA_UTIL_COMMON_SRC
+    stdstream_line_reader.cpp
+)
+
+# A generic library of useful functions for various testing purposes
+add_library( qa_utils
+    ${QA_UTIL_COMMON_SRC}
+)
+
+include_directories( BEFORE ${INC_BEFORE} )
+
+target_link_libraries( qa_utils
+    common
+    ${wxWidgets_LIBRARIES}
+)
+
+target_include_directories( qa_utils PUBLIC
+    ${CMAKE_CURRENT_SOURCE_DIR}
+)
\ No newline at end of file
diff --git a/qa/qa_utils/scoped_timer.h b/qa/qa_utils/scoped_timer.h
new file mode 100644
index 000000000..028bfef46
--- /dev/null
+++ b/qa/qa_utils/scoped_timer.h
@@ -0,0 +1,62 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+ */
+
+#ifndef SCOPED_TIMER_H
+#define SCOPED_TIMER_H
+
+#include <chrono>
+
+/**
+ * A simple RAII class to measure the time of an operation.
+ *
+ * ON construction, a timer is started, and on destruction, the timer is
+ * ended, and the time difference is written into the given duration
+ */
+template<typename DURATION>
+class SCOPED_TIMER
+{
+    using CLOCK = std::chrono::steady_clock;
+    using TIME_PT = std::chrono::time_point<CLOCK>;
+
+public:
+    SCOPED_TIMER( DURATION& aDuration ):
+        m_duration( aDuration )
+    {
+        m_start = CLOCK::now();
+    }
+
+    ~SCOPED_TIMER()
+    {
+        const auto end = CLOCK::now();
+
+        // update the output
+        m_duration = std::chrono::duration_cast<DURATION>( end - m_start );
+    }
+
+private:
+
+    DURATION& m_duration;
+    TIME_PT m_start;
+};
+
+#endif // SCOPED_TIMER_h
\ No newline at end of file
diff --git a/tools/io_benchmark/stdstream_line_reader.cpp b/qa/qa_utils/stdstream_line_reader.cpp
similarity index 91%
rename from tools/io_benchmark/stdstream_line_reader.cpp
rename to qa/qa_utils/stdstream_line_reader.cpp
index cb8ddabdc..3f6596fdb 100644
--- a/tools/io_benchmark/stdstream_line_reader.cpp
+++ b/qa/qa_utils/stdstream_line_reader.cpp
@@ -1,7 +1,7 @@
 /*
  * This program source code file is part of KiCad, a free EDA CAD application.
  *
- * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ * Copyright (C) 2018 KiCad Developers, see AUTHORS.txt for contributors.
  *
  * This program is free software; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License
@@ -58,10 +58,10 @@ char* STDISTREAM_LINE_READER::ReadLine()
 }
 
 
-void STDISTREAM_LINE_READER::setStream( std::istream& aStream )
+void STDISTREAM_LINE_READER::SetStream( std::istream& aStream )
 {
     // Could be done with a virtual getStream function, but the
-    // virtual function call is a noticable (but minor) penalty within
+    // virtual function call is a noticeable (but minor) penalty within
     // ReadLine() in tight loops
     m_stream = &aStream;
 }
@@ -77,7 +77,7 @@ IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxFileName& aFileName )  :
         THROW_IO_ERROR( msg );
     }
 
-    setStream( m_fStream );
+    SetStream( m_fStream );
 
     m_source = aFileName.GetFullName();
 }
diff --git a/tools/io_benchmark/stdstream_line_reader.h b/qa/qa_utils/stdstream_line_reader.h
similarity index 92%
rename from tools/io_benchmark/stdstream_line_reader.h
rename to qa/qa_utils/stdstream_line_reader.h
index 5ccf86f3c..81c238dc6 100644
--- a/tools/io_benchmark/stdstream_line_reader.h
+++ b/qa/qa_utils/stdstream_line_reader.h
@@ -44,9 +44,11 @@ public:
 
     char* ReadLine()  override;
 
-protected:
-
-    void setStream( std::istream&  aStream );
+    /**
+     * Set the stream for this line reader.
+     * @param aStream a stream to read
+     */
+    void SetStream( std::istream&  aStream );
 
 private:
     std::string m_buffer;
diff --git a/tools/io_benchmark/CMakeLists.txt b/tools/io_benchmark/CMakeLists.txt
index 2e50d5901..bbab2a9c6 100644
--- a/tools/io_benchmark/CMakeLists.txt
+++ b/tools/io_benchmark/CMakeLists.txt
@@ -3,7 +3,6 @@ include_directories( BEFORE ${INC_BEFORE} )
 
 set( IOBENCHMARK_SRCS
     io_benchmark.cpp
-    stdstream_line_reader.cpp
 )
 
 add_executable( io_benchmark
@@ -12,6 +11,7 @@ add_executable( io_benchmark
 
 target_link_libraries( io_benchmark
     common
+    qa_utils
     ${wxWidgets_LIBRARIES}
 )
 
-- 
2.19.0