kicad-developers team mailing list archive

[John Beard <john.j.beard@xxxxxxxxx>]

Hi Wayne,

Sorry for the delay, I've been a bit distracted by other things. The
attached patch adds io_benchmark as a new target under tools. For now,
it only includes LINE_READER input benchmarks.

It currently supports:

* Existing FILE_LINE_READERs
* Raw std::ifstream reading (with getline, so no line length limits)
* LINE_READER based on std::istream, ditto for line length limits, and
providing a simple ifstream implementation (pass a filename)
* wxInputStream LINE_READERs, for both File and FFile variants.

Running this on Linux, on a 6.4MB file made by concatenaing all .lib
files in the kicad library (for somewhat realistic data), gives:

IO Bench Mark Util
  Benchmark file: /tmp/all.lib
  Repetitions:    5

std::fstream              794645 lines, acc: 62758460 in 41 ms
std::fstream, reused      794645 lines, acc: 62758460 in 41 ms
RICHIO                    794645 lines, acc: 62758460 in 224 ms
RICHIO, reused            794645 lines, acc: 62758460 in 223 ms
std::ifstream L_R         794645 lines, acc: 62758460 in 54 ms
std::ifstream L_R, reused 794645 lines, acc: 62758460 in 52 ms
wxIStream                 794645 lines, acc: 62758460 in 8817 ms
wxIStream, reused         794645 lines, acc: 62758460 in 9115 ms
wxFFIStream               794645 lines, acc: 62758460 in 1444 ms
wxFFIStream, reused       794645 lines, acc: 62758460 in 1441 ms

Note that this only considers the LINE_READER classes, which are the
right tool when reading line-based formats like .lib, but not really
for things like sexp which don't have to have line breaks at all, or
if they do, don't have to put them in "friendly" places. But that's
the parser's problem!

Cheers,

John

On Thu, Mar 2, 2017 at 7:31 AM, Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
> On 2/19/2017 4:27 AM, John Beard wrote:
>> On Sat, Feb 18, 2017 at 1:08 AM, Nox <noxfiregalaxy@xxxxxxxxx> wrote:
>>> What about wxFFileInputStream instead of wxFileInputStream?
>>>
>>
>> wxFFileInputStream appears to be about 5-6 times faster than
>> wxFileInputStream, but that's still much much slower than RICHIO or
>> std::ifstream.
>>
>> $ qa/io_benchmark/io_benchmark /tmp/all.lib 2
>> IO Bench Mark Util
>>   Benchmark file: /tmp/all.lib
>>   Repetitions:    2
>> std::fstream              317858 lines, acc: 25103384 in 16 ms
>> std::fstream, reused      317858 lines, acc: 25103384 in 16 ms
>> RICHIO                    317858 lines, acc: 25103384 in 91 ms
>> RICHIO, reused            317858 lines, acc: 25103384 in 90 ms
>> New fstream IO            317858 lines, acc: 25103384 in 19 ms
>> New fstream IO, reused    317858 lines, acc: 25103384 in 19 ms
>> wxIStream                 317858 lines, acc: 25103384 in 3558 ms
>> wxIStream, reused         317858 lines, acc: 25103384 in 3429 ms
>> wxFFIStream               317858 lines, acc: 25103384 in 589 ms
>> wxFFIStream, reused       317858 lines, acc: 25103384 in 602 ms
>>
>
> John,
>
> Did you create a patch for your wxInputStream benchmarking code?  I
> would like to merge this.  I think it would be a useful tool for developers.
>
> Thanks,
>
> wayne
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp

From c9b03211ea274800611cc39331b814f73ef0a1e1 Mon Sep 17 00:00:00 2001
From: John Beard <john.j.beard@xxxxxxxxx>
Date: Wed, 15 Feb 2017 16:47:41 +0800
Subject: [PATCH] IO benchmark program: RICHIO vs std::fstream and wxStream

This compares the performance of RICHIO line readers against other
implementations, and can be used for profiling and optimisation.

Current benchmarks provided:

* richio FILE_LINE_READER
* Raw std::ifstream (no LINE_READER wrapper), using getline (so no line
length limiting)
* LINE_READER wrapper around std::istream, with a std::ifstream
implementation
* Existing richio wxInputStream wrappers (with File and FFile
implemntations)
---
 tools/CMakeLists.txt                         |   2 +
 tools/io_benchmark/CMakeLists.txt            |  17 ++
 tools/io_benchmark/io_benchmark.cpp          | 321 +++++++++++++++++++++++++++
 tools/io_benchmark/stdstream_line_reader.cpp |  90 ++++++++
 tools/io_benchmark/stdstream_line_reader.h   |  70 ++++++
 5 files changed, 500 insertions(+)
 create mode 100644 tools/io_benchmark/CMakeLists.txt
 create mode 100644 tools/io_benchmark/io_benchmark.cpp
 create mode 100644 tools/io_benchmark/stdstream_line_reader.cpp
 create mode 100644 tools/io_benchmark/stdstream_line_reader.h

diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt
index 21f47d0a7..7955536b5 100644
--- a/tools/CMakeLists.txt
+++ b/tools/CMakeLists.txt
@@ -35,3 +35,5 @@ add_executable( property_tree
 target_link_libraries( property_tree
     ${wxWidgets_LIBRARIES}
     )
+
+add_subdirectory( io_benchmark )
diff --git a/tools/io_benchmark/CMakeLists.txt b/tools/io_benchmark/CMakeLists.txt
new file mode 100644
index 000000000..2e50d5901
--- /dev/null
+++ b/tools/io_benchmark/CMakeLists.txt
@@ -0,0 +1,17 @@
+
+include_directories( BEFORE ${INC_BEFORE} )
+
+set( IOBENCHMARK_SRCS
+    io_benchmark.cpp
+    stdstream_line_reader.cpp
+)
+
+add_executable( io_benchmark
+    ${IOBENCHMARK_SRCS}
+)
+
+target_link_libraries( io_benchmark
+    common
+    ${wxWidgets_LIBRARIES}
+)
+
diff --git a/tools/io_benchmark/io_benchmark.cpp b/tools/io_benchmark/io_benchmark.cpp
new file mode 100644
index 000000000..3eea0c095
--- /dev/null
+++ b/tools/io_benchmark/io_benchmark.cpp
@@ -0,0 +1,321 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+ */
+
+#include <wx/wx.h>
+#include <richio.h>
+
+#include <chrono>
+#include <ios>
+#include <functional>
+#include <iostream>
+
+#include <fstream>
+
+#include <wx/wfstream.h>
+
+#include "stdstream_line_reader.h"
+
+
+using CLOCK = std::chrono::steady_clock;
+using TIME_PT = std::chrono::time_point<CLOCK>;
+
+
+struct BENCH_REPORT
+{
+    unsigned linesRead;
+
+    /**
+     * Char accumulator, used to prevent compilers optimising away
+     * otherwise unused line buffers, and also as a primitive sanity
+     * check that the same lines were read
+     */
+    unsigned charAcc;
+
+    std::chrono::milliseconds benchDurMs;
+};
+
+
+using BENCH_FUNC = std::function<void(const wxString&, int, BENCH_REPORT&)>;
+
+
+struct BENCHMARK
+{
+    char triggerChar;
+    BENCH_FUNC func;
+    wxString name;
+};
+
+
+/**
+ * Benchmark using a raw std::ifstream, with no LINE_READER
+ * wrapper. The stream is recreated for each cycle.
+ */
+static void bench_fstream( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    std::string line;
+
+    for( int i = 0; i < aReps; ++i)
+    {
+        std::ifstream fstr( aFile );
+
+        while( getline( fstr, line ) )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) line[0];
+        }
+
+        fstr.close();
+    }
+}
+
+
+/**
+ * Benchmark using a raw std::ifstream, with no LINE_READER
+ * wrapper. The stream is not recreated for each cycle, just reset.
+ */
+static void bench_fstream_reuse( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    std::string line;
+    std::ifstream fstr( aFile );
+
+    for( int i = 0; i < aReps; ++i)
+    {
+        while( getline( fstr, line ) )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) line[0];
+        }
+        fstr.clear() ;
+        fstr.seekg(0, std::ios::beg) ;
+    }
+
+    fstr.close();
+}
+
+
+/**
+ * Benchmark using a given LINE_READER implementation.
+ * The LINE_READER is recreated for each cycle.
+ */
+template<typename LR>
+static void bench_line_reader( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    for( int i = 0; i < aReps; ++i)
+    {
+        LR fstr( aFile );
+        while( fstr.ReadLine() )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) fstr.Line()[0];
+        }
+    }
+}
+
+
+/**
+ * Benchmark using a given LINE_READER implementation.
+ * The LINE_READER is rewound for each cycle, not recreated.
+ */
+template<typename LR>
+static void bench_line_reader_reuse( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    LR fstr( aFile );
+    for( int i = 0; i < aReps; ++i)
+    {
+
+        while( fstr.ReadLine() )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) fstr.Line()[0];
+        }
+
+        fstr.Rewind();
+    }
+}
+
+
+/**
+ * Benchmark using an INPUTSTREAM_LINE_READER with a given
+ * wxInputStream implementation.
+ * The INPUTSTREAM_LINE_READER is reset for each cycle.
+ */
+template<typename S>
+static void bench_wxis( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    S fileStream( aFile );
+
+    for( int i = 0; i < aReps; ++i)
+    {
+        INPUTSTREAM_LINE_READER istr( &fileStream, aFile );
+
+        while( istr.ReadLine() )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) istr.Line()[0];
+        }
+
+        fileStream.SeekI( 0 );
+    }
+}
+
+
+/**
+ * Benchmark using an INPUTSTREAM_LINE_READER with a given
+ * wxInputStream implementation.
+ * The INPUTSTREAM_LINE_READER is recreated for each cycle.
+ */
+template<typename S>
+static void bench_wxis_reuse( const wxString& aFile, int aReps, BENCH_REPORT& report )
+{
+    S fileStream( aFile );
+    INPUTSTREAM_LINE_READER istr( &fileStream, aFile );
+
+    for( int i = 0; i < aReps; ++i)
+    {
+        while( istr.ReadLine() )
+        {
+            report.linesRead++;
+            report.charAcc += (unsigned char) istr.Line()[0];
+        }
+
+        fileStream.SeekI( 0 );
+    }
+}
+
+/**
+ * List of available benchmarks
+ */
+static std::vector<BENCHMARK> benchmarkList =
+{
+    { 'f', bench_fstream, "std::fstream" },
+    { 'F', bench_fstream_reuse, "std::fstream, reused" },
+    { 'r', bench_line_reader<FILE_LINE_READER>, "RICHIO" },
+    { 'R', bench_line_reader_reuse<FILE_LINE_READER>, "RICHIO, reused" },
+    { 'n', bench_line_reader<IFSTREAM_LINE_READER>, "std::ifstream L_R" },
+    { 'N', bench_line_reader_reuse<IFSTREAM_LINE_READER>, "std::ifstream L_R, reused" },
+    { 'w', bench_wxis<wxFileInputStream>, "wxFileIStream" },
+    { 'W', bench_wxis<wxFileInputStream>, "wxFileIStream, reused" },
+    { 'g', bench_wxis<wxFFileInputStream>, "wxFFileIStream" },
+    { 'G', bench_wxis_reuse<wxFFileInputStream>, "wxFFileIStream, reused" },
+};
+
+
+/**
+ * Construct string of all flags used for specifying benchmarks
+ * on the command line
+ */
+static wxString getBenchFlags()
+{
+    wxString flags;
+
+    for( auto& bmark : benchmarkList )
+    {
+        flags << bmark.triggerChar;
+    }
+
+    return flags;
+}
+
+
+/**
+ * Usage description of a benchmakr spec
+ */
+static wxString getBenchDescriptions()
+{
+    wxString desc;
+
+    for( auto& bmark : benchmarkList )
+    {
+        desc << "    " << bmark.triggerChar << ": " << bmark.name << "\n";
+    }
+
+    return desc;
+}
+
+
+BENCH_REPORT executeBenchMark( const BENCHMARK& aBenchmark, int aReps,
+        const wxString& aFilename )
+{
+    BENCH_REPORT report = {};
+
+    TIME_PT start = CLOCK::now();
+    aBenchmark.func( aFilename, aReps, report );
+    TIME_PT end = CLOCK::now();
+
+    using std::chrono::milliseconds;
+    using std::chrono::duration_cast;
+
+    report.benchDurMs = duration_cast<milliseconds>( end - start );
+
+    return report;
+}
+
+
+enum RET_CODES
+{
+    BAD_ARGS = 1,
+};
+
+
+int main( int argc, char* argv[] )
+{
+    auto& os = std::cout;
+
+    if (argc < 3)
+    {
+        os << "Usage: " << argv[0] << " <FILE> <REPS> [" << getBenchFlags() << "]\n\n";
+        os << "Benchmarks:\n";
+        os << getBenchDescriptions();
+        return BAD_ARGS;
+    }
+
+    wxString inFile( argv[1] );
+
+    long reps = 0;
+    wxString( argv[2] ).ToLong( &reps );
+
+    // get the benchmark to do, or all of them if nothing given
+    wxString bench;
+    if ( argc == 4 )
+        bench = argv[3];
+
+    os << "IO Bench Mark Util" << std::endl;
+
+    os << "  Benchmark file: " << inFile << std::endl;
+    os << "  Repetitions:    " << (int) reps << std::endl;
+    os << std::endl;
+
+    for( auto& bmark : benchmarkList )
+    {
+        if( bench.size() && !bench.Contains( bmark.triggerChar ) )
+            continue;
+
+        BENCH_REPORT report = executeBenchMark( bmark, reps, inFile );
+
+        os << wxString::Format( "%-25s %u lines, acc: %u in %u ms",
+                bmark.name, report.linesRead, report.charAcc, (int) report.benchDurMs.count() )
+            << std::endl;;
+    }
+
+    return 0;
+}
diff --git a/tools/io_benchmark/stdstream_line_reader.cpp b/tools/io_benchmark/stdstream_line_reader.cpp
new file mode 100644
index 000000000..8a0954a8e
--- /dev/null
+++ b/tools/io_benchmark/stdstream_line_reader.cpp
@@ -0,0 +1,90 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+ */
+
+#include "stdstream_line_reader.h"
+
+#include <ios>
+
+STDISTREAM_LINE_READER::STDISTREAM_LINE_READER() :
+    LINE_READER( 0 ),
+    m_stream( nullptr )
+{
+    line = nullptr;
+    lineNum = 0;
+}
+
+
+STDISTREAM_LINE_READER::~STDISTREAM_LINE_READER()
+{
+    // this is only a view into a string, it cant be deleted by the base
+    line = nullptr;
+}
+
+
+char* STDISTREAM_LINE_READER::ReadLine() throw( IO_ERROR )
+{
+    getline( *m_stream, m_buffer );
+
+    m_buffer.append( 1, '\n' );
+
+    length = m_buffer.size();
+    line = (char*) m_buffer.data(); //ew why no const??
+
+    // lineNum is incremented even if there was no line read, because this
+    // leads to better error reporting when we hit an end of file.
+    ++lineNum;
+
+    return m_stream->eof() ? nullptr : line;
+}
+
+
+void STDISTREAM_LINE_READER::setStream( std::istream& aStream )
+{
+    // Could be done with a virtual getStream function, but the
+    // virtual function call is a noticable (but minor) penalty within
+    // ReadLine() in tight loops
+    m_stream = &aStream;
+}
+
+
+IFSTREAM_LINE_READER::IFSTREAM_LINE_READER( const wxString& aFileName ) throw( IO_ERROR ) :
+        m_fStream( std::ifstream( aFileName ) )
+{
+    if( !m_fStream.is_open() )
+    {
+        wxString msg = wxString::Format(
+            _( "Unable to open filename '%s' for reading" ), aFileName.GetData() );
+        THROW_IO_ERROR( msg );
+    }
+
+    setStream( m_fStream );
+
+    source = aFileName;
+}
+
+
+void IFSTREAM_LINE_READER::Rewind()
+{
+    m_fStream.clear() ;
+    m_fStream.seekg(0, std::ios::beg );
+}
diff --git a/tools/io_benchmark/stdstream_line_reader.h b/tools/io_benchmark/stdstream_line_reader.h
new file mode 100644
index 000000000..1fa18a262
--- /dev/null
+++ b/tools/io_benchmark/stdstream_line_reader.h
@@ -0,0 +1,70 @@
+/*
+ * This program source code file is part of KiCad, a free EDA CAD application.
+ *
+ * Copyright (C) 2017 KiCad Developers, see AUTHORS.txt for contributors.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, you may find one here:
+ * http://www.gnu.org/licenses/old-licenses/gpl-2.0.html
+ * or you may search the http://www.gnu.org website for the version 2 license,
+ * or you may write to the Free Software Foundation, Inc.,
+ * 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
+ */
+
+#ifndef FSTREAM_LINE_READER_H
+#define FSTREAM_LINE_READER_H
+
+#include <richio.h>
+
+#include <istream>
+#include <fstream>
+
+/**
+ * LINE_READER that wraps a given std::istream instance.
+ */
+class STDISTREAM_LINE_READER : public LINE_READER
+{
+public:
+
+    STDISTREAM_LINE_READER();
+
+    ~STDISTREAM_LINE_READER();
+
+    char* ReadLine() throw( IO_ERROR ) override;
+
+protected:
+
+    void setStream( std::istream&  aStream );
+
+private:
+    std::string m_buffer;
+    std::istream* m_stream;
+};
+
+
+/**
+ * LINE_READER interface backed by std::ifstream
+ */
+class IFSTREAM_LINE_READER : public STDISTREAM_LINE_READER
+{
+public:
+
+    IFSTREAM_LINE_READER( const wxString& aFileName ) throw( IO_ERROR );
+
+    void Rewind();
+
+private:
+    std::ifstream m_fStream;
+};
+
+#endif // FSTREAM_LINE_READER_H
-- 
2.12.0