kicad-developers team mailing list archive

[Wayne Stambaugh <stambaughw@xxxxxxxxx>]

John,

On 10/29/2018 11:50 AM, John Beard wrote:
> On Mon, Oct 29, 2018 at 3:33 PM Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
>>
>> The parser (at least the pcb parser I wrote) can be used a parse only
>> tool.  I cannot speak about the other file format parsers but hopefully
>> the authors provided low level parse only functionality for testing
>> purposes.  If not, I'm fine with adding it.
> 
> Yes, that's what the fuzzing tool hooks into right now, but it only
> works for kicad formats.
> 
>> The github and pcb plugins use the same parser and formatter code but
>> provide a unique interface to the under lying handling of the files.
> 
> Sure, that was more of an example of a plugin that doesn't actually
> read a local file using the "filename" (and thus could not use a
> stream).
> 
>> This depends on what you are trying to test.  If it's just the parser
>> and output formatters, then you really don't need the plugin interface.
>>  If your testing the plugins themselves, then you need to add the
>> individual test code for each plugin.
> 
> That makes sense.
> 
> Mostly I'd like to make it possible to get each input/output under
> test so that we can make sure they're able to read/write files
> correctly. For most testing, this is as simple as feeding them files
> from a corpus of known good/bad files and checking the outputs are
> somehow "right" (this can actually be quite hard, as if you don't
> trust the code-under-test's logic, how do you check you wrote it
> correctly?! The same applies to plotting outputs)
> 
> I would imagine that, as we have no formal tests of any format
> currently, that most tests would be for:
> 
> * simple read/write of a few known good/bad files and make sure the
> good ones work and the bad ones are caught
> * files that are reported to provoke bugs, and the tests ensure that
> bug is fixed for a minimal example
> * areas of refactoring to ensure the outputs remain the same
> 
> A campaign for a full-coverage of each (nominally working) parser is
> probably not a wise use of time, despite the warm fuzzy feelings they
> might cause!

Any testing we can add is an improvement.  This has always been a weak
spot for the project.  I appreciate your efforts on improving our
testing capabilities.

Cheers,

Wayne

> 
> Cheers,
> 
> John
>