kicad-developers team mailing list archive

[John Beard <john.j.beard@xxxxxxxxx>]

On Mon, Oct 29, 2018 at 3:33 PM Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
>
> The parser (at least the pcb parser I wrote) can be used a parse only
> tool.  I cannot speak about the other file format parsers but hopefully
> the authors provided low level parse only functionality for testing
> purposes.  If not, I'm fine with adding it.

Yes, that's what the fuzzing tool hooks into right now, but it only
works for kicad formats.

> The github and pcb plugins use the same parser and formatter code but
> provide a unique interface to the under lying handling of the files.

Sure, that was more of an example of a plugin that doesn't actually
read a local file using the "filename" (and thus could not use a
stream).

> This depends on what you are trying to test.  If it's just the parser
> and output formatters, then you really don't need the plugin interface.
>  If your testing the plugins themselves, then you need to add the
> individual test code for each plugin.

That makes sense.

Mostly I'd like to make it possible to get each input/output under
test so that we can make sure they're able to read/write files
correctly. For most testing, this is as simple as feeding them files
from a corpus of known good/bad files and checking the outputs are
somehow "right" (this can actually be quite hard, as if you don't
trust the code-under-test's logic, how do you check you wrote it
correctly?! The same applies to plotting outputs)

I would imagine that, as we have no formal tests of any format
currently, that most tests would be for:

* simple read/write of a few known good/bad files and make sure the
good ones work and the bad ones are caught
* files that are reported to provoke bugs, and the tests ensure that
bug is fixed for a minimal example
* areas of refactoring to ensure the outputs remain the same

A campaign for a full-coverage of each (nominally working) parser is
probably not a wise use of time, despite the warm fuzzy feelings they
might cause!

Cheers,

John