← Back to team overview

kicad-developers team mailing list archive

Re: [RFC] Symbol library file format


I like the idea of using something as Protobuf and I agree fully with the benefits, especially since one can add/remove fields with minimal impact.

Basically the S-expression system used now is looking very much like a reinvented XML to me anyway, and storing protobuf-defined stuff as XML or similar seems actually nice. There is one catch, and that is that we have to support opening a newer file, in an old software, and then store it again, without losing data that the software is not aware of. Or implement a way of not being able to store values in older software, when they open something newer.

There is also a middle way here, and that is to actually implement a Protobuf to S-Expression decoder/encoder, with the real benefit of actually defining fields in a modern well-known way, where the specification and implementetation does not have to manually be synced in code, comments, and a google doc. I have yet to see anything actually stay synchronized in such a manner over time, and many bugs manifest themself in these synchronization attempts. Anyway to avoid having to change the file-format another time, or add extra files to the side, I think that using an IDF is great next-step, mostly since the tooling, libraries and workflows for these are better defined.

But to be honest, I have a hard time understanding why we have to stick to the KiCad S-Expression, when there are quite readable text-formats that are widely supported already.

I know the requirement for the file format is readability, but I have yet to find and editor that actually understands the KiCad S-Expression (I have not searched extensively), but JSON,XML,YAML are usually read just fine, with syntax highlighting out of box. And an IDF would make these discussions quite reduntant, since changing file formats would be a minimal change in code, and not as now, where it is actually quite time-consuming.

- Kristoffer

On 2019-01-02 01:37, Andrew Lutsenko wrote:
Hi Wayne,

I would like to take this opportunity to do an elevator pitch for idea of using one of IDL languages widely accepted in the industry like Apache Thrift or Google Protobufs to define formats in KiCad.
There are few large benefits in favor of using such languages:

1. They are self documenting. No more keeping a google doc in sync with sources. 2. They are easily extensible. Just add a field, old parsers will ignore it, new ones will pick it up. Need to deprecate a field? Just add it's ID to reserved list to never reuse it again. 3. They have code generators for pretty much all commonly used languages. That means anyone can pick KiCad file and just parse it in Java/Go/Haskell or whatever language they fancy without porting over s-expressions library or meticulously studying the file format doc. This opens lot's of possibilities for third party tools to be added to KiCad ecosystem. Writing a web viewer for schematic/pcb would be a piece of cake for example.

Other probably less impactful benefits:
4. Easy to serialize/encode in multiple formats. Need to send data over network in compact form? No problem, just serialize using compact binary protocol. Need to store in text file? just use text encoder. 5. Code generators will reduce amount of boilerplate in KiCad source. Only actual application logic needs to be added on top of generated data objects.

I have personally worked extensively with both Thrift and Protobufs, I think for KICad use case proto is better fit. Thrift has a lot more library support for client/server RPC logic and defining RPCs is core part of the language but we don't need any of that (at least for now). Proto has all of that as extensions but it's core is just definition of data types and it has better support for plain text format.
Here are docs for both:

Let me know if any of that sounds interesting and if you have any questions. I think this is worth investing time into and I'm willing to help if needed.


On Tue, Jan 1, 2019 at 11:59 AM Wayne Stambaugh <stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>> wrote:

    I have updated and published the symbol file format[1] for comment.
    Hopefully there isn't too much to change.  The only thing to really
    finalize is the internal units.  The initial concept was unitless but
    the more I think about it and discuss with other developers, it makes
    more sense to use units for the following reasons:

    1. It's easier to visualize in your head how the symbols on a
    given page
    size will layout.

    2. Converting from other file formats (Eagle, Altium, etc) will be
    easier since most if not all of them have a defined unit.

    I'm thinking 10u (or possibly 100u) will make a good internal units
    value.  Once we nail down the units, I will update the file format
    document accordingly.

    Please keep in mind that this is the symbol library file format
    so things like constraints belong in the schematic file format.  I
    be posting the schematic file format as soon as I finish updating it.




    Mailing list: https://launchpad.net/~kicad-developers
    Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
    Unsubscribe : https://launchpad.net/~kicad-developers
    More help   : https://help.launchpad.net/ListHelp

Mailing list: https://launchpad.net/~kicad-developers
Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Follow ups