kicad-developers team mailing list archive

[John Beard <john.j.beard@xxxxxxxxx>]

Sorry - forwarding to the list. That's what happens when I do emails too late!

---------- Forwarded message ---------
From: Andrew Lutsenko <anlutsenko@xxxxxxxxx>
Date: Fri, Jan 4, 2019 at 12:50 AM
Subject: Re: [Kicad-developers] [RFC] Symbol library file format
To: John Beard <john.j.beard@xxxxxxxxx>


Sorry, link to github:
https://github.com/protocolbuffers/protobuf/blob/master/src/README.md

Regards,
Andrew

On Thu, Jan 3, 2019 at 4:50 PM Andrew Lutsenko <anlutsenko@xxxxxxxxx> wrote:
>
> John,
>
> > Sorry for the text wall: I'm not even disagreeing!
> On the contrary, this is healthy technical discussion and is most welcome.
>
> I noticed that you didn't reply all so your email didn't go to kicad mailing list, not sure if this was intentional, feel free to add it back to recipients list.
>
> > I don't think that discussion is (currently) bogged down in format
> > stuff - so far it's mostly model related
> There are a lot of resolved comments that you can view if you press the small speech bubble next to share button. Good chunk of them are format related.
>
> > This is exactly the direction we should be looking at for the API.
> >... But the API is a different
> > issue. Using the same technique for both would be a tidy solution, and
> > protobuf is a highly language-agnostic solution to boot.
> So we are in agreement. I'm pitching this for eeschema mostly because a file incompatible change is happening either way and this is a good opportunity to switch formats too.
> Eeschema currently doesn't have an API but adding it on top of well defined protobuf based data model will be a lot easier and it should be stable with virtually no effort.
> If this approach is accepted we can direct our attention to pcbnew later.
>
> > A "proper" s-exp parser deals in only atoms, and pairs of atoms (and
> > lists, which are just nested pairs in a pretty dress).
> Except the format currently proposed has lots of tuples that go beyond just pairs. It would be fine if it was just pairs of atoms and lists of pairs, that would be essentially json with round braces.
> But as it is now we are trying to do same thing as in pcbnew: lists of arbitrary length tuples with fixed and in the long term fragile ordering of fields that need supporting documentation to make heads and tails of.
>
> > While that is evolving, start new discussions about the best format to
> > represent that information. For example, a set of .proto files to
> > describe the model.
> Yes I should probably do that. I can not guarantee that I will commit enough time to constantly keep proto model in sync with ongoing discussion since it's a fast moving target right now. But I'll do what I can.
>
> > I for one have a question about Protobufs, but I don't want to start
> > talking specifics in this thread.
> I'm happy to answer your questions if I can help. Please do also take a look at official, very high quality, documentation:
> https://developers.google.com/protocol-buffers/docs/cpptutorial
> Github repo has extensive instructions on installation
>
>
>
>
> On Thu, Jan 3, 2019 at 4:12 PM John Beard <john.j.beard@xxxxxxxxx> wrote:
>>
>> On Thu, Jan 3, 2019 at 7:51 PM Andrew Lutsenko <anlutsenko@xxxxxxxxx> wrote:
>>
>> > If you read through Wayne's google doc comments you will see how tightly coupled is the discussion of the data model to file format
>>
>> I don't think that discussion is (currently) bogged down in format
>> stuff - so far it's mostly model related - "Should we have a width
>> field for text?", "Can this one be repeated?", "How do you represent
>> an arc?", etc. It is incontrovertibly true that protobuf is a tighter
>> way to describe these properties, and, should the model change,
>> provides neater migrations pathways. But the discussion right now
>> doesn't seem to be overly format-specific to me.
>>
>> > Imagine if pcbnew plugin API was proto based.
>>
>> This is exactly the direction we should be looking at for the API. All
>> (or nearly all, some of the more "core" things might be impractical)
>> tools should use a single interface to the data model (so a built in
>> tool is basically an internal plugin). But the API is a different
>> issue. Using the same technique for both would be a tidy solution, and
>> protobuf is a highly language-agnostic solution to boot.
>>
>> > Or you could just use a library that takes care of parsing for you, does it in all major languages for free, represents formal definition of your data model and will not break when you add a field.
>>
>> A "proper" s-exp parser deals in only atoms, and pairs of atoms (and
>> lists, which are just nested pairs in a pretty dress). If you need to
>> touch your parser because you added a new atom to your format,
>> something is wrong. In this case, the phrase "s-exp parser" is
>> directly analogous to "XML library" or "libprotobuf". Note: this is
>> *not* how the PCB_IO parser currently does it - that binds format and
>> model into a single hybrid thing.
>>
>> Now, whether the protobuf library provides useful tools *on top of*
>> the IO layer, that is a very valid point (certainly it is designed
>> with a view towards evolving formats and formal specification), and
>> universal implementation and active user base is very attractive.
>>
>> > > I think useful comments to the proposed format should see beyond the actual low level representation of the data and talk about the overall model being used to store it.
>> > I agree completely. Protobufs help with decoupling that.
>>
>> Let's decouple it, then! Can I suggest that (if there *is* management
>> support for discussion of non-s-exp formats, of course) that the
>> current discussion focuses on "a list of features we wish to see in
>> the symbol format"? That is: at this time, focus on the data to go in
>> the model. That's where the critical issues will be. Our current
>> document is a collaborative-editable commented s-exp file and it's a
>> functional substrate for discussion for now.
>>
>> While that is evolving, start new discussions about the best format to
>> represent that information. For example, a set of .proto files to
>> describe the model. Then people can see what that proposal would
>> entail. These can start now and the formats can evolve with the model.
>> How the evolving format keeps up with the evolving model with be the
>> perfect acid test to see how changing models are handled!
>>
>> I for one have a question about Protobufs, but I don't want to start
>> talking specifics in this thread.
>>
>> Sorry for the text wall: I'm not even disagreeing!
>>
>> Cheers,
>>
>> John