kicad-developers team mailing list archive

["Patrick" <pmaupin@...>]

--- In kicad-devel@xxxxxxxxxxxxxxx, Dick Hollenbeck <dick@...> wrote:
>
> Patrick wrote:
> >>> I agree that the current library and schematic file formats are not very
> >>> human readable. I believe they are legacy file formats that have been
> >>> with the project for some time. I am not familiar with JSON so I cannot
> >>> comment on how effective it would be. One thing I think it should
> >>> support is wxInputStream and wxOutputStream ( or C++ library istream and
> >>> ostream ). This way you only have to write one parser and one output
> >>> format to support streams from any of the derived input and output
> >>> stream classes. If you us wxInputStream to write you parser, you get
> >>> socket, file, zip, and tar input streams for very little extra effort.
> >>> With all of the new library discussions, I could see where any one of
> >>> the derived input and/or output streams would be useful.
> >>>
> >>> Wayne
> >>>
> >>>   
> >>>       
> >>
> >> "Existing tools" can be found in many places. I vote for the parser we 
> >> already have in the specctra import, and its LineReader input 
> >> abstraction which can easily be adapted to grab lines from any source, 
> >> including wxInputStream.
> >>
> >> I find the specctra dsn style more agreeable than JSON.
> >>
> >> The specctra parsing technique is easily adapted to any DSN like 
> >> grammar, and the lexer only needs a new table of token keywords and 
> >> matching enums to handle any task we could throw at it. And it reports 
> >> errors down the line number and character offset, and those errors use 
> >> text that we control, and the error reporting is kept out of the library 
> >> by use of exceptions.
> >>
> >> It is *still* what I would use for any new work involving structural 
> >> text data files.
> >>
> >> But what about the user experience?
> >>
> >> Having a well defined API for part retrieval, 
> >> serialization/deserialization (in a java like sense), browsing, 
> >> searching, and adoption, means that Kicad could have any one of a number 
> >> of competing libraries. 
> >>
> >> If you separate the notion of library maintenance from library usage, in 
> >> the same way that you can be a visitor to a physical library yet don't 
> >> have to work there like the actual librarian does to keep the place 
> >> orderly, then you as a library user can be positioned to pick from a 
> >> number of competing libraries. My suggestion is to define the *usage 
> >> API*, not the management API. The librarian may be in python on the 
> >> other side of a DBUS bridge, or on the other side of the world.
> >>
> >> By having a well defined library usage API which is implemented by any 
> >> number of *competing* C++ classes derived from a common C++ *usage API* 
> >> definition class using virtual functions, those derived classes can live 
> >> in any number of DSOs and give the user choices. The librarians can do 
> >> things the way they want, but the usage API is a contract that they all 
> >> have to honor.
> >>
> >> This is why I don't think the file format is even remotely important to 
> >> the usage API. Because such an API should not concern itself with file 
> >> formats, but in memory structures, tags, pins, parts, serialized byte 
> >> streams, and the like. And more than one usage API implementation can 
> >> be active in the Kicad program concurrently via an array of plugins.  
> >> The competing implementations would be called plugins, and would be 
> >> under the control of a plugin manager which can seamlessly blend 
> >> together these separate sources into one cohesive user experience.
> >>
> >>
> >> This is a divide and conquer strategy. We divide *part maintenance* 
> >> from *part usage* tasks, with the hope and vision that we can spin off 
> >> competing implementations, and competing library communities. One 
> >> community could be website based, one could be python local, a 3rd could 
> >> be C++ local. In each case those librarian communities would need to 
> >> implement as a minimum a C++ plugin which honors the "usage API", from 
> >> there they could go over the web, through DBUS to python, or to a local 
> >> disk directory directly.
> >>
> >> Some of these libraries could even be commercial, and require you to pay 
> >> a subscription fee for the work required to maintain them, or tap into 
> >> existing commercial libraries. I think the concept of creating a 
> >> business domain under Kicad in which competing implementations can 
> >> co-exist would be healthy for Kicad. And it is not unreasonable for all 
> >> existing libraries to be spun off into one or more separate projects, 
> >> although the licensing of those spin-offs would be important details 
> >> that would need to respect existing license terms with continuity. But 
> >> lets remember that the licenses used by Kicad currently involve code 
> >> *ownership*, not fees for service oriented labor or work.
> >>
> >> Also remember that certain geda parts can already be imported into 
> >> Kicad, but my vision formalizes that basic first step into a much 
> >> broader strategy. The problem gets simpler if you start with the usage 
> >> API, and this is why I suggested we focus on the user experience. The 
> >> usage API could even go so far as to describe a modification that you 
> >> need to an existing part, such as a pin count change, etc.
> >>
> >> Think about it.
> >>     
> >
> > I like the vision of dividing part maintenance away from part usage.
> >
> > But part of the vision needs to be that, if I tar up a project and thenuntar it two years later to make a board revision, I can still do that revision even if one of my library sources is off-line or even out of business. Xilinx's licensing model is pretty good. Nothing you buy from them everstops working; you just promise not to use it for anything new after your one-year subscription is up. (And open source is fundamentally incompatible with DRM in any case, because a determined programmer can "free" his dataquite easily.)
> >
> > So, if I can tar/untar a project and have it work, that means that KiCad needs (as it has now) some sort of "cache" library format (and, preferably, the ability to read cache files from older versions).
> >   
> Correct. I had referred to some serialization in a Java like sense.  
> This was my attempt to simply say the same thing you are saying here.  
> "Java like serialization" was not the best description of this need, 
> because as you state, we need a common serialization format for the 
> cache, not a class specific one. But I still think we can call it 
> serialization - deserialization, and I still think the DSN format and 
> code structure that I put into the specctra support would best deal with 
> this.

That makes good sense. And I have no issue at all with a readable textual format.

> 
> > And, while it's not really necessary to expose that format to the rest of the world, if it's reasonably readable text and well-documented, and thebase KiCad needs to have a reader for the format anyway, you can make a case that the cache library format (for a given version of KiCad) is the "canonical" form of the library, and you could advance various arguments for keeping it relatively stable, or making a few minor enhancements in the format, etc.
> >
> > But you are absolutely right that making the file format itself the focus of the effort to clean up the library management user experience is putting the cart before the horse, although some decisions made in determining the library management architecture will undoubtedly impact the cached fileformat by, for example, requiring the storage of additional metadata.
> >
> > I think the plug-in architecture you describe could go a long way toward solving the library management problem, but for it to be used, the very first released version has to have really good support for moving parts and even entire libraries around from plugin to plugin.  
> 
> No, I was not thinking of this "library modify" capability as part of 
> the "usage API". I think the "modify" capability is in a library 
> "management API". And it should be possible to have plugins which only 
> implement a "usage API" and not a full "management API". There needs to 
> be at least one plugin which implements the management API, and that 
> would be the plugin which does approximately what the current library 
> manipulation code does (by manipulation, I mean modify, not simply 
> read). But then since there may only be one library manager that uses 
> the management API, we don't need to rush out the formalization of the 
> management API in to the same level of diligence as the usage API.
> 
> Some kicad library plugins could provide a read only library service.

I agree that the ability to implement a read-only service would be useful and even necessary. (The ability to download a library from the web in the way I mentioned would typically be through a read-only page.) I guess what I was trying to say is that something which uses the KiCad plugin manager ought to be able to read a part or an entire library from a read-only plugin, allow the user to modify a component, and write out to a read/write library. I think we're on the same page here.

> 
> > (And I have been involved in lots of grandiose overarching designs thatonly ever got one or two plugins written for them.)
> >   
> 
> Well this is not your strongest point. To cast doubt on something that 
> does not yet exist in the middle of a brain storming session is not 
> really productive, especially when the folks in the room have more than 
> enough capability to pull off what they are proposing.

I meant no offense, and I am certainly not trying to cast aspersions on your capabilities. I have no doubt whatsoever that you are fully capable of building whatever you propose. My point, though badly expressed, was simply that it is very easy (and I have seen it, indeed participated in it, firsthand) to over-engineer something to support all kinds of potential use-casesand re-use by others, and then when it is deployed and the dust has settled, even though it is working as advertised and you did a fine job and everybody's happy, the *actual* *use* of your wonderful invention is limited to such a small subset of what you designed that you could have saved a lot oftime and effort by building something much simpler in the first place. Now, since I can't read your mind, and don't know exactly how you are planning to hook the plugins up, I cannot presume to know whether you are anywherenear falling into this particular trap or not. It's just a cautionary tale.

> > So I absolutely agree with you that user experience is paramount, but in so doing I put the plugin manager in the same category as the file format-- a nice piece of technology that might solve part of the problem.
> >
> > Personally, if I were doing this, though (but of course I'm not, and I know that doesn't give me a vote :) being the lazy programmer that I am, I would eschew the plugin support for a much simpler architecture, and try toleverage existing web infrastructure as much as possible. For example, itwould be relatively easy to add the ability to open a socket to read a webpage, and the ability to parse HTML and look for special tags that indicated a valid library was inside.
> >   
> 
> Your HTML parser could be done in a "usage API" plugin, so your 
> "implementation" of a read only library part retrieval mechanism can 
> have its place, one that the plugin architecture provides for you. So 
> your idea is not mutually exclusive with mine.

Absolutely agreed. It should be a piece of cake with the available lower-level data serializer/deserializer you describe.

> > Now, you've got google finding your parts for you, you've got a web page with pinouts and text and 3D models, or however fancy the designer wants to get in communicating his library to humans, and KiCad can load it from the same URL the user uses to look at it.
> >
> > Your documentation needs shift. Sure, you're documenting a file format(which you really should do anyway for the cache file), but instead of documenting an API, you are documenting how you containerize your file inside an HTML document with a few tags.
> >
> > A guy who uses the library editor to make one part can now share it with the world without having to write or understand a KiCad plugin -- he justneeds basic HTML skills. 
> 
> The library editor becomes a library specific tool under my vision. In 
> some libraries parts could be stored in a database and autogenerated 
> with fortran or front panel switches, or chipmonks (pun intended). It 
> does not matter, as long as the usage API is there to pull parts out 
> into Kicad, and then from those in RAM structures we can cache the parts 
> on disk. Writing a plugin does not have to be hard, especially if 
> there were samples. As always, there are people available for hire to 
> do these things.

I can see your point, and I agree that the plugin architecture can remove the debate about where/how libraries should be stored from KiCad development, and let non-core developers play with their own visions more easily. So,if the plugin architecture you are envisioning is a small enough effort, you should build it and let them come. It's only if it starts to be big enough to start costing serious development time that you should blow it off and decide to hard code support for "files and XXX", where "XXX" is web, or database, or whatever.

I have to admit that I did not understand, at first, that you were proposing a standard serialization/deserialization mechanism that *could* be common(at least reused) by each plugin. So I was worried that a plugin would bemore work than it appears you meant for it to be.

Since you have expressed that there should be a canonical text format for library data, the very first plugin I would like to see is one which can invoke an executable in a separate process and communicate with it via stdin/stdout. Then I can build my real plugin underneath the plugin manager in Python :)

> 
> Remember Kicad is on the cusp of serving the needs of the corporate 
> world. The corporate world is used to paying for functional software.  

Agreed, but the corporate world is getting used to a lot of free software as well, and getting used to having to do a bit of maintenance on it as well. This is a dynamic that will be interesting to watch over the next few years. I think organizations are happy to pay for functionality, but will increasingly balk at paying royalties for functionality they already paid foronce.

> Functional software is *never* free to everyone, somebody always had to 
> pay for it somewhere along the way.
> 
> Dick
> 
> 
> 
> 
> > And when he *does* share it on the web, and some random Joe Schmoe googles for a particular weird connector part number and stumbles across the page with the fancy 3D models, that will give him pause, and increase KiCad mindshare, in a way that no number of postings of "Download this and try the new plugin manager!" could possibly touch.
> >
> > Just my two cents -- do with it what you will.
> >
> > Regards,
> > Pat