kicad-developers team mailing list archive
Mailing list archive
Re: Re: New library file format
Dick Hollenbeck <dick@...>
Sun, 08 Nov 2009 23:54:20 -0600
Thunderbird 18.104.22.168 (X11/20090817)
Wayne Stambaugh wrote:
--- In kicad-devel@xxxxxxxxxxxxxxx, Wayne Stambaugh <stambaughw@...> wrote:
Refer to my previous thread,
and a wiki page about new library system spec,
I'd like to ask a few questions.
1) Is there developers working on the new library file format specification? I may be able to help.
The only proposed change to the library file format that I am aware of
is the one I proposed which is to merge the document file *.dcm with the
library file *.lib. I held off to see what additional information may
be required by the library updates that some of the other developers are
2) Is there the document for the current file format? I tried to find it with no luck.
Yes. You can check it out of SVN. The Kicad wiki has all the
information you need. It is in an OpenOffice.org format.
3) Would it be possible to use an existing database engine (eg. sqlite) to store library/module information?
Yes it would be possible. I think you will find most developers on this
list (this one include) prefer text file formats so that they can be
created, manipulated, and viewed with their favorite editor and command
line tools. I for one do not want the component definitions themselves
to be saved in a database. Tagging and file path information could be
put in a database for searching purposes but not the actual component
definitions themselves. If an acceptable search solution using tags in
plain text files cannot be developed, then a database may be an option.
I don't believe we have actually gotten to that point yet.
Thank you Wayne, the documents are lied in kicad-doc directory. I also prefer text. Some folks here think that it would be good if the file is more readable by human. I would like to propose a new file format based on an existing tool such as JSON (wxJSON, http://wxcode.sourceforge.net/docs/wxjson/). I think this will make the file more readable, flexible and extensible.
I agree that the current library and schematic file formats are not very
human readable. I believe they are legacy file formats that have been
with the project for some time. I am not familiar with JSON so I cannot
comment on how effective it would be. One thing I think it should
support is wxInputStream and wxOutputStream ( or C++ library istream and
ostream ). This way you only have to write one parser and one output
format to support streams from any of the derived input and output
stream classes. If you us wxInputStream to write you parser, you get
socket, file, zip, and tar input streams for very little extra effort.
With all of the new library discussions, I could see where any one of
the derived input and/or output streams would be useful.
"Existing tools" can be found in many places. I vote for the parser we
already have in the specctra import, and its LineReader input
abstraction which can easily be adapted to grab lines from any source,
I find the specctra dsn style more agreeable than JSON.
The specctra parsing technique is easily adapted to any DSN like
grammar, and the lexer only needs a new table of token keywords and
matching enums to handle any task we could throw at it. And it reports
errors down the line number and character offset, and those errors use
text that we control, and the error reporting is kept out of the library
by use of exceptions.
It is *still* what I would use for any new work involving structural
text data files.
But what about the user experience?
Having a well defined API for part retrieval,
serialization/deserialization (in a java like sense), browsing,
searching, and adoption, means that Kicad could have any one of a number
of competing libraries.
If you separate the notion of library maintenance from library usage, in
the same way that you can be a visitor to a physical library yet don't
have to work there like the actual librarian does to keep the place
orderly, then you as a library user can be positioned to pick from a
number of competing libraries. My suggestion is to define the *usage
API*, not the management API. The librarian may be in python on the
other side of a DBUS bridge, or on the other side of the world.
By having a well defined library usage API which is implemented by any
number of *competing* C++ classes derived from a common C++ *usage API*
definition class using virtual functions, those derived classes can live
in any number of DSOs and give the user choices. The librarians can do
things the way they want, but the usage API is a contract that they all
have to honor.
This is why I don't think the file format is even remotely important to
the usage API. Because such an API should not concern itself with file
formats, but in memory structures, tags, pins, parts, serialized byte
streams, and the like. And more than one usage API implementation can
be active in the Kicad program concurrently via an array of plugins.
The competing implementations would be called plugins, and would be
under the control of a plugin manager which can seamlessly blend
together these separate sources into one cohesive user experience.
This is a divide and conquer strategy. We divide *part maintenance*
from *part usage* tasks, with the hope and vision that we can spin off
competing implementations, and competing library communities. One
community could be website based, one could be python local, a 3rd could
be C++ local. In each case those librarian communities would need to
implement as a minimum a C++ plugin which honors the "usage API", from
there they could go over the web, through DBUS to python, or to a local
disk directory directly.
Some of these libraries could even be commercial, and require you to pay
a subscription fee for the work required to maintain them, or tap into
existing commercial libraries. I think the concept of creating a
business domain under Kicad in which competing implementations can
co-exist would be healthy for Kicad. And it is not unreasonable for all
existing libraries to be spun off into one or more separate projects,
although the licensing of those spin-offs would be important details
that would need to respect existing license terms with continuity. But
lets remember that the licenses used by Kicad currently involve code
*ownership*, not fees for service oriented labor or work.
Also remember that certain geda parts can already be imported into
Kicad, but my vision formalizes that basic first step into a much
broader strategy. The problem gets simpler if you start with the usage
API, and this is why I suggested we focus on the user experience. The
usage API could even go so far as to describe a modification that you
need to an existing part, such as a pin count change, etc.
Think about it.