kicad-developers team mailing list archive

[Dick Hollenbeck <dick@xxxxxxxxxxx>]

On 10/06/2010 03:27 PM, Wayne Stambaugh wrote:
> On 10/6/2010 3:17 PM, Dick Hollenbeck wrote:
>   
>> On 10/05/2010 07:04 PM, Wayne Stambaugh wrote:
>>     
>>> On 10/5/2010 10:33 AM, Dick Hollenbeck wrote:
>>>   
>>>       
>>>> I get paid to brainstorm, so brace yourself.
>>>>
>>>> Continuing with library brainstorming, the following is put on the table
>>>> for consideration and eventual evaluation. Brace yourself, this is out
>>>> of the box.  Open up your mind before proceeding.
>>>>
>>>> Remember the HTML editors which let you view HTML source and the
>>>> resultant GUI presentation at the same time?  One panel has the html
>>>> source, another shows you its effects.  The following concepts use that
>>>> UI paradigm.  Use the source Luke, use the source.
>>>>
>>>>
>>>> A library is not a library file:
>>>> -------------------------------
>>>>
>>>> The concept of a *library file* is gone.  There are none.  The concept
>>>> of a library can still exist but it is a RAM resident concept.  Each
>>>> component that we traditionally think of residing in a *library file* is
>>>> actually in its own file on disk in a directory (or repo or schematic). 
>>>> A library is only built in RAM during the loading of a collection of
>>>> components.
>>>>
>>>>
>>>> Loading components:
>>>> ------------------
>>>>
>>>> The act of loading components utilizes the notion of inheritance and
>>>> classical file inclusion like the C++ #include.  Let's go back an
>>>> understand inheritance thinking about Java. In a Java virtual machine
>>>> each class definition container has a method table.  That table is
>>>> initially populated with methods from the base class.  As the base class
>>>> is extended some of those methods are overloaded (replaced) with ones
>>>> from the derived class in the internal method table.  Additional fields
>>>> can be introduced in derived classes also, into what is a field table
>>>> for that class.  Now lets move that conceptual model to a component. 
>>>> Lets assume we have a pin table, and a graphics table.  We give a
>>>> component the ability to inherit from: 1) a base code fragment,  or 2)
>>>> symbol or 3) component.  In this case you can do pin additions, or pin
>>>> modifications, pin swapping, in the same way you can overload a Java
>>>> class's method table.  Pins can be overloaded or replaced, and graphical
>>>> primitives can be inherited and extended.
>>>>     
>>>>         
>>> Conceptually I really like this idea from a coding stand point.  If you
>>> are careful you could implement this so that the code maps one to one to
>>> your new conceptual model.  This would make it easier to implement new
>>> features will breaking the existing ones.  The only issue I see is how
>>> to convert existing libraries into the new format or do we abandon
>>> backwards capability and push on towards a superior implementation.
>>>   
>>>       
>> I think changing the COMPONENT C++ object is wise.  We have to have a
>> few s-expression keywords that we would map onto a 1 for 1 C++ object:
>>
>> 1) (component ...)
>>
>> for the pin table:
>>
>> 2) (pin ...)   # w/coordinates relative to the component's (x,y) origin.
>>
>>
>> for the graphics table:
>>
>> 3) (line ...)
>> 4) (arc ...)
>> 5) (circle ...)
>> 6) (rect ...)
>>
>> : yes more!
>>
>> into a separate property table goes each property:
>>
>> 7) field or property
>>
>>
>>
>> The rest of the keywords do not need to map onto "objects" but could map
>> onto "actions", and one that I can think of is "pin_del":
>>
>> (pin_del 12)
>>
>> Which would delete pin 12 should it already have been inherited.
>>
>> So maybe you have "field_del"  or "property_del" also.  Trying to delete
>> graphics is more cumbersome.
>>
>> We can and should write a conversion program for existing libraries to
>> new stuff.
>>
>>
>> class COMPONENT
>> {
>>     pins              // pin table
>>
>>     properties        // aka fields collection
>>
>>     graphics          // graphics primitives
>> }
>>
>> Obviously keyword "pin" comes with a type which would imply the fixed
>> graphics for it.
>>
>> more below:
>>
>>     
>>>   
>>>       
>>>> The magic happens during the loading phase of the S-expressions, so that
>>>> the "built up" component is fully assembled in RAM by using pieces from
>>>> various places "out there", while *loading*.
>>>>
>>>> If desired, the various pieces can be ear-marked with their origins in
>>>> full recognition of the existence of inheritance.  One might do this
>>>> only to facilitate graphical editing of a derived component, although if
>>>> the grammar is sweet enough, textual editing may be enough for all
>>>> editing of derived components.
>>>>     
>>>>         
>>> I have always preferred a well designed textual editing interface for
>>> creating drawings.  In many ways I find it easier than performing these
>>> operations graphically.  I however probably do not represent the larger
>>> group of users who prefer a GUI editor.  This may be one of those ideas
>>> to have our users group liaison check in with the users group to see how
>>> they feel about it.
>>>
>>>   
>>>       
>>>> Re-loading is simply done by clicking a "parse" or "show" button.  The
>>>> the parser reloads the source stream and does file inclusions as called
>>>> for.  A portion of that source stream comes from the textual UI panel. 
>>>> Inherited portions come from a library (which is now a RAM resident
>>>> concept only).
>>>>
>>>>
>>>> Example
>>>> -------
>>>>
>>>> (component SOMENAME
>>>>   # inherit is like C++'s #include
>>>>   (inherit SOMEOTHERNAME WHERE)
>>>>   (pin 12 ... WHERE...ORIENTATION)
>>>>   (pin 13 ... WHERE...ORIENTATION)
>>>>   (pin 14 ...)
>>>> )
>>>>
>>>> Pin 12 and 13 are provided here and they are overloads because 12 and 13
>>>> have already been defined in SOMEOTHERNAME.  So this could be the means
>>>> of a *pin swap*.   However pin 14 is new, since SOMEOTHERNAME did not
>>>> have one, so this is an extension. In one example we see both a)
>>>> overloading and b) extension.
>>>>
>>>> This example would be a new SOMENAME component, and its graphics are
>>>> fully inherited from SOMEOTHERNAME, and can be extended here, but not
>>>> overloaded in any way that I can think up.
>>>>
>>>>
>>>> Where do they come from? and component versioning:
>>>> -------------------------------------------------
>>>> The ability to reference an external component by a GUID (globally
>>>> unique ID) of some kind lets you pull in components from anywhere in the
>>>> world.  A GUID inherently needs to respect versioning.  That is to say,
>>>> if you make an edit to a component that somebody else depends on, you
>>>> still have to provide the original version also.
>>>>     
>>>>         
>>> This is a great idea.  This prevents unwanted changes to your schematic
>>> when the underlying component is revised.  Doesn't Bazaar use the GUID
>>> concept for internally keeping track of revisions?  This would dovetail
>>> nicely into your GUID concept.  Where does this leave the concept of
>>> project libraries?  
>>>       
>>
>> As was suggested by Amir, we need to abstract the loading of libraries. 
>> A library is a RAM resident concept only, and should be thought of as a
>> cache.
>>
>> Maybe we should use the term "library source" to contemplate what we now
>> think of as a "library file".
>>
>> I see the need for four types of library sources:
>>
>> 1) a local library source which is simply a directory on the user's
>> disk, with separate component files in it.  This is one of two that are
>> writable.  The others are read only.
>>
>> 2) a local library source which comes from a SCCS repo, and is a repo
>> accessible via the local file system.  This is a read only library source.
>>
>> 3) any schematic with a "parts list" in it is a library source for a new
>> other schematic.  This is a read only source until you load that parts
>> list containing schematic purposefully to edit it.
>>
>> 4) a remote library source which comes from a global server and uses a
>> SCCS repo on the back end.  This is a read only library source.
>>
>>
>> Once these library sources are loaded into libraries in RAM, then their
>> distinctions are trivialized.  Because all access would be by library,
>> not library source.
>>
>> If 1) is coded first, with a view to reserve a revision ID slot as part
>> of the GUID, then we could walk before we run, and the user could use
>> the library source as a "working directory" in a SCCS on his own,
>> without our knowledge, until we can run.  And during the migration
>> phase, I think this alludes to a 5) type of library source, namely the
>> read only form of 1) which is needed for the kicad supplied components
>> until they can be put behind a SCCS and become 4). 
>>
>> The user would have to create his own 2) by "checking out" 4), and I
>> don't think we have to do that for him.  I for one would hope to have to
>> never do it, since I have decent Internet access.
>>
>> The downside of coding 1) *first* is that you won't pay as much
>> attention to the caching needs.
>>
>>
>>     
>>> Do they become a local library repo in project
>>> directory or does all this component information get saved in the
>>> schematic file?
>>>   
>>>       
>> If I get my parts list support (support here meaning user acceptance or
>> functionality) then all the components used in a schematic will *always*
>> reside in full in the parts list.  No exceptions.  The act of populating
>> a schematic specific parts list (well that is redundant since a parts
>> list by my definition is schematic specific), is a function of a good UI
>> and it would pull components in from various libraries.  Or symbols
>> which can be embellished in the parts list arena with s-expression
>> extension.  That parts list is available to other schematics, but only
>> as a library source, not as a parts list again, since a parts list must
>> live in the schematic that uses it.  In the parts list, there are
>> s-expressions defining what that part is, but inheritance can be used to
>> reference components in any library using a GUID.
>>
>>
>>
>>     
>>>   
>>>       
>>>> Enter the version control system.  Since each component and component
>>>> source code fragment exists in its own file, a source code control
>>>> system is used to house these files, and they can be accessed by using a
>>>> SCCS repo, file name and version number, over the Internet, or locally.
>>>>
>>>> One possibility for this is to build a tcp socket oriented bridge into
>>>> Bazaar's access API, which I assume lets you access any version of any
>>>> archived file.  Only one such server is needed for the "kicad supplied
>>>> libraries".  Behind the scenes the bridge/server simply uses launchad's
>>>> bazaar based repo.
>>>>
>>>> The loading of code fragments from afar need to be cached in RAM, so the
>>>> concept of a *library* does not go away.  A library is a RAM container
>>>> that caches those things coming from afar in order to enhance speed. 
>>>> But remember, kicad supplied libraries are essentially "read only" (for
>>>> all but those with the generating scripts).  So putting them under SCCS
>>>> and in launchpad as a primary place of dwelling is reasonable for those
>>>> with Internet access.  Others could check out the library branch.
>>>>     
>>>>         
>>> Does launchpad allow you to access repos without SSH keys? 
>>>       
>> I said launchpad, but probably should have said bazaar, and I don't know
>> the answer to your question.
>>     
> This makes more sense.  Some repo with public read only access for users
> eliminates the need for SSH logins in launchpad.
>
>   
>>     
>>>  It may be a
>>> bit cumbersome for users to have to create a launchpad ID just to check
>>> out or keep up to date with the main repo.  
>>>       
>> Indeed, and is a game breaker.  We only need read only access for the
>> kicad supplied components.  For remote access I think we need to keep
>> http: protocol on the table, and simply put a server up on a box
>> somewhere that hides the SCCS as a read only back end.  That can even be
>> coded in a language other than C++.  We only need one for the project,
>> but do not preclude 3rd party offerings.  We could make the server open
>> source so it can be re-used.  However if the  librarians don't put
>> barriers up, the kicad supplied remote library source can be modified
>> with the librariann's approval so multiple remote library sources are
>> not necessarily preferred.
>>
>>
>>     
>>> Maybe we could provide a
>>> guest user ID and SSH key to do this.
>>>   
>>>       
>>
>>     
>>>   
>>>       
>>>> How?
>>>> ----
>>>>
>>>> I can have this coded by Tuesday, given enough incentive.  (I'm just not
>>>> going to say which Tuesday.)
>>>>     
>>>>         
>>> Do I want to know what "enough incentive" entails?  Seriously though, it
>>> is a neat concept which you obviously have give a lot of thought to and
>>> I'm certainly interested in seeing how this would work.  Please keep me
>>> posted as I still have quite a bit of component library object clean up
>>> work to do.  If you have any questions about any of the component
>>> library object code or if there is anything I can do to help let me know.
>>>   
>>>       
>> The loading and searching functions need to be identified and put into a
>> "LIBRARY_SOURCE" C++ class hierarchy to facilitate loading abstraction. 
>> This is a way to abstract the library loading, and remember most
>> versions of these loaders are read only, in fact most are.  For the
>> LIBRARYs that can be saved, we could do an end run around the
>> LIBRARY_SOURCE API, since the only case we have in my design is type 1)
>> and this is a simple matter of saving the s-expression file to a local
>> disk file.  Even type 3) is read only.  To edit 3) you have to load that
>> schematic with a view to edit it, then you can save the entire schematic
>> to disk with its parts list.
>>     
> I wonder if it would be beneficial to create a LIBRARY_SOURCE for reading the
> existing library file format to maintain backwards compatibility.  The source
> code already exists to read these files.  Granted you wouldn't get any of the
> fancy new features like inheritance but you at least would not loose your
> current investment.  I don't know that an end around the LIBRARY_SOURCE API
> would be necessary.  I'm sure you could create a LIBRARY_SOURCE object that can
> provide writing so a derived class like OLD_KICAD_LIBRARY_SOURCE could even
> write library files in the old format ( not that you would want to ).
>   


I'm not a fan of adding write access functions to the LIBRARY_SOURCE
abstraction, not in any case.
I think you need to do an end run, there's only one special case of a
library source that is writable.

This is where I claim KISS.

Moving on to what the LIBRARY_SOURCE has to do, it has to give you the
sexpr text in the COMPONENT or symbol, shown below.

class COMPONENT or SYMBOL (i.e. CODE_FRAGMENT)
{
    pins              // pin table

    properties        // aka fields collection

    graphics          // graphics primitives

    std::string    sexpr;
}


The sexpr is the source to the component.  So unless you can load your old format components and create the new source file for it, I'd say this is wasted time.    Its a mess.

IMHO, it is better to run a batch file and convert everything up front one time.  
Somebody can still use the old version with the old software.


class LIBRARY
{
    COMPONENTS  collection;  some have been loaded, some are simply
known to be available on a deferred basis.

    LIBRARY_SOURCE* loader;
}


LIBRARY_SOURCE* is simply a text file fetcher, and by that I mean a
std::string reader just like you read a HTML page before you display it
in a browser.

class LIBRARY_SOURCE
{

    virtual std::string Read( GUID )
    :
    :
    list directory

    list directory on a component.  the final path fragment is the
version number.

    search for name value pairs in the properties
}


LIBRARY_SOURCE::Read() just reads a blob of text that is sexpr, and you
have to save that in COMPONENT::sexpr, at some point it will be parsed. 
It might not be a full component, so parsing it outside of a component
might not make sense until it is inherited by something that inherits or
includes it.  You might parse it just to find any nested inherits or
includes, such that those portions can also be cached in their
respective LIBRARYs.


Finally, parse the top most goody with STRING_LINE_READER and DSNLEXER
into the top most including COMPONENT.


I might base the LIBRARY_SOURCE API on a subset of SFTP like functions,
limited to read only access.  It is sort of a virtual file system, but
dumber, since you only need read only access, directory listing, and
something that lets you search.  You can pull it off using http protocol
behind the scenes,

When talking to the server, on the other side of the world, you do not
need to know you are using a SCCS because the revision could be like a
final path component in the GUID.  Or something like a file extension:  
.456


As far as the server goes:

The server accesses the SCCS and caches the part by revision ID, say in
a hashtable.  It responds to http protocol requests.  Because the
server's commitment is to honor the same sexpr text for any GUID, it in
theory never has to flush its cache.  New commits to the SCCS by the
librarians go in as a newer version of a part and not actual
replacement.  Like a webpage, and because of inclusion, you may have to
hit various LIBRARY_SOURCEs multiple times get everything into your
local LIBRARYs in RAM.

So for the remote case:

EESCHEMA -> LIBRARY -> LIBRARY_SOURCE -> http -> SERVER, say in python
or java or apache using websvn derivative -> SCCS

An calming consideration in all this is that much of what we are dealing
with here is read only, cached data, and it eventually ALL finds it way
into one of the open LIBRARYs (caches).  So it is cached in our local
LIBRARY and if remote, also in the server.  With a goofy query string
the server could give you SVG instead of sexpr so you can actually hit
it with a web browser also and see what you are looking at.

To do the SVG form, the server would have to in line expand all the
inheritance, and this would require that the repo not reference anything
outside itself.

>   
>> To test whether you get what I am saying, and fully understand the
>> concept of a parts list, it should be noted that I did not list "parts
>> list" as a "library source" for the schematic in which it resides.  A
>> parts list role is very unique with respect to the schematic in which it
>> resides.  It is the instantiation factory for that schematic and no
>> others.  However it can act as a library source for other schematics. 
>> This design will keep the parts list complete within every schematic,
>> and every schematic will stand alone, now and forever.  :) But this
>> assumes the inherited components' library sources remain accessible
>> forever, and with a GUID that includes a version number, and with a
>> stable remote library source, I think it could actually happen that way.
>>     
> If we could pull this off, that would be excellent.  One of the never ending
> headaches with every CAD system I have ever used is libraries.  It would be
> great if I could copy a project folder from one computer to another without
> having to remember to copy all of the library dependencies to each computer.
>
>   
>> The notion of loading needs to be better defined, since *deferring* may
>> be desired in the remote case.  There should be a "get contents"
>> (directory listing) function with a component specific load after that. 
>> Otherwise the remote library sources could be too slow.  Some more
>> thought needs to be put into this.
>>     
> The remote case is going to be the most difficult one to solve. 

Not by much.  Not if the APIs are designed properly up front, the
abstraction makes the client code identical.

The server does caching too, and I think a good sized server would have
all the components cached in RAM anyway.  It would be like hitting a
website that has all the pages already cached in RAM.  And if we use
http protocol for this, it is very do-able.


>  I suppose you
> could check to see if the local copy of the library repo is up to date with the
> remote repo if remote access is available and update the local repo when
> necessary.  You would always use the local copy of the repo when editing a
> schematic to prevent the latency of accessing the remote repo.
>   

For my personal use, I would not even feel compelled to copy the repo. 
It is "out there" and ready, on the net.  A typical component would have
what, less than 300 character's in the http content part of the reply?


Dick