kicad-developers team mailing list archive

[Wayne Stambaugh <stambaughw@xxxxxxxxxxx>]

On 10/6/2010 3:17 PM, Dick Hollenbeck wrote:
> On 10/05/2010 07:04 PM, Wayne Stambaugh wrote:
>> On 10/5/2010 10:33 AM, Dick Hollenbeck wrote:
>>   
>>> I get paid to brainstorm, so brace yourself.
>>>
>>> Continuing with library brainstorming, the following is put on the table
>>> for consideration and eventual evaluation. Brace yourself, this is out
>>> of the box.  Open up your mind before proceeding.
>>>
>>> Remember the HTML editors which let you view HTML source and the
>>> resultant GUI presentation at the same time?  One panel has the html
>>> source, another shows you its effects.  The following concepts use that
>>> UI paradigm.  Use the source Luke, use the source.
>>>
>>>
>>> A library is not a library file:
>>> -------------------------------
>>>
>>> The concept of a *library file* is gone.  There are none.  The concept
>>> of a library can still exist but it is a RAM resident concept.  Each
>>> component that we traditionally think of residing in a *library file* is
>>> actually in its own file on disk in a directory (or repo or schematic). 
>>> A library is only built in RAM during the loading of a collection of
>>> components.
>>>
>>>
>>> Loading components:
>>> ------------------
>>>
>>> The act of loading components utilizes the notion of inheritance and
>>> classical file inclusion like the C++ #include.  Let's go back an
>>> understand inheritance thinking about Java. In a Java virtual machine
>>> each class definition container has a method table.  That table is
>>> initially populated with methods from the base class.  As the base class
>>> is extended some of those methods are overloaded (replaced) with ones
>>> from the derived class in the internal method table.  Additional fields
>>> can be introduced in derived classes also, into what is a field table
>>> for that class.  Now lets move that conceptual model to a component. 
>>> Lets assume we have a pin table, and a graphics table.  We give a
>>> component the ability to inherit from: 1) a base code fragment,  or 2)
>>> symbol or 3) component.  In this case you can do pin additions, or pin
>>> modifications, pin swapping, in the same way you can overload a Java
>>> class's method table.  Pins can be overloaded or replaced, and graphical
>>> primitives can be inherited and extended.
>>>     
>> Conceptually I really like this idea from a coding stand point.  If you
>> are careful you could implement this so that the code maps one to one to
>> your new conceptual model.  This would make it easier to implement new
>> features will breaking the existing ones.  The only issue I see is how
>> to convert existing libraries into the new format or do we abandon
>> backwards capability and push on towards a superior implementation.
>>   
> 
> I think changing the COMPONENT C++ object is wise.  We have to have a
> few s-expression keywords that we would map onto a 1 for 1 C++ object:
> 
> 1) (component ...)
> 
> for the pin table:
> 
> 2) (pin ...)   # w/coordinates relative to the component's (x,y) origin.
> 
> 
> for the graphics table:
> 
> 3) (line ...)
> 4) (arc ...)
> 5) (circle ...)
> 6) (rect ...)
> 
> : yes more!
> 
> into a separate property table goes each property:
> 
> 7) field or property
> 
> 
> 
> The rest of the keywords do not need to map onto "objects" but could map
> onto "actions", and one that I can think of is "pin_del":
> 
> (pin_del 12)
> 
> Which would delete pin 12 should it already have been inherited.
> 
> So maybe you have "field_del"  or "property_del" also.  Trying to delete
> graphics is more cumbersome.
> 
> We can and should write a conversion program for existing libraries to
> new stuff.
> 
> 
> class COMPONENT
> {
>     pins              // pin table
> 
>     properties        // aka fields collection
> 
>     graphics          // graphics primitives
> }
> 
> Obviously keyword "pin" comes with a type which would imply the fixed
> graphics for it.
> 
> more below:
> 
>>   
>>> The magic happens during the loading phase of the S-expressions, so that
>>> the "built up" component is fully assembled in RAM by using pieces from
>>> various places "out there", while *loading*.
>>>
>>> If desired, the various pieces can be ear-marked with their origins in
>>> full recognition of the existence of inheritance.  One might do this
>>> only to facilitate graphical editing of a derived component, although if
>>> the grammar is sweet enough, textual editing may be enough for all
>>> editing of derived components.
>>>     
>> I have always preferred a well designed textual editing interface for
>> creating drawings.  In many ways I find it easier than performing these
>> operations graphically.  I however probably do not represent the larger
>> group of users who prefer a GUI editor.  This may be one of those ideas
>> to have our users group liaison check in with the users group to see how
>> they feel about it.
>>
>>   
>>> Re-loading is simply done by clicking a "parse" or "show" button.  The
>>> the parser reloads the source stream and does file inclusions as called
>>> for.  A portion of that source stream comes from the textual UI panel. 
>>> Inherited portions come from a library (which is now a RAM resident
>>> concept only).
>>>
>>>
>>> Example
>>> -------
>>>
>>> (component SOMENAME
>>>   # inherit is like C++'s #include
>>>   (inherit SOMEOTHERNAME WHERE)
>>>   (pin 12 ... WHERE...ORIENTATION)
>>>   (pin 13 ... WHERE...ORIENTATION)
>>>   (pin 14 ...)
>>> )
>>>
>>> Pin 12 and 13 are provided here and they are overloads because 12 and 13
>>> have already been defined in SOMEOTHERNAME.  So this could be the means
>>> of a *pin swap*.   However pin 14 is new, since SOMEOTHERNAME did not
>>> have one, so this is an extension. In one example we see both a)
>>> overloading and b) extension.
>>>
>>> This example would be a new SOMENAME component, and its graphics are
>>> fully inherited from SOMEOTHERNAME, and can be extended here, but not
>>> overloaded in any way that I can think up.
>>>
>>>
>>> Where do they come from? and component versioning:
>>> -------------------------------------------------
>>> The ability to reference an external component by a GUID (globally
>>> unique ID) of some kind lets you pull in components from anywhere in the
>>> world.  A GUID inherently needs to respect versioning.  That is to say,
>>> if you make an edit to a component that somebody else depends on, you
>>> still have to provide the original version also.
>>>     
>> This is a great idea.  This prevents unwanted changes to your schematic
>> when the underlying component is revised.  Doesn't Bazaar use the GUID
>> concept for internally keeping track of revisions?  This would dovetail
>> nicely into your GUID concept.  Where does this leave the concept of
>> project libraries?  
> 
> 
> As was suggested by Amir, we need to abstract the loading of libraries. 
> A library is a RAM resident concept only, and should be thought of as a
> cache.
> 
> Maybe we should use the term "library source" to contemplate what we now
> think of as a "library file".
> 
> I see the need for four types of library sources:
> 
> 1) a local library source which is simply a directory on the user's
> disk, with separate component files in it.  This is one of two that are
> writable.  The others are read only.
> 
> 2) a local library source which comes from a SCCS repo, and is a repo
> accessible via the local file system.  This is a read only library source.
> 
> 3) any schematic with a "parts list" in it is a library source for a new
> other schematic.  This is a read only source until you load that parts
> list containing schematic purposefully to edit it.
> 
> 4) a remote library source which comes from a global server and uses a
> SCCS repo on the back end.  This is a read only library source.
> 
> 
> Once these library sources are loaded into libraries in RAM, then their
> distinctions are trivialized.  Because all access would be by library,
> not library source.
> 
> If 1) is coded first, with a view to reserve a revision ID slot as part
> of the GUID, then we could walk before we run, and the user could use
> the library source as a "working directory" in a SCCS on his own,
> without our knowledge, until we can run.  And during the migration
> phase, I think this alludes to a 5) type of library source, namely the
> read only form of 1) which is needed for the kicad supplied components
> until they can be put behind a SCCS and become 4). 
> 
> The user would have to create his own 2) by "checking out" 4), and I
> don't think we have to do that for him.  I for one would hope to have to
> never do it, since I have decent Internet access.
> 
> The downside of coding 1) *first* is that you won't pay as much
> attention to the caching needs.
> 
> 
>> Do they become a local library repo in project
>> directory or does all this component information get saved in the
>> schematic file?
>>   
> 
> If I get my parts list support (support here meaning user acceptance or
> functionality) then all the components used in a schematic will *always*
> reside in full in the parts list.  No exceptions.  The act of populating
> a schematic specific parts list (well that is redundant since a parts
> list by my definition is schematic specific), is a function of a good UI
> and it would pull components in from various libraries.  Or symbols
> which can be embellished in the parts list arena with s-expression
> extension.  That parts list is available to other schematics, but only
> as a library source, not as a parts list again, since a parts list must
> live in the schematic that uses it.  In the parts list, there are
> s-expressions defining what that part is, but inheritance can be used to
> reference components in any library using a GUID.
> 
> 
> 
>>   
>>> Enter the version control system.  Since each component and component
>>> source code fragment exists in its own file, a source code control
>>> system is used to house these files, and they can be accessed by using a
>>> SCCS repo, file name and version number, over the Internet, or locally.
>>>
>>> One possibility for this is to build a tcp socket oriented bridge into
>>> Bazaar's access API, which I assume lets you access any version of any
>>> archived file.  Only one such server is needed for the "kicad supplied
>>> libraries".  Behind the scenes the bridge/server simply uses launchad's
>>> bazaar based repo.
>>>
>>> The loading of code fragments from afar need to be cached in RAM, so the
>>> concept of a *library* does not go away.  A library is a RAM container
>>> that caches those things coming from afar in order to enhance speed. 
>>> But remember, kicad supplied libraries are essentially "read only" (for
>>> all but those with the generating scripts).  So putting them under SCCS
>>> and in launchpad as a primary place of dwelling is reasonable for those
>>> with Internet access.  Others could check out the library branch.
>>>     
>> Does launchpad allow you to access repos without SSH keys? 
> 
> I said launchpad, but probably should have said bazaar, and I don't know
> the answer to your question.

This makes more sense.  Some repo with public read only access for users
eliminates the need for SSH logins in launchpad.

> 
>>  It may be a
>> bit cumbersome for users to have to create a launchpad ID just to check
>> out or keep up to date with the main repo.  
> 
> Indeed, and is a game breaker.  We only need read only access for the
> kicad supplied components.  For remote access I think we need to keep
> http: protocol on the table, and simply put a server up on a box
> somewhere that hides the SCCS as a read only back end.  That can even be
> coded in a language other than C++.  We only need one for the project,
> but do not preclude 3rd party offerings.  We could make the server open
> source so it can be re-used.  However if the  librarians don't put
> barriers up, the kicad supplied remote library source can be modified
> with the librariann's approval so multiple remote library sources are
> not necessarily preferred.
> 
> 
>> Maybe we could provide a
>> guest user ID and SSH key to do this.
>>   
> 
> 
>>   
>>>
>>> How?
>>> ----
>>>
>>> I can have this coded by Tuesday, given enough incentive.  (I'm just not
>>> going to say which Tuesday.)
>>>     
>> Do I want to know what "enough incentive" entails?  Seriously though, it
>> is a neat concept which you obviously have give a lot of thought to and
>> I'm certainly interested in seeing how this would work.  Please keep me
>> posted as I still have quite a bit of component library object clean up
>> work to do.  If you have any questions about any of the component
>> library object code or if there is anything I can do to help let me know.
>>   
> 
> The loading and searching functions need to be identified and put into a
> "LIBRARY_SOURCE" C++ class hierarchy to facilitate loading abstraction. 
> This is a way to abstract the library loading, and remember most
> versions of these loaders are read only, in fact most are.  For the
> LIBRARYs that can be saved, we could do an end run around the
> LIBRARY_SOURCE API, since the only case we have in my design is type 1)
> and this is a simple matter of saving the s-expression file to a local
> disk file.  Even type 3) is read only.  To edit 3) you have to load that
> schematic with a view to edit it, then you can save the entire schematic
> to disk with its parts list.

I wonder if it would be beneficial to create a LIBRARY_SOURCE for reading the
existing library file format to maintain backwards compatibility.  The source
code already exists to read these files.  Granted you wouldn't get any of the
fancy new features like inheritance but you at least would not loose your
current investment.  I don't know that an end around the LIBRARY_SOURCE API
would be necessary.  I'm sure you could create a LIBRARY_SOURCE object that can
provide writing so a derived class like OLD_KICAD_LIBRARY_SOURCE could even
write library files in the old format ( not that you would want to ).

> 
> To test whether you get what I am saying, and fully understand the
> concept of a parts list, it should be noted that I did not list "parts
> list" as a "library source" for the schematic in which it resides.  A
> parts list role is very unique with respect to the schematic in which it
> resides.  It is the instantiation factory for that schematic and no
> others.  However it can act as a library source for other schematics. 
> This design will keep the parts list complete within every schematic,
> and every schematic will stand alone, now and forever.  :) But this
> assumes the inherited components' library sources remain accessible
> forever, and with a GUID that includes a version number, and with a
> stable remote library source, I think it could actually happen that way.

If we could pull this off, that would be excellent.  One of the never ending
headaches with every CAD system I have ever used is libraries.  It would be
great if I could copy a project folder from one computer to another without
having to remember to copy all of the library dependencies to each computer.

> 
> The notion of loading needs to be better defined, since *deferring* may
> be desired in the remote case.  There should be a "get contents"
> (directory listing) function with a component specific load after that. 
> Otherwise the remote library sources could be too slow.  Some more
> thought needs to be put into this.

The remote case is going to be the most difficult one to solve.  I suppose you
could check to see if the local copy of the library repo is up to date with the
remote repo if remote access is available and update the local repo when
necessary.  You would always use the local copy of the repo when editing a
schematic to prevent the latency of accessing the remote repo.

Wayne

> 
> 
> Dick
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>