kicad-developers team mailing list archive

[Clemens Koller <cko@xxxxxxxxx>]

Hello, Wayne & all!

Let me add some ideas - hoping they are not completely new:
Thinking about using: https://libgit2.github.com/ ?
Is says: 100% Cross Platform and Zero Dependencies right at the top...

And: In the long-term we might also think about using git for project and design management and versioning. (We could even store undo's and redo's in git?)
The commercial tools I've seen fail really *badly* when there is the need to compare complex designs or to do some version management along with the project documentation.
I've seen some crappy .HTML reports after running through a patchwork of painfully slow tools, ending with a miserable "graphical diff" of two layouts.

Regards,

Clemens

On 2017-02-17 15:30, Wayne Stambaugh wrote:
> On 2/16/2017 6:19 PM, Oliver Walters wrote:
>> Hi everyone,
>>
>> I am looking for some input on some thoughts I have regarding the future
>> of the Kicad "standard libraries" on GitHub. I think there are some
>> serious maintainability issues with the way the libraries are currently
>> handled, and this will only get worse once the new eeschema / .sweet
>> changes are rolled in.
>>
>> 1. Number of repositories
>>
>> There are currently 96 .pretty repos at https://github.com/KiCad . This
>> is a huge number of repos to maintain especially considering that for
>> each pull request, the user has to fork / branch / PR yet another
>> repository (and then the librarians have to do the same to check it ).
>> And then, you must pull the latest changes from (up to) 96 repos to keep
>> all the libs up to date.
>>
>> There are currently 90 .lib files (symbol libraries)
>> at https://github.com/KiCad/kicad-library . If the .sweet format follows
>> the .pretty format we will be asking users (at least the ones to want to
>> make use of the GitHub library feature) to manage over 180 separate
>> GitHub repositories.
>>
>> That's a whole lotta repos.
>>
>> 2. Library Management Tools
>>
>> Currently we are investigating using some CI tools to help manage pull
>> requests. If we go ahead with this we need to roll out these changes to
>> 96 repositories. And then maintain those CI tools.
>>
>> 3. GitHub "plugin" drawbacks
>>
>> AFAIK (from discussing this issue on IRC) the Kicad git plugin does not
>> actually make use of git, but rather makes use of the download-as-zip
>> functionality that github provides. This, to me, presents the following
>> issues:
>>
>> i) GitHub may break this feature at any time - even by changing the
>> format of the download URL!
>>
>> ii) Incompatible with non-github repositories - No facility to link in
>> with company repositories, for e.g., that are not served on GitHub
> 
> This would require a new plugin (git) using git as a back end.  This has
> been discussed many times in the past.  No one has stepped up to
> implement it yet.  Of course this is only useful if the user's repos
> (assuming they use repos) are managed in git.  If not, then yet another
> plugin would have to be written.  This is why the I/O manager supports
> plugins.
> 
>>
>> iii) Excessive bandwidth - Each time a .pretty lib is loaded, it
>> downloads a zip of the entire library? If we were making use of git
>> functionality it would be simple to just update the library (and store a
>> local cache) and thus only download the diff rather than re-download the
>> entire thing. 
> 
> Once again, a git plugin solves this problem nicely.
> 
>>
>> iv) Increased github load - As a follow-on from c) if Kicad attracts
>> significant user base then constant download of github libs will impact
>> significantly on github bandwidth. Other repositories have been shut or
>> throttled for high bandwidth use.
> 
> Yep.  That has always been a potential issue.
> 
>>
>> 4. User Ease-Of-Use
>>
>> Many users (newbies, especially) are confused by the library management.
>> Other EDA packages provide links to packaged library downloads, whereas
>> the initial library setup for Kicad is a lot more obtuse.
> 
> Yep.  AFAIK, most EDA apps aren't cross platform and/or don't care about
> portability across platforms.  KiCad does.  This means we have to
> implement a system to handle that which will always be more obtuse than
> just using hard coded paths which is the lowest common denominator.
> There is nothing preventing users from using hard coded paths if they
> don't care about portability.
> 
>>
>> Suggestions:
>>
>> I suggest that this problem be tackled as follows:
>>
>> A) Combine .pretty libs. The entire size of the pretty libraries is
>> currently ~50MB which is not huge. These should be combined into a
>> single repository on github which would make library management
>> infinitely easier.
> 
> The only way this makes sense is if you change the github plugin to
> include a path inside the repo i.e. ${KIGITHUB}/Resistors_SMD.pretty so
> that all footprints don't get parsed when cvpcb loads.  This would would
> have to happen before you convert to a single repo.
> 
>>
>> B) Single repo for .sweet libs (future) - As per A)
> 
> Initially this wont be an issue.  I have no short term plans to provide
> a github plugin for symbol libraries so a single repo will not be an issue.
> 
>>
>> C) Single repo for .3dshapes libs - Currently there is no real reason
>> that the 3D shapes have to be with the symbols. In fact, they are really
>> bloating the symbols library!
> 
> They are bloat no matter where you put them but putting them in their
> own repo makes sense.  Be ready for some angry users when their 3D
> models go missing from their design unless you come up with a seamless
> way to handle this.
> 
>>
>> D) Versioned downloads on the website - The website should offer a
>> libraries page under ./Downloads which has versioned releases of the
>> libraries. Users can choose to download the entire set of
>> footprints/symbols/3dmodels, or download subdirectories individually.
>> This presents an easy introduction to the libraries and also means that
>> users can be assured that unless they specifically download a newer
>> version, their libraries are not going to change underneath them.
> 
> Our package and web-site devs will need to buy into this but I think
> this is a good idea for users who prefer more static local libraries and
> can update on their own terms.
> 
>>
>> E) As an extension of D) users who wish to contribute or be at the
>> bleeding-edge of the libraries can fork the repos (3, not 186) and keep
>> these up to date very easily. The 3D model lib will be large but a
>> one-time download is fine as git will keep the changes up to date with
>> very little bandwidth.
> 
> Makes sense.
> 
>>
>> F) Improve the github plugin - This should be turned into a proper GIT
>> plugin (not a github plugin). This will have the following advantages
>> i) Not reliant on github (users can manage their own git repos for e.g.)
>> ii) Potential to partially check out repos (e.g. only pull a single
>> subdirectory)
> 
> See above.  The github plugin was designed for read only access to
> libraries on github not direct access to a git repo.  A git plugin would
> work for any git repo (local or remote) and would include write access
> as well.  Any takers?
> 
>>
>> OR
>>
>> G) Remove the github plugin - Library versioning might be better
>> performed by the user external to Kicad. I do not rely on Kicad to
>> manage the libraries for me, I do that in git externally. I know that
>> many other users do this too. From an architectural point of view, I do
>> not think that Kicad should "force" the user to manage their libraries
>> in this way. Should we add hard-coded support for other online services
>> too? Personally I think the github plugin makes life harder than any
>> problems it solves.
> 
> Removing the github plugin makes no sense.  No one is forced to use the
> github plugin.  Users are free to use the kicad plugin for file based
> libraries.  We can discuss changing the default configuration from the
> github plugin to the kicad plugin.  Once again, this will require buy in
> from the package devs as this may be a significant burden for them.
> 
>>
>> TL;DR - If / when the .sweet format change goes ahead, if we add another
>> ~100 github repositories, the librarians are going to go postal.
> 
> Since there will be no .sweet github plugin initially.  We can safely
> put all of the symbol libraries in a single repo.  Maybe that will
> motive someone to write a git plugin, hint, hint ;)
> 
>>
>> Cheers,
>> Oliver
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>