kicad-developers team mailing list archive

[Oliver Walters <oliver.henry.walters@xxxxxxxxx>]

Hi everyone,

I am looking for some input on some thoughts I have regarding the future of
the Kicad "standard libraries" on GitHub. I think there are some serious
maintainability issues with the way the libraries are currently handled,
and this will only get worse once the new eeschema / .sweet changes are
rolled in.

1. Number of repositories

There are currently 96 .pretty repos at https://github.com/KiCad . This is
a huge number of repos to maintain especially considering that for each
pull request, the user has to fork / branch / PR yet another repository
(and then the librarians have to do the same to check it ). And then, you
must pull the latest changes from (up to) 96 repos to keep all the libs up
to date.

There are currently 90 .lib files (symbol libraries) at
https://github.com/KiCad/kicad-library . If the .sweet format follows the
.pretty format we will be asking users (at least the ones to want to make
use of the GitHub library feature) to manage over 180 separate GitHub
repositories.

That's a whole lotta repos.

2. Library Management Tools

Currently we are investigating using some CI tools to help manage pull
requests. If we go ahead with this we need to roll out these changes to 96
repositories. And then maintain those CI tools.

3. GitHub "plugin" drawbacks

AFAIK (from discussing this issue on IRC) the Kicad git plugin does not
actually make use of git, but rather makes use of the download-as-zip
functionality that github provides. This, to me, presents the following
issues:

i) GitHub may break this feature at any time - even by changing the format
of the download URL!

ii) Incompatible with non-github repositories - No facility to link in with
company repositories, for e.g., that are not served on GitHub

iii) Excessive bandwidth - Each time a .pretty lib is loaded, it downloads
a zip of the entire library? If we were making use of git functionality it
would be simple to just update the library (and store a local cache) and
thus only download the diff rather than re-download the entire thing.

iv) Increased github load - As a follow-on from c) if Kicad attracts
significant user base then constant download of github libs will impact
significantly on github bandwidth. Other repositories have been shut or
throttled for high bandwidth use.

4. User Ease-Of-Use

Many users (newbies, especially) are confused by the library management.
Other EDA packages provide links to packaged library downloads, whereas the
initial library setup for Kicad is a lot more obtuse.

Suggestions:

I suggest that this problem be tackled as follows:

A) Combine .pretty libs. The entire size of the pretty libraries is
currently ~50MB which is not huge. These should be combined into a single
repository on github which would make library management infinitely easier.

B) Single repo for .sweet libs (future) - As per A)

C) Single repo for .3dshapes libs - Currently there is no real reason that
the 3D shapes have to be with the symbols. In fact, they are really
bloating the symbols library!

D) Versioned downloads on the website - The website should offer a
libraries page under ./Downloads which has versioned releases of the
libraries. Users can choose to download the entire set of
footprints/symbols/3dmodels, or download subdirectories individually. This
presents an easy introduction to the libraries and also means that users
can be assured that unless they specifically download a newer version,
their libraries are not going to change underneath them.

E) As an extension of D) users who wish to contribute or be at the
bleeding-edge of the libraries can fork the repos (3, not 186) and keep
these up to date very easily. The 3D model lib will be large but a one-time
download is fine as git will keep the changes up to date with very little
bandwidth.

F) Improve the github plugin - This should be turned into a proper GIT
plugin (not a github plugin). This will have the following advantages
i) Not reliant on github (users can manage their own git repos for e.g.)
ii) Potential to partially check out repos (e.g. only pull a single
subdirectory)

OR

G) Remove the github plugin - Library versioning might be better performed
by the user external to Kicad. I do not rely on Kicad to manage the
libraries for me, I do that in git externally. I know that many other users
do this too. From an architectural point of view, I do not think that Kicad
should "force" the user to manage their libraries in this way. Should we
add hard-coded support for other online services too? Personally I think
the github plugin makes life harder than any problems it solves.

TL;DR - If / when the .sweet format change goes ahead, if we add another
~100 github repositories, the librarians are going to go postal.

Cheers,
Oliver