kicad-developers team mailing list archive

[Cirilo Bernardo <cirilo.bernardo@xxxxxxxxx>]

Hi Oliver,

 I agree with pretty much everything, especially removing the 3D models
from the tree. I have argued for that a number of times in the past but I
think as library dictator you may simply have to enforce it at some stage
even if angry hordes complain later.

On Fri, Feb 17, 2017 at 10:19 AM, Oliver Walters
<oliver.henry.walters@xxxxxxxxx> wrote:
> Hi everyone,
>
> I am looking for some input on some thoughts I have regarding the future of
> the Kicad "standard libraries" on GitHub. I think there are some serious
> maintainability issues with the way the libraries are currently handled, and
> this will only get worse once the new eeschema / .sweet changes are rolled
> in.
>
> 1. Number of repositories
>
> There are currently 96 .pretty repos at https://github.com/KiCad . This is a
> huge number of repos to maintain especially considering that for each pull
> request, the user has to fork / branch / PR yet another repository (and then
> the librarians have to do the same to check it ). And then, you must pull
> the latest changes from (up to) 96 repos to keep all the libs up to date.
>
> There are currently 90 .lib files (symbol libraries) at
> https://github.com/KiCad/kicad-library . If the .sweet format follows the
> .pretty format we will be asking users (at least the ones to want to make
> use of the GitHub library feature) to manage over 180 separate GitHub
> repositories.
>
> That's a whole lotta repos.
>

I would suggest putting a commit freeze on all current repos and merging
them into one footprint repo. Contributors can pull the new repo and
continue their work from there with little hassle. Individual (frozen) repos
should remain until we're happy that everything is working and any helper
scripts have been updated to work with the new structure.

Changes to the schematic repos should be made when we get to the point
of converting existing symbols - at that point a new schematic repo should
be created and the 3D models moved.

Some thought needs to go into how we associate data between the 3
disjoint repos (symbols, footprints, 3D). One simple solution is to require
the repos to be cloned to specific directories at the same directory level,
for example:

kicad_libs
+--symbols
+--footprints
+--3D

With such an arrangement we can always define a ${KICAD_DATA_ROOT}
and make modifications to the filename search algorithms; stock symbols
and footprints can then reference a 3D model via ${KICAD_DATA_ROOT}/3D.

Some thought needs to go into the github plugin and how it can continue
to be useful to people who do use it. I don't believe there will be any
problems encountered in merging the footprint libs which would not be
encountered with the libs separated as they currently are. Pegging specific
files to specific commits will always be a nightmare to manage and simply
isn't a problem for the kicad team to solve. One thing which may displease
users is that they will not see a "Group_A" vs "Group_B" of footprints -
it will appear that all footprints are stuffed into one single group, but this
sort of thing can all be managed.

> 2. Library Management Tools
>
> Currently we are investigating using some CI tools to help manage pull
> requests. If we go ahead with this we need to roll out these changes to 96
> repositories. And then maintain those CI tools.
>

Yup, that's just crazy. Aim for the 3, but put on your flameproof suit.

> 3. GitHub "plugin" drawbacks
>
> AFAIK (from discussing this issue on IRC) the Kicad git plugin does not
> actually make use of git, but rather makes use of the download-as-zip
> functionality that github provides. This, to me, presents the following
> issues:
>
> i) GitHub may break this feature at any time - even by changing the format
> of the download URL!
>
> ii) Incompatible with non-github repositories - No facility to link in with
> company repositories, for e.g., that are not served on GitHub
>
> iii) Excessive bandwidth - Each time a .pretty lib is loaded, it downloads a
> zip of the entire library? If we were making use of git functionality it
> would be simple to just update the library (and store a local cache) and
> thus only download the diff rather than re-download the entire thing.
>
> iv) Increased github load - As a follow-on from c) if Kicad attracts
> significant user base then constant download of github libs will impact
> significantly on github bandwidth. Other repositories have been shut or
> throttled for high bandwidth use.
>

Definitely a problem. Personally I'd prefer a custom tool running over git
to help users manage selection of footprints and their revisions etc. - the
question is who has time to do that. Personally I'd never used the github
plugin as it just doesn't sit well with the way I do things. Other people
love it though. Some serious thought needs to go into the concept of
remote access of footprints etc. Actually using git rather than pulling
zip files with no history is much more inviting - we can then use commit
hashes of the files to help manage revision control of those files within
a project. (Do I update this footprint, can I roll it back, is there a new
revision available, etc.)

> 4. User Ease-Of-Use
>
> Many users (newbies, especially) are confused by the library management.
> Other EDA packages provide links to packaged library downloads, whereas the
> initial library setup for Kicad is a lot more obtuse.
>
> Suggestions:
>
> I suggest that this problem be tackled as follows:
>
> A) Combine .pretty libs. The entire size of the pretty libraries is
> currently ~50MB which is not huge. These should be combined into a single
> repository on github which would make library management infinitely easier.
>

Definitely.

> B) Single repo for .sweet libs (future) - As per A)
>

Definitely - as I said above, this should be done when we've converted and
tested the .lib files to the new format.

> C) Single repo for .3dshapes libs - Currently there is no real reason that
> the 3D shapes have to be with the symbols. In fact, they are really bloating
> the symbols library!
>

Definitely - I've always argued that they have no business being in there and
simply waste bandwidth for people who have no interest whatsoever in VRML
models. The ability to use STEP models in the dev branch will make matters
much worse.

> D) Versioned downloads on the website - The website should offer a libraries
> page under ./Downloads which has versioned releases of the libraries. Users
> can choose to download the entire set of footprints/symbols/3dmodels, or
> download subdirectories individually. This presents an easy introduction to
> the libraries and also means that users can be assured that unless they
> specifically download a newer version, their libraries are not going to
> change underneath them.
>

I prefer to tag a git repository and provide instructions for retrieval. This is
easily done with a single repository for each of symbols, footprints, and 3D;
it is an absolute nuisance with the current huge number of footprint repos
(that tagging script had better work just right ...). Github in particular has a
"Releases" scheme in its API. Maybe that could be of some help.

> E) As an extension of D) users who wish to contribute or be at the
> bleeding-edge of the libraries can fork the repos (3, not 186) and keep
> these up to date very easily. The 3D model lib will be large but a one-time
> download is fine as git will keep the changes up to date with very little
> bandwidth.
>
> F) Improve the github plugin - This should be turned into a proper GIT
> plugin (not a github plugin). This will have the following advantages
> i) Not reliant on github (users can manage their own git repos for e.g.)
> ii) Potential to partially check out repos (e.g. only pull a single
> subdirectory)
>
> OR
>
> G) Remove the github plugin - Library versioning might be better performed
> by the user external to Kicad. I do not rely on Kicad to manage the
> libraries for me, I do that in git externally. I know that many other users
> do this too. From an architectural point of view, I do not think that Kicad
> should "force" the user to manage their libraries in this way. Should we add
> hard-coded support for other online services too? Personally I think the
> github plugin makes life harder than any problems it solves.
>

I'm all for (G), but see my notes above. I don't find the github plugin useful
for me, but we have to be mindful of people who do use it.

- Cirilo

> TL;DR - If / when the .sweet format change goes ahead, if we add another
> ~100 github repositories, the librarians are going to go postal.
>
> Cheers,
> Oliver
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>