kicad-developers team mailing list archive

[Nick Østergaard <oe.nick@xxxxxxxxx>]

2017-02-17 1:18 GMT+01:00 Cirilo Bernardo <cirilo.bernardo@xxxxxxxxx>:
> Hi Oliver,
>
>  I agree with pretty much everything, especially removing the 3D models
> from the tree. I have argued for that a number of times in the past but I
> think as library dictator you may simply have to enforce it at some stage
> even if angry hordes complain later.
>
> On Fri, Feb 17, 2017 at 10:19 AM, Oliver Walters
> <oliver.henry.walters@xxxxxxxxx> wrote:
>> Hi everyone,
>>
>> I am looking for some input on some thoughts I have regarding the future of
>> the Kicad "standard libraries" on GitHub. I think there are some serious
>> maintainability issues with the way the libraries are currently handled, and
>> this will only get worse once the new eeschema / .sweet changes are rolled
>> in.
>>
>> 1. Number of repositories
>>
>> There are currently 96 .pretty repos at https://github.com/KiCad . This is a
>> huge number of repos to maintain especially considering that for each pull
>> request, the user has to fork / branch / PR yet another repository (and then
>> the librarians have to do the same to check it ). And then, you must pull
>> the latest changes from (up to) 96 repos to keep all the libs up to date.
>>
>> There are currently 90 .lib files (symbol libraries) at
>> https://github.com/KiCad/kicad-library . If the .sweet format follows the
>> .pretty format we will be asking users (at least the ones to want to make
>> use of the GitHub library feature) to manage over 180 separate GitHub
>> repositories.
>>
>> That's a whole lotta repos.
>>
>
> I would suggest putting a commit freeze on all current repos and merging
> them into one footprint repo. Contributors can pull the new repo and
> continue their work from there with little hassle. Individual (frozen) repos
> should remain until we're happy that everything is working and any helper
> scripts have been updated to work with the new structure.
>
> Changes to the schematic repos should be made when we get to the point
> of converting existing symbols - at that point a new schematic repo should
> be created and the 3D models moved.
>
> Some thought needs to go into how we associate data between the 3
> disjoint repos (symbols, footprints, 3D). One simple solution is to require
> the repos to be cloned to specific directories at the same directory level,
> for example:
>
> kicad_libs
> +--symbols
> +--footprints
> +--3D
>
> With such an arrangement we can always define a ${KICAD_DATA_ROOT}
> and make modifications to the filename search algorithms; stock symbols
> and footprints can then reference a 3D model via ${KICAD_DATA_ROOT}/3D.
>
> Some thought needs to go into the github plugin and how it can continue
> to be useful to people who do use it. I don't believe there will be any
> problems encountered in merging the footprint libs which would not be
> encountered with the libs separated as they currently are. Pegging specific
> files to specific commits will always be a nightmare to manage and simply
> isn't a problem for the kicad team to solve. One thing which may displease
> users is that they will not see a "Group_A" vs "Group_B" of footprints -
> it will appear that all footprints are stuffed into one single group, but this
> sort of thing can all be managed.
>
>> 2. Library Management Tools
>>
>> Currently we are investigating using some CI tools to help manage pull
>> requests. If we go ahead with this we need to roll out these changes to 96
>> repositories. And then maintain those CI tools.
>>

I don't see how having many repos dictate that we need so many "CI
tools" individually. We could potentially have the CI to generate
previews or somthing similar for each pull request.

>
> Yup, that's just crazy. Aim for the 3, but put on your flameproof suit.
>
>> 3. GitHub "plugin" drawbacks
>>
>> AFAIK (from discussing this issue on IRC) the Kicad git plugin does not
>> actually make use of git, but rather makes use of the download-as-zip
>> functionality that github provides. This, to me, presents the following
>> issues:
>>
>> i) GitHub may break this feature at any time - even by changing the format
>> of the download URL!
>>
>> ii) Incompatible with non-github repositories - No facility to link in with
>> company repositories, for e.g., that are not served on GitHub
>>
>> iii) Excessive bandwidth - Each time a .pretty lib is loaded, it downloads a
>> zip of the entire library? If we were making use of git functionality it
>> would be simple to just update the library (and store a local cache) and
>> thus only download the diff rather than re-download the entire thing.

IIRC there is a session cache, so it will only re-download it when you
restart kicad or pcbnew.

>>
>> iv) Increased github load - As a follow-on from c) if Kicad attracts
>> significant user base then constant download of github libs will impact
>> significantly on github bandwidth. Other repositories have been shut or
>> throttled for high bandwidth use.
>>
>
> Definitely a problem. Personally I'd prefer a custom tool running over git
> to help users manage selection of footprints and their revisions etc. - the
> question is who has time to do that. Personally I'd never used the github
> plugin as it just doesn't sit well with the way I do things. Other people
> love it though. Some serious thought needs to go into the concept of
> remote access of footprints etc. Actually using git rather than pulling
> zip files with no history is much more inviting - we can then use commit
> hashes of the files to help manage revision control of those files within
> a project. (Do I update this footprint, can I roll it back, is there a new
> revision available, etc.)
>
>> 4. User Ease-Of-Use
>>
>> Many users (newbies, especially) are confused by the library management.
>> Other EDA packages provide links to packaged library downloads, whereas the
>> initial library setup for Kicad is a lot more obtuse.
>>
>> Suggestions:
>>
>> I suggest that this problem be tackled as follows:
>>
>> A) Combine .pretty libs. The entire size of the pretty libraries is
>> currently ~50MB which is not huge. These should be combined into a single
>> repository on github which would make library management infinitely easier.
>>
>
> Definitely.
>
>> B) Single repo for .sweet libs (future) - As per A)
>>
>
> Definitely - as I said above, this should be done when we've converted and
> tested the .lib files to the new format.

I don't think it is best to combine the pretty libs. I think it is
nice to store the data that way. We could start using a tool like
"repo" (from google, used to manage android sources) which is a tool
to manage many git repos. Then all that is needed when a new lib is
added is to add one line in a manifest file, and it is possible to
point to speficic versions in each repo making it easy to fetch a
specific release.

>
>> C) Single repo for .3dshapes libs - Currently there is no real reason that
>> the 3D shapes have to be with the symbols. In fact, they are really bloating
>> the symbols library!
>>
>
> Definitely - I've always argued that they have no business being in there and
> simply waste bandwidth for people who have no interest whatsoever in VRML
> models. The ability to use STEP models in the dev branch will make matters
> much worse.
>
>> D) Versioned downloads on the website - The website should offer a libraries
>> page under ./Downloads which has versioned releases of the libraries. Users
>> can choose to download the entire set of footprints/symbols/3dmodels, or
>> download subdirectories individually. This presents an easy introduction to
>> the libraries and also means that users can be assured that unless they
>> specifically download a newer version, their libraries are not going to
>> change underneath them.
>>
>
> I prefer to tag a git repository and provide instructions for retrieval. This is
> easily done with a single repository for each of symbols, footprints, and 3D;
> it is an absolute nuisance with the current huge number of footprint repos
> (that tagging script had better work just right ...). Github in particular has a
> "Releases" scheme in its API. Maybe that could be of some help.
>
>> E) As an extension of D) users who wish to contribute or be at the
>> bleeding-edge of the libraries can fork the repos (3, not 186) and keep
>> these up to date very easily. The 3D model lib will be large but a one-time
>> download is fine as git will keep the changes up to date with very little
>> bandwidth.
>>
>> F) Improve the github plugin - This should be turned into a proper GIT
>> plugin (not a github plugin). This will have the following advantages
>> i) Not reliant on github (users can manage their own git repos for e.g.)
>> ii) Potential to partially check out repos (e.g. only pull a single
>> subdirectory)
>>
>> OR
>>
>> G) Remove the github plugin - Library versioning might be better performed
>> by the user external to Kicad. I do not rely on Kicad to manage the
>> libraries for me, I do that in git externally. I know that many other users
>> do this too. From an architectural point of view, I do not think that Kicad
>> should "force" the user to manage their libraries in this way. Should we add
>> hard-coded support for other online services too? Personally I think the
>> github plugin makes life harder than any problems it solves.
>>

We are not really forcing users to manage their libraries in this way.
It is just the default.


>
> I'm all for (G), but see my notes above. I don't find the github plugin useful
> for me, but we have to be mindful of people who do use it.
>
> - Cirilo
>
>> TL;DR - If / when the .sweet format change goes ahead, if we add another
>> ~100 github repositories, the librarians are going to go postal.
>>
>> Cheers,
>> Oliver
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp