← Back to team overview

kicad-developers team mailing list archive

Re: GitHub Plugin (my nemesis)


Hi All

I have to agree with some of the other posters, why not use git as it was intended?

Specifically I think the following makes sense.....

- Keep the multiple repositories, this helps when a company only wants to make certain sets of libs available to it's staff

- Have a master repository that includes all other repositories as submodules.

- Have branches that are matched to kicad versions. This allows footprint changes in a version safe way.

- Use standard "git clone" (initial download) and "git pull" (update) on the main repo which provides the entire set of available libs without actually downloading the content for all libs.

- When a specific lib is needed, do similar to what we do now, and use "git submodule update --init --recursive $submoduleName" to just pull that specific submodule

- Allow the "main" library repo URI to be altered. This enables a companies fork of the repo to be used.

- Allow the "main" library repo URI to be ANY valid git URI. That means a repo on a local fileserver rather than a http server can be used. Along with various security options etc.

- Add a fairly simple scripted tool that's run "on release" to retrieve a README.md (an possibly a descriptor/index file) from each actual library repo and update those within the "main" repo. This removes the need to have the current case where manual edits are required to keep the "main" repo in sync with what's available in the individual lib's.

To add a new lib, it's then as simple as (from the main repo) doing something like....
- "git submodule add $URI"
- "./scripts/update-library-indexes.sh"
- "git commit -a -m $'added new module "modulename"\nupdated all module descriptions and index'"
- "git push"

All of the above allows the Main repo to be forked by a company (or individual) and have their own custom repo's added very easily. It even allows libraries to be easily excluded or replaced, all using extremely well developed management tools.

On a side note, git submodules are stored in the main repository basically as URI that includes a commit reference. That means it's easy to specify a specific library version to include in a specific branch/tag of the main repo.

As for the person that said "git isn't available for windows", sorry, that's bunkum. There are many many development environments out there that ONLY run on windows that have git integration. A quick google search for "git for windows" will show what you need to know there.

Finally, wrapping git to duplicate the effective API that's currently in use should be relatively trivial, resulting in virtually no code changes required to KICAD it's self. Assuming the existing functionality is cleanly wrapped by an API, the only real change to KICAD would be the swapout of the module, and addition of an easy way to change the URI

I know I haven't covered everything in this email, but it should be a good outline for further discussion

On 22/09/17 09:21, Oliver Walters wrote:
Hi all,

Ok, now that the website integration with the libraries is (pretty much) done, and the licensing issue seems to be sorted, there is one final puzzle piece to solve before I'm happy with the state of the libraries for a v5 release.

*Goal: *Merge all footprint library repositories into a single repo to solve the ongoing dramas of maintaining 100+ repos.

*Problem*: The *only* thing standing in the way of just doing this is that some users like the GitHub plugin and previous instruction is that this functionality cannot be removed.

The GitHub plugin functions by downloading a .zip file of a .pretty repo. If we merge all footprint libs into a single repo with multiple subdirectories, this will not work anymore (as GitHub dosen't allow you to download a .zip of a single subdirectory).

Merging the repos is the *right thing to do*. But how to proceed?

/a) Drop github plugin feature, replace with library-download tool/

I don't think it is a good idea to live-load library data from GitHub (a lot of other users agree too). It's slow, and a waste of bandwidth to re-download the libs all the time.

We drop support for loading libraries direct from GitHub. However, we add a tool for downloading libraries from GitHub and storing to disk. Users can update as they like. This can be integrated in KiCad and new users can run this tool when they first install KICad. This means that no libs need to distributed with the installer and users can update to latest libs whenever they want.

/b) Improve github plugin to allow subdirectory traversal/

This is difficult and will only result in the plugin being slower. There are two ways I can see to do this:

i. Use Git API - tools exist that use this functionality - https://github.com/KinoLien/gitzip ii. Use subversion - GitHub actually provides subversion API - https://www.seanw.org/blog/download-git-repo-subdirectory/


In either case, I think that using the subversion tool to partially download the libraries would be a good approach (I assume quicker than using wget and the GitHub API).

1. Does anyone have any experience using the C API for SVN?
2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn - and much easier to use. However, can we make the library download tools dependent on enabling the python plugin? 3. Is there a way to checkout a subversion remote to memory (to replicate the functionality of the current GitHub plugin)? If not, I'm not sure how to approach option b) above.

Feedback appreciated. I think that it is very important especially for new users that this is improved. The GitHub plugin is constantly causing headaches!


Mailing list: https://launchpad.net/~kicad-developers
Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp
David Godfrey
SB Tech Services
mb: +61 437 286 200 <tel:+61437286200>

Follow ups