← Back to team overview

kicad-developers team mailing list archive

Re: GitHub Plugin (my nemesis)


Thanks for your detailed description. I think it is a nice way to go. However two remarks:
Does it simplify the maintenance of the kicad library by the librarians? Is it github compatible? If not, we would need to find another platform to host the libraries. 

If I look for git submodule, there is a lot of different opinions on why you should or should not use it. (e.g. https://www.google.de/amp/s/codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/amp/). Does it apply in our case? What about git subtree? 

Best regards 

Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <info@xxxxxxxxxxx>:
>Hi All
>I have to agree with some of the other posters, why not use git as it 
>was intended?
>Specifically I think the following makes sense.....
>- Keep the multiple repositories, this helps when a company only wants 
>to make certain sets of libs available to it's staff
>- Have a master repository that includes all other repositories as 
>- Have branches that are matched to kicad versions. This allows 
>footprint changes in a version safe way.
>- Use standard "git clone" (initial download) and "git pull" (update)
>the main repo which provides the entire set of available libs without 
>actually downloading the content for all libs.
>- When a specific lib is needed, do similar to what we do now, and use 
>"git submodule update --init --recursive $submoduleName" to just pull 
>that specific submodule
>- Allow the "main" library repo URI to be altered. This enables a 
>companies fork of the repo to be used.
>- Allow the "main" library repo URI to be ANY valid git URI. That means
>a repo on a local fileserver rather than a http server can be used. 
>Along with various security options etc.
>- Add a fairly simple scripted tool that's run "on release" to retrieve
>a README.md (an possibly a descriptor/index file) from each actual 
>library repo and update those within the "main" repo. This removes the 
>need to have the current case where manual edits are required to keep 
>the "main" repo in sync with what's available in the individual lib's.
>To add a new lib, it's then as simple as (from the main repo) doing 
>something like....
>- "git submodule add $URI"
>- "./scripts/update-library-indexes.sh"
>- "git commit -a -m $'added new module "modulename"\nupdated all module
>descriptions and index'"
>- "git push"
>All of the above allows the Main repo to be forked by a company (or 
>individual) and have their own custom repo's added very easily.
>It even allows libraries to be easily excluded or replaced, all using 
>extremely well developed management tools.
>On a side note, git submodules are stored in the main repository 
>basically as URI that includes a commit reference.
>That means it's easy to specify a specific library version to include
>a specific branch/tag of the main repo.
>As for the person that said "git isn't available for windows", sorry, 
>that's bunkum.
>There are many many development environments out there that ONLY run on
>windows that have git integration.
>A quick google search for "git for windows" will show what you need to 
>know there.
>Finally, wrapping git to duplicate the effective API that's currently
>use should be relatively trivial, resulting in virtually no code
>required to KICAD it's self.
>Assuming the existing functionality is cleanly wrapped by an API, the 
>only real change to KICAD would be the swapout of the module, and 
>addition of an easy way to change the URI
>I know I haven't covered everything in this email, but it should be a 
>good outline for further discussion
>On 22/09/17 09:21, Oliver Walters wrote:
>> Hi all,
>> Ok, now that the website integration with the libraries is (pretty 
>> much) done, and the licensing issue seems to be sorted, there is one 
>> final puzzle piece to solve before I'm happy with the state of the 
>> libraries for a v5 release.
>> *Goal: *Merge all footprint library repositories into a single repo
>> solve the ongoing dramas of maintaining 100+ repos.
>> *Problem*: The *only* thing standing in the way of just doing this is
>> that some users like the GitHub plugin and previous instruction is 
>> that this functionality cannot be removed.
>> The GitHub plugin functions by downloading a .zip file of a .pretty 
>> repo. If we merge all footprint libs into a single repo with multiple
>> subdirectories, this will not work anymore (as GitHub dosen't allow 
>> you to download a .zip of a single subdirectory).
>> Merging the repos is the *right thing to do*. But how to proceed?
>> *Options:*
>> *
>> *
>> /a) Drop github plugin feature, replace with library-download tool/
>> I don't think it is a good idea to live-load library data from GitHub
>> (a lot of other users agree too). It's slow, and a waste of bandwidth
>> to re-download the libs all the time.
>> We drop support for loading libraries direct from GitHub. However, we
>> add a tool for downloading libraries from GitHub and storing to disk.
>> Users can update as they like. This can be integrated in KiCad and
>> users can run this tool when they first install KICad. This means
>> no libs need to distributed with the installer and users can update
>> latest libs whenever they want.
>> /b) Improve github plugin to allow subdirectory traversal/
>> This is difficult and will only result in the plugin being slower. 
>> There are two ways I can see to do this:
>> i. Use Git API - tools exist that use this functionality - 
>> https://github.com/KinoLien/gitzip
>> ii. Use subversion - GitHub actually provides subversion API - 
>> https://www.seanw.org/blog/download-git-repo-subdirectory/
>> *Subversion*
>> In either case, I think that using the subversion tool to partially 
>> download the libraries would be a good approach (I assume quicker
>> using wget and the GitHub API).
>> 1. Does anyone have any experience using the C API for SVN?
>> 2. The Python bindings are pretty good - 
>> https://pypi.python.org/pypi/svn - and much easier to use. However, 
>> can we make the library download tools dependent on enabling the 
>> python plugin?
>> 3. Is there a way to checkout a subversion remote to memory (to 
>> replicate the functionality of the current GitHub plugin)? If not,
>> not sure how to approach option b) above.
>> Feedback appreciated. I think that it is very important especially
>> new users that this is improved. The GitHub plugin is constantly 
>> causing headaches!
>> Cheers,
>> Oliver
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>David Godfrey
>SB Tech Services
>mb: +61 437 286 200 <tel:+61437286200>
>Mailing list: https://launchpad.net/~kicad-developers
>Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>Unsubscribe : https://launchpad.net/~kicad-developers
>More help   : https://help.launchpad.net/ListHelp

Follow ups