kicad-developers team mailing list archive

[Oliver Walters <oliver.henry.walters@xxxxxxxxx>]

Simon, David,

Thanks for the synopsis of your idea David, there are certainly some
advantages to using the submodule approach.

Is it github compatible?


Simon, yes git-submodule is compatible with github. I have a test
repository here - https://github.com/kicad/kicad-footprints - this contains
a submodule for each of the current footprint directories.

Does it simplify the maintenance of the kicad library by the librarians?


Compared to having all the .pretty folders as subdirectories of a single
repo, no. Reorganising libraries or adding new ones is tedious, and for
each of the 100+ .pretty repos, we (the librarians) have to manage:

* Pull requests and isues
* CI tools (e.g. Travis KLC scripts)
* Updating each library when new requirements are enacted (e.g. KLC)
* Ensure that changes in one repo do not conflict with a different repo
* Handle reclassification or moving of footprints between repos

It is also quite possible for submodules to become out of sync. The link
that Simon posted has some a good example of how this can occur (detached
head). I have some projects at work that contain about 5 submodules and it
rapidly becomes difficult to deal with even for a competent user of Git.

It becomes harder for users, too. KiCad users cannot, in the general case,
be classified as competent users of git. Currently, users who wish to
contribute to the footprint libraries have a hard time. For each library
they want to contribute to, they have to fork and manage another
repository. Personally I have done this for 50 separate repositories.

Further, once changes are made to each submodule, the master repository
then needs to be updated otherwise it's not up to date with the latest
changes. Perhaps we can use git hooks here, but this is just further
complication!

Let's also consider consistency. The footprint libraries are (currently)
split into separate repos. However the symbol libraries and 3d model
libraries are not. Should we split them too? Now we have 300+ repositories
to manage.

Let's not make this an academic problem and say "but it's possible to
manage with this complex set of tools"! From the perspective of a
contributor, and as the person who has to manage the libraries, the simpler
the better.

Oliver



On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers <simon.kueppers@xxxxxx>
wrote:

> Thanks for your detailed description. I think it is a nice way to go.
> However two remarks:
> Does it simplify the maintenance of the kicad library by the librarians?
> Is it github compatible? If not, we would need to find another platform to
> host the libraries.
>
> If I look for git submodule, there is a lot of different opinions on why
> you should or should not use it. (e.g. https://www.google.de/amp/s/
> codingkilledthecat.wordpress.com/2012/04/28/why-your-
> company-shouldnt-use-git-submodules/amp/). Does it apply in our case?
> What about git subtree?
>
> Best regards
> Simon
>
> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <
> info@xxxxxxxxxxx>:
>
>> Hi All
>>
>> I have to agree with some of the other posters, why not use git as it
>> was intended?
>>
>> Specifically I think the following makes sense.....
>>
>> - Keep the multiple repositories, this helps when a company only wants
>> to make certain sets of libs available to it's staff
>>
>> - Have a master repository that includes all other repositories as
>> submodules.
>>
>> - Have branches that are matched to kicad versions. This allows
>> footprint changes in a version safe way.
>>
>> - Use standard "git clone" (initial download) and "git pull" (update) on
>> the main repo which provides the entire set of available libs without
>> actually downloading the content for all libs.
>>
>> - When a specific lib is needed, do similar to what we do now, and use
>> "git submodule update --init --recursive $submoduleName" to just pull
>> that specific submodule
>>
>> - Allow the "main" library repo URI to be altered. This enables a
>> companies fork of the repo to be used.
>>
>> - Allow the "main" library repo URI to be ANY valid git URI. That means
>> a repo on a local fileserver rather than a http server can be used.
>> Along with various security options etc.
>>
>> - Add a fairly simple scripted tool that's run "on release" to retrieve
>> a README.md (an possibly a descriptor/index file) from each actual
>> library repo and update those within the "main" repo. This removes the
>> need to have the current case where manual edits are required to keep
>> the "main" repo in sync with what's available in the individual lib's.
>>
>> To add a new lib, it's then as simple as (from the main repo) doing
>> something like....
>> - "git submodule add $URI"
>> - "./scripts/update-library-indexes.sh"
>> - "git commit -a -m $'added new module "modulename"\nupdated all module
>> descriptions and index'"
>> - "git push"
>>
>>
>> All of the above allows the Main repo to be forked by a company (or
>> individual) and have their own custom repo's added very easily.
>> It even allows libraries to be easily excluded or replaced, all using
>> extremely well developed management tools.
>>
>> On a side note, git submodules are stored in the main repository
>> basically as URI that includes a commit reference.
>> That means it's easy to specify a specific library version to include in
>> a specific branch/tag of the main repo.
>>
>>
>> As for the person that said "git isn't available for windows", sorry,
>> that's bunkum.
>> There are many many development environments out there that ONLY run on
>> windows that have git integration.
>> A quick google search for "git for windows" will show what you need to
>> know there.
>>
>> Finally, wrapping git to duplicate the effective API that's currently in
>> use should be relatively trivial, resulting in virtually no code changes
>> required to KICAD it's self.
>> Assuming the existing functionality is cleanly wrapped by an API, the
>> only real change to KICAD would be the swapout of the module, and
>> addition of an easy way to change the URI
>>
>>
>> I know I haven't covered everything in this email, but it should be a
>> good outline for further discussion
>>
>>
>> On 22/09/17 09:21, Oliver Walters wrote:
>>
>>>  Hi all,
>>>
>>>  Ok, now that the website integration with the libraries is (pretty
>>>  much) done, and the licensing issue seems to be sorted, there is one
>>>  final puzzle piece to solve before I'm happy with the state of the
>>>  libraries for a v5 release.
>>>
>>>  *Goal: *Merge all footprint library repositories into a single repo to
>>>  solve the ongoing dramas of maintaining 100+ repos.
>>>
>>>  *Problem*: The *only* thing standing in the way of just doing this is
>>>  that some users like the GitHub plugin and previous instruction is
>>>  that this functionality cannot be removed.
>>>
>>>  The GitHub plugin functions by downloading a .zip file of a .pretty
>>>  repo. If we merge all footprint libs into a single repo with multiple
>>>  subdirectories, this will not work anymore (as GitHub dosen't allow
>>>  you to download a .zip of a single subdirectory).
>>>
>>>  Merging the repos is the *right thing to do*. But how to proceed?
>>>
>>>  *Options:*
>>>  *
>>>  *
>>>  /a) Drop github plugin feature, replace with library-download tool/
>>>
>>>  I don't think it is a good idea to live-load library data from GitHub
>>>  (a lot of other users agree too). It's slow, and a waste of bandwidth
>>>  to re-download the libs all the time.
>>>
>>>  We drop support for loading libraries direct from GitHub. However, we
>>>  add a tool for downloading libraries from GitHub and storing to disk.
>>>  Users can update as they like. This can be integrated in KiCad and new
>>>  users can run this tool when they first install KICad. This means that
>>>  no libs need to distributed with the installer and users can update to
>>>  latest libs whenever they want.
>>>
>>>  /b) Improve github plugin to allow subdirectory traversal/
>>>
>>>  This is difficult and will only result in the plugin being slower.
>>>  There are two ways I can see to do this:
>>>
>>>  i. Use Git API - tools exist that use this functionality -
>>>  https://github.com/KinoLien/gitzip
>>>  ii. Use subversion - GitHub actually provides subversion API -
>>>  https://www.seanw.org/blog/download-git-repo-subdirectory/
>>>
>>>
>>>  *Subversion*
>>>
>>>  In either case, I think that using the subversion tool to partially
>>>  download the libraries would be a good approach (I assume quicker than
>>>  using wget and the GitHub API).
>>>
>>>  1. Does anyone have any experience using the C API for SVN?
>>>  2. The Python bindings are pretty good -
>>>  https://pypi.python.org/pypi/svn - and much easier to use. However,
>>>  can we make the library download tools dependent on enabling the
>>>  python plugin?
>>>  3. Is there a way to checkout a subversion remote to memory (to
>>>  replicate the functionality of the current GitHub plugin)? If not, I'm
>>>  not sure how to approach option b) above.
>>>
>>>
>>>  Feedback appreciated. I think that it is very important especially for
>>>  new users that this is improved. The GitHub plugin is constantly
>>>  causing headaches!
>>>
>>>  Cheers,
>>>  Oliver
>>>
>>>
>>> ------------------------------
>>>
>>>  Mailing list: https://launchpad.net/~kicad-developers
>>>  Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>>  Unsubscribe : https://launchpad.net/~kicad-developers
>>>  More help   : https://help.launchpad.net/ListHelp
>>>
>> Regards
>> David Godfrey
>> SB Tech Services
>> mb: +61 437 286 200 <+61%20437%20286%20200> <tel:+61437286200 <+61%20437%20286%20200>>
>>
>> ------------------------------
>>
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>