kicad-developers team mailing list archive

[Diego Herranz <diegoherranz@xxxxxxxxxxxxxxxx>]

I completely understand Oliver's view on maintaining every individual repo
even if they are submodules of a kicad-footprint repo.

Even from an a user perspective (which is clearly easier than the
maintainer one), I like knowing what progress is being made, so I'd like to
subscribe for updates of a single repo on github, to follow commits, pull
requests, etc. There's no way I will subscribe to every repo and what's
most difficult, checking to make sure that I subscribe for new repos as
they are created too.

I have basically no say here, but I'd go for a single repo and no
submodules.

Thanks,
Diego

On Thu, Sep 28, 2017 at 6:53 PM, Bernhard Stegmaier <stegmaier@xxxxxxxxxxxxx
> wrote:

> Just my 2 cents… I have seen some projects using submodules (open source
> and in my company).
> I have seen lots of problems and complaints (… doesn’t compile …) just
> because normal users or even normal developers (being no git experts)
> didn’t sync all submodules correctly.
> Thinking of how many problems are already seen now by users which are not
> used to version control systems (because they just don’t understand what’s
> happening under the hood) I wouldn’t want to force them to use git
> submodules… at least not as long as this is not completely hidden by some
> users friendly and dead simple KiCad GUI.
>
>
> Regards,
> Bernhard
>
> On 28. Sep 2017, at 13:29, Oliver Walters <oliver.henry.walters@xxxxxxxxx>
> wrote:
>
> Simon, David,
>
> Thanks for the synopsis of your idea David, there are certainly some
> advantages to using the submodule approach.
>
> Is it github compatible?
>
>
> Simon, yes git-submodule is compatible with github. I have a test
> repository here - https://github.com/kicad/kicad-footprints - this
> contains a submodule for each of the current footprint directories.
>
> Does it simplify the maintenance of the kicad library by the librarians?
>
>
> Compared to having all the .pretty folders as subdirectories of a single
> repo, no. Reorganising libraries or adding new ones is tedious, and for
> each of the 100+ .pretty repos, we (the librarians) have to manage:
>
> * Pull requests and isues
> * CI tools (e.g. Travis KLC scripts)
> * Updating each library when new requirements are enacted (e.g. KLC)
> * Ensure that changes in one repo do not conflict with a different repo
> * Handle reclassification or moving of footprints between repos
>
> It is also quite possible for submodules to become out of sync. The link
> that Simon posted has some a good example of how this can occur (detached
> head). I have some projects at work that contain about 5 submodules and it
> rapidly becomes difficult to deal with even for a competent user of Git.
>
> It becomes harder for users, too. KiCad users cannot, in the general case,
> be classified as competent users of git. Currently, users who wish to
> contribute to the footprint libraries have a hard time. For each library
> they want to contribute to, they have to fork and manage another
> repository. Personally I have done this for 50 separate repositories.
>
> Further, once changes are made to each submodule, the master repository
> then needs to be updated otherwise it's not up to date with the latest
> changes. Perhaps we can use git hooks here, but this is just further
> complication!
>
> Let's also consider consistency. The footprint libraries are (currently)
> split into separate repos. However the symbol libraries and 3d model
> libraries are not. Should we split them too? Now we have 300+ repositories
> to manage.
>
> Let's not make this an academic problem and say "but it's possible to
> manage with this complex set of tools"! From the perspective of a
> contributor, and as the person who has to manage the libraries, the simpler
> the better.
>
> Oliver
>
>
>
> On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers <simon.kueppers@xxxxxx>
> wrote:
>
>> Thanks for your detailed description. I think it is a nice way to go.
>> However two remarks:
>> Does it simplify the maintenance of the kicad library by the librarians?
>> Is it github compatible? If not, we would need to find another platform to
>> host the libraries.
>>
>> If I look for git submodule, there is a lot of different opinions on why
>> you should or should not use it. (e.g. https://www.google.de/amp/s/co
>> dingkilledthecat.wordpress.com/2012/04/28/why-your-company-
>> shouldnt-use-git-submodules/amp/). Does it apply in our case? What about
>> git subtree?
>>
>> Best regards
>> Simon
>>
>> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <
>> info@xxxxxxxxxxx>:
>>
>>> Hi All
>>>
>>> I have to agree with some of the other posters, why not use git as it
>>> was intended?
>>>
>>> Specifically I think the following makes sense.....
>>>
>>> - Keep the multiple repositories, this helps when a company only wants
>>> to make certain sets of libs available to it's staff
>>>
>>> - Have a master repository that includes all other repositories as
>>> submodules.
>>>
>>> - Have branches that are matched to kicad versions. This allows
>>> footprint changes in a version safe way.
>>>
>>> - Use standard "git clone" (initial download) and "git pull" (update) on
>>> the main repo which provides the entire set of available libs without
>>> actually downloading the content for all libs.
>>>
>>> - When a specific lib is needed, do similar to what we do now, and use
>>> "git submodule update --init --recursive $submoduleName" to just pull
>>> that specific submodule
>>>
>>> - Allow the "main" library repo URI to be altered. This enables a
>>> companies fork of the repo to be used.
>>>
>>> - Allow the "main" library repo URI to be ANY valid git URI. That means
>>> a repo on a local fileserver rather than a http server can be used.
>>> Along with various security options etc.
>>>
>>> - Add a fairly simple scripted tool that's run "on release" to retrieve
>>> a README.md <http://readme.md/> (an possibly a descriptor/index file) from each actual
>>> library repo and update those within the "main" repo. This removes the
>>> need to have the current case where manual edits are required to keep
>>> the "main" repo in sync with what's available in the individual lib's.
>>>
>>> To add a new lib, it's then as simple as (from the main repo) doing
>>> something like....
>>> - "git submodule add $URI"
>>> - "./scripts/update-library-indexes.sh"
>>> - "git commit -a -m $'added new module "modulename"\nupdated all module
>>> descriptions and index'"
>>> - "git push"
>>>
>>>
>>> All of the above allows the Main repo to be forked by a company (or
>>> individual) and have their own custom repo's added very easily.
>>> It even allows libraries to be easily excluded or replaced, all using
>>> extremely well developed management tools.
>>>
>>> On a side note, git submodules are stored in the main repository
>>> basically as URI that includes a commit reference.
>>> That means it's easy to specify a specific library version to include in
>>> a specific branch/tag of the main repo.
>>>
>>>
>>> As for the person that said "git isn't available for windows", sorry,
>>> that's bunkum.
>>> There are many many development environments out there that ONLY run on
>>> windows that have git integration.
>>> A quick google search for "git for windows" will show what you need to
>>> know there.
>>>
>>> Finally, wrapping git to duplicate the effective API that's currently in
>>> use should be relatively trivial, resulting in virtually no code changes
>>> required to KICAD it's self.
>>> Assuming the existing functionality is cleanly wrapped by an API, the
>>> only real change to KICAD would be the swapout of the module, and
>>> addition of an easy way to change the URI
>>>
>>>
>>> I know I haven't covered everything in this email, but it should be a
>>> good outline for further discussion
>>>
>>>
>>> On 22/09/17 09:21, Oliver Walters wrote:
>>>
>>>>  Hi all,
>>>>
>>>>  Ok, now that the website integration with the libraries is (pretty
>>>>  much) done, and the licensing issue seems to be sorted, there is one
>>>>  final puzzle piece to solve before I'm happy with the state of the
>>>>  libraries for a v5 release.
>>>>
>>>>  *Goal: *Merge all footprint library repositories into a single repo to
>>>>  solve the ongoing dramas of maintaining 100+ repos.
>>>>
>>>>  *Problem*: The *only* thing standing in the way of just doing this is
>>>>  that some users like the GitHub plugin and previous instruction is
>>>>  that this functionality cannot be removed.
>>>>
>>>>  The GitHub plugin functions by downloading a .zip file of a .pretty
>>>>  repo. If we merge all footprint libs into a single repo with multiple
>>>>  subdirectories, this will not work anymore (as GitHub dosen't allow
>>>>  you to download a .zip of a single subdirectory).
>>>>
>>>>  Merging the repos is the *right thing to do*. But how to proceed?
>>>>
>>>>  *Options:*
>>>>  *
>>>>  *
>>>>  /a) Drop github plugin feature, replace with library-download tool/
>>>>
>>>>  I don't think it is a good idea to live-load library data from GitHub
>>>>  (a lot of other users agree too). It's slow, and a waste of bandwidth
>>>>  to re-download the libs all the time.
>>>>
>>>>  We drop support for loading libraries direct from GitHub. However, we
>>>>  add a tool for downloading libraries from GitHub and storing to disk.
>>>>  Users can update as they like. This can be integrated in KiCad and new
>>>>  users can run this tool when they first install KICad. This means that
>>>>  no libs need to distributed with the installer and users can update to
>>>>  latest libs whenever they want.
>>>>
>>>>  /b) Improve github plugin to allow subdirectory traversal/
>>>>
>>>>  This is difficult and will only result in the plugin being slower.
>>>>  There are two ways I can see to do this:
>>>>
>>>>  i. Use Git API - tools exist that use this functionality -
>>>>  https://github.com/KinoLien/gitzip
>>>>  ii. Use subversion - GitHub actually provides subversion API -
>>>>  https://www.seanw.org/blog/download-git-repo-subdirectory/
>>>>
>>>>
>>>>  *Subversion*
>>>>
>>>>  In either case, I think that using the subversion tool to partially
>>>>  download the libraries would be a good approach (I assume quicker than
>>>>  using wget and the GitHub API).
>>>>
>>>>  1. Does anyone have any experience using the C API for SVN?
>>>>  2. The Python bindings are pretty good -
>>>>  https://pypi.python.org/pypi/svn - and much easier to use. However,
>>>>  can we make the library download tools dependent on enabling the
>>>>  python plugin?
>>>>  3. Is there a way to checkout a subversion remote to memory (to
>>>>  replicate the functionality of the current GitHub plugin)? If not, I'm
>>>>  not sure how to approach option b) above.
>>>>
>>>>
>>>>  Feedback appreciated. I think that it is very important especially for
>>>>  new users that this is improved. The GitHub plugin is constantly
>>>>  causing headaches!
>>>>
>>>>  Cheers,
>>>>  Oliver
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>>  Mailing list: https://launchpad.net/~kicad-developers
>>>>  Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>>>  Unsubscribe : https://launchpad.net/~kicad-developers
>>>>  More help   : https://help.launchpad.net/ListHelp
>>>>
>>> Regards
>>> David Godfrey
>>> SB Tech Services
>>> mb: +61 437 286 200 <+61%20437%20286%20200> <tel:+61437286200 <+61%20437%20286%20200>>
>>>
>>> ------------------------------
>>>
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>