kicad-developers team mailing list archive

[Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx>]

Hi folks!

Good news.

On my dev machine, I've got the DMG, the apps, all the frameworks,
everything, all notarized.  The apps are all signed even!

We don't need to change the symlinks! (I'm a little bummed that the
relayed question->answer from WWDC wasn't accurate, but that was 3
years ago, and there have been changes to how notarization is enforced
as recently as 2020/02/03. :) )

What was it?  A combination of two things:

* Moving some Python and ngspice things around in the bundle so that
only certain types of executables were in certain directories of the
bundle
* Apple says to submit a zip to their notarization service.  If you
create a zip with "zip", it won't work. You need to use ditto with a
few different arguments.  I suspect that this was messing with the
symlinks.  *sigh*

Alright, so what's next?

1) I want to remove all the KiCad patches from kicad-mac-builder,
long-term.  As part of that, there's a few changes from this effort
that I want to get into the KiCad cmake files.  I think they're
innocent and gentle but we can discuss them on the MRs I'll make. :)
However, I have an old patch to create a distributable Python
Framework that I don't really want to merge into our tree--and I am
personally OK with leaving it as a patch in kicad-mac-builder, since
a) it has been that way since I created it years ago and 2) my next
effort after this notarization/signing is 100% done is getting Python
3 working on the macOS builds, and as part of that I'll be getting rid
of this patch.  At that point kicad-mac-builder won't have any KiCad
patches at all.

2) It is certainly possible that I broke things or made minor mistakes
with signing/notarization, so I will generate some of these builds on
my dev machine and get them available as testing builds for people to
test.

3) It is possible there are ways that users use the apps now that will
break.  For instance, it will no longer be OK for users to put their
own files inside the bundle.  This is a good thing, but we may need to
prepare some instructions or something to help break it to users--if
anyone has strong opinions here please let me know.  I am terrified
that users are doing this, downloading a new version of KiCad,
overwriting their old "executables" and then losing their data files.
I have read of at least one report of this happening on kicad.info and
it breaks my heart.   I know there has been some confusion about this
point technically but see
https://developer.apple.com/library/archive/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG304
for details.

4) I have some kicad-mac-builder changes to make to get all of this
signing and notarization stuff working on CI.  I expect this will take
at least a few days as the part I need to modify the most is a part
that's been a pain lately and needs to be gutted anyway...

5) When all of this is done, and this is stable, Python 3 on macOS is
next on my plate.

Thanks folks!

If anyone has any questions or comments or opinions on this stuff, let
me know, otherwise I'll keep at it :)

Adam Wolf

On Mon, Feb 3, 2020 at 10:05 AM Bernhard Stegmaier
<stegmaier@xxxxxxxxxxxxx> wrote:
>
> Adam,
>
> One last thing, just to be sure… you are notarizing the final kicad.app after the “make install”?
> If you use the (intermediate) one of the build directory, the sub-bundles might not yet be in place as they should…
>
>
> Bernhard
>
> > On 3. Feb 2020, at 16:59, Bernhard Stegmaier <stegmaier@xxxxxxxxxxxxx> wrote:
> >
> > OK, I see… this is weird.
> > My understanding from what I read in the documentation was, that it should sign everything in the top-level bundle (kicad.app) including sub-apps you might have in kicad.app/Contents/Applications.
> >
> >
> > Regards,
> > Bernhard
> >
> >> On 3. Feb 2020, at 16:47, Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx> wrote:
> >>
> >> I am not notarizing the DMGs.  While this is possible, it has not been
> >> necessary so far.
> >>
> >> When I tried notarizing kicad.app but not the others, when I move to a
> >> new computer, it complains that eeschema.app is not notarized.
> >>
> >> The problem is not putting the signed kicad.app into an unsigned dmg.
> >>
> >> On Mon, Feb 3, 2020 at 8:11 AM Bernhard Stegmaier
> >> <stegmaier@xxxxxxxxxxxxx> wrote:
> >>>
> >>> Hi Adam,
> >>>
> >>> I am also no fan of the symlinks, but having a different approach will
> >>> be probably some work.
> >>>
> >>>> I had someone ask if what we do would work during WWDC and I was told
> >>>> it would not work.  I consistently get "the signature is invalid" when
> >>>> signing while we have symlinks, and when I remove the symlinks and
> >>>> just sign KiCad.app this error goes away.
> >>>
> >>> I don't doubt that the symlinks in the DMG don't work.
> >>> What you explained is exactly what I had in mind:
> >>> (1) Sign *only* kicad.app as is. No complete DMG with symlinks or
> >>> whatever.
> >>> (2) Create DMG with previously signed kicad.app and symlinks, libraries
> >>> and whatever you put into. Don't try to notarize this DMG, DMG is just
> >>> a container.
> >>>
> >>> Doesn't that work?
> >>> kicad.app is signed and the DMG should just acts as some kind of zip
> >>> file then... ?
> >>>
> >>> If the problem is putting the signed kicad.app into a (unsigned) DMG,
> >>> maybe just distributing via .zip would be also a viable way meanwhile?
> >>> Many other applications also do this...
> >>>
> >>>
> >>> Regards,
> >>> Bernhard
> >>>
> >>> Am 3.2.2020 14:46, schrieb Adam Wolf:
> >>>> Bernhard,
> >>>>
> >>>> I have no personal vendetta against the symlinks.
> >>>>
> >>>> I had someone ask if what we do would work during WWDC and I was told
> >>>> it would not work.  I consistently get "the signature is invalid" when
> >>>> signing while we have symlinks, and when I remove the symlinks and
> >>>> just sign KiCad.app this error goes away.
> >>>>
> >>>> I am not sure if Apple gives themselves special entitlements that mere
> >>>> mortals don't get.  I'm not sure if I'm just not able to get it to
> >>>> work.
> >>>>
> >>>> Nothing I have done so far relies on the symlinks going away, so if
> >>>> you think you can make it work, please let me know.
> >>>>
> >>>> My personal suggestion for working around the symlinks issue was not
> >>>> to just copy things, but rather just have a single KiCad.app that
> >>>> would open itself in different ways of given a different type of file,
> >>>> but others on the bug tracker preferred trying to copy things first.
> >>>>
> >>>> Frankly, it's exhausting spending all this time on things that users
> >>>> don't see, when there are so many interesting fun things we could be
> >>>> working on instead.
> >>>>
> >>>> In terms of what I am signing and notarizing, I have tried signing and
> >>>> notarizing the app, the dmg, all the apps, basically every
> >>>> combination.  Apple's rules are extremely fickle here, and you could
> >>>> even notarize unsigned things.  They explicitly say the rules about
> >>>> what you can notarize are hidden from developers!
> >>>>
> >>>> Adam
> >>>>
> >>>> On Mon, Feb 3, 2020, 1:08 AM Bernhard Stegmaier
> >>>> <stegmaier@xxxxxxxxxxxxx> wrote:
> >>>>
> >>>>> Hi Adam,
> >>>>>
> >>>>> I still don’t get it:
> >>>>>> Our current
> >>>>>> strategy of symlinking into the kicad.app bundle does not work
> >>>>> with
> >>>>>> macOS signing.
> >>>>>
> >>>>> Xcode has e.g. Instruments application in
> >>>>> Xcode.app/Contents/Applications/Instruments.app
> >>>>> If I symlink it (for example) to
> >>>>> /Applications/Instruments.app
> >>>>> It runs without any complaints when started via the symlink.
> >>>>>
> >>>>> What do you notarize?
> >>>>> The overall dmg with the symlink?
> >>>>> Have you already tried to only notarize kicad.app (no dmg, no
> >>>>> symlinks) and put it into the dmg with symlinks afterwards?
> >>>>> Another quick fix could be some script that can be run to create the
> >>>>> symlinks on user machine?
> >>>>>
> >>>>> A simple copy of the apps won’t work.
> >>>>> You need to change everything wrt shared libraries in KiCad code and
> >>>>> cmake script.
> >>>>>
> >>>>> In the end, you will duplicate all libraries and support stuff.
> >>>>> Probably not a big deal for eeschema and the other small apps, but I
> >>>>> guess for pcbnew.
> >>>>> Means duplicating all the python, nags-ice, etc. stuff.
> >>>>> And also, all stuff like templates, scripts, etc.
> >>>>> Users shouldn’t fiddle around in the .app, but could get really
> >>>>> messy if they now put (template, python, spice?) stuff in kicad.app
> >>>>> or pcbnew.app and then something doesn’t work in one or the
> >>>>> other...
> >>>>>
> >>>>> Regards,
> >>>>> Bernhard
> >>>>>
> >>>>>> On 3. Feb 2020, at 02:00, Adam Wolf
> >>>>> <adamwolf@xxxxxxxxxxxxxxxxxxxx> wrote:
> >>>>>>
> >>>>>> Hi folks!
> >>>>>>
> >>>>>> Apple is changing how the lack of notarization looks like to users
> >>>>> on
> >>>>>> Catalina starting tomorrow.  It is not clear what will happen when
> >>>>>> folks download new versions of KiCad after tonight.
> >>>>>>
> >>>>>> For the past two months I've been working hard--I've got a tech
> >>>>> demo
> >>>>>> locally here that has signatures and notarization on macOS, but
> >>>>> it's
> >>>>>> not ready for primetime.  For instance, I have removed the other
> >>>>> .apps
> >>>>>> and just have kicad.app.  There's changes I made to kicad that
> >>>>>> probably belong in kicad-mac-builder--and, well, let's just say
> >>>>> it's a
> >>>>>> tech demo :)
> >>>>>>
> >>>>>> The main things that remain are:
> >>>>>> 1) Figure out a good solution for the symlinked .apps.  Our
> >>>>> current
> >>>>>> strategy of symlinking into the kicad.app bundle does not work
> >>>>> with
> >>>>>> macOS signing.  I think the current contender is to copy instead
> >>>>> of
> >>>>>> symlink.  I am not sure how much extra space that will take up but
> >>>>>> it's a good try.  This is definitely something I can do, but since
> >>>>>> it's something that can be done on its own, it's a prime contender
> >>>>> for
> >>>>>> someone looking to help out.
> >>>>>>
> >>>>>> 2) Another issue is that there are strict rules about where in the
> >>>>>> bundle code, data, and executable non-Mach-O files live.  For
> >>>>>> instance, one of the signing blockers is ngspice, because it
> >>>>> mingles
> >>>>>> scripts and Mach-O binaries and then we put them in
> >>>>> Contents/Plugins.
> >>>>>> For more details, see
> >>>>>>
> >>>>>
> >>>> https://developer.apple.com/library/archive/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG201.
> >>>>>> The big change for KiCad itself is where the Python scripts are
> >>>>>> stored--I've fixed this in my branch, but now I have to go through
> >>>>> and
> >>>>>> audit and fixup our partner packages, like OCE/OCC and ngspice.
> >>>>> If
> >>>>>> you want to help with this, it's going to be a big job but I'm
> >>>>> willing
> >>>>>> to put in the time to teach if you're willing to put in the time
> >>>>> to
> >>>>>> learn :)
> >>>>>>
> >>>>>> I was really hoping I could get this done before Apple turned up
> >>>>> the
> >>>>>> enforcement on notarization, but that's going to happen.  After
> >>>>>> tomorrow, it'll be clearer what Apple is doing.  There might be
> >>>>> some
> >>>>>> quick changes to make that will improve things for our users
> >>>>> without
> >>>>>> getting all of this done.
> >>>>>>
> >>>>>> Adam Wolf
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Mailing list: https://launchpad.net/~kicad-developers
> >>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>>>>> Unsubscribe : https://launchpad.net/~kicad-developers
> >>>>>> More help   : https://help.launchpad.net/ListHelp
> >
>