kicad-developers team mailing list archive

[Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx>]

My first email to the list about my signing efforts are from early 2017!

On Tue, Feb 4, 2020 at 4:12 PM Jeff Young <jeff@xxxxxxxxx> wrote:
>
> Awesome!  Great job, Adam.
>
>
> > On 4 Feb 2020, at 22:03, Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > Hi folks!
> >
> > Good news.
> >
> > On my dev machine, I've got the DMG, the apps, all the frameworks,
> > everything, all notarized.  The apps are all signed even!
> >
> > We don't need to change the symlinks! (I'm a little bummed that the
> > relayed question->answer from WWDC wasn't accurate, but that was 3
> > years ago, and there have been changes to how notarization is enforced
> > as recently as 2020/02/03. :) )
> >
> > What was it?  A combination of two things:
> >
> > * Moving some Python and ngspice things around in the bundle so that
> > only certain types of executables were in certain directories of the
> > bundle
> > * Apple says to submit a zip to their notarization service.  If you
> > create a zip with "zip", it won't work. You need to use ditto with a
> > few different arguments.  I suspect that this was messing with the
> > symlinks.  *sigh*
> >
> > Alright, so what's next?
> >
> > 1) I want to remove all the KiCad patches from kicad-mac-builder,
> > long-term.  As part of that, there's a few changes from this effort
> > that I want to get into the KiCad cmake files.  I think they're
> > innocent and gentle but we can discuss them on the MRs I'll make. :)
> > However, I have an old patch to create a distributable Python
> > Framework that I don't really want to merge into our tree--and I am
> > personally OK with leaving it as a patch in kicad-mac-builder, since
> > a) it has been that way since I created it years ago and 2) my next
> > effort after this notarization/signing is 100% done is getting Python
> > 3 working on the macOS builds, and as part of that I'll be getting rid
> > of this patch.  At that point kicad-mac-builder won't have any KiCad
> > patches at all.
> >
> > 2) It is certainly possible that I broke things or made minor mistakes
> > with signing/notarization, so I will generate some of these builds on
> > my dev machine and get them available as testing builds for people to
> > test.
> >
> > 3) It is possible there are ways that users use the apps now that will
> > break.  For instance, it will no longer be OK for users to put their
> > own files inside the bundle.  This is a good thing, but we may need to
> > prepare some instructions or something to help break it to users--if
> > anyone has strong opinions here please let me know.  I am terrified
> > that users are doing this, downloading a new version of KiCad,
> > overwriting their old "executables" and then losing their data files.
> > I have read of at least one report of this happening on kicad.info and
> > it breaks my heart.   I know there has been some confusion about this
> > point technically but see
> > https://developer.apple.com/library/archive/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG304
> > for details.
> >
> > 4) I have some kicad-mac-builder changes to make to get all of this
> > signing and notarization stuff working on CI.  I expect this will take
> > at least a few days as the part I need to modify the most is a part
> > that's been a pain lately and needs to be gutted anyway...
> >
> > 5) When all of this is done, and this is stable, Python 3 on macOS is
> > next on my plate.
> >
> > Thanks folks!
> >
> > If anyone has any questions or comments or opinions on this stuff, let
> > me know, otherwise I'll keep at it :)
> >
> > Adam Wolf
> >
> > On Mon, Feb 3, 2020 at 10:05 AM Bernhard Stegmaier
> > <stegmaier@xxxxxxxxxxxxx> wrote:
> >>
> >> Adam,
> >>
> >> One last thing, just to be sure… you are notarizing the final kicad.app after the “make install”?
> >> If you use the (intermediate) one of the build directory, the sub-bundles might not yet be in place as they should…
> >>
> >>
> >> Bernhard
> >>
> >>> On 3. Feb 2020, at 16:59, Bernhard Stegmaier <stegmaier@xxxxxxxxxxxxx> wrote:
> >>>
> >>> OK, I see… this is weird.
> >>> My understanding from what I read in the documentation was, that it should sign everything in the top-level bundle (kicad.app) including sub-apps you might have in kicad.app/Contents/Applications.
> >>>
> >>>
> >>> Regards,
> >>> Bernhard
> >>>
> >>>> On 3. Feb 2020, at 16:47, Adam Wolf <adamwolf@xxxxxxxxxxxxxxxxxxxx> wrote:
> >>>>
> >>>> I am not notarizing the DMGs.  While this is possible, it has not been
> >>>> necessary so far.
> >>>>
> >>>> When I tried notarizing kicad.app but not the others, when I move to a
> >>>> new computer, it complains that eeschema.app is not notarized.
> >>>>
> >>>> The problem is not putting the signed kicad.app into an unsigned dmg.
> >>>>
> >>>> On Mon, Feb 3, 2020 at 8:11 AM Bernhard Stegmaier
> >>>> <stegmaier@xxxxxxxxxxxxx> wrote:
> >>>>>
> >>>>> Hi Adam,
> >>>>>
> >>>>> I am also no fan of the symlinks, but having a different approach will
> >>>>> be probably some work.
> >>>>>
> >>>>>> I had someone ask if what we do would work during WWDC and I was told
> >>>>>> it would not work.  I consistently get "the signature is invalid" when
> >>>>>> signing while we have symlinks, and when I remove the symlinks and
> >>>>>> just sign KiCad.app this error goes away.
> >>>>>
> >>>>> I don't doubt that the symlinks in the DMG don't work.
> >>>>> What you explained is exactly what I had in mind:
> >>>>> (1) Sign *only* kicad.app as is. No complete DMG with symlinks or
> >>>>> whatever.
> >>>>> (2) Create DMG with previously signed kicad.app and symlinks, libraries
> >>>>> and whatever you put into. Don't try to notarize this DMG, DMG is just
> >>>>> a container.
> >>>>>
> >>>>> Doesn't that work?
> >>>>> kicad.app is signed and the DMG should just acts as some kind of zip
> >>>>> file then... ?
> >>>>>
> >>>>> If the problem is putting the signed kicad.app into a (unsigned) DMG,
> >>>>> maybe just distributing via .zip would be also a viable way meanwhile?
> >>>>> Many other applications also do this...
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Bernhard
> >>>>>
> >>>>> Am 3.2.2020 14:46, schrieb Adam Wolf:
> >>>>>> Bernhard,
> >>>>>>
> >>>>>> I have no personal vendetta against the symlinks.
> >>>>>>
> >>>>>> I had someone ask if what we do would work during WWDC and I was told
> >>>>>> it would not work.  I consistently get "the signature is invalid" when
> >>>>>> signing while we have symlinks, and when I remove the symlinks and
> >>>>>> just sign KiCad.app this error goes away.
> >>>>>>
> >>>>>> I am not sure if Apple gives themselves special entitlements that mere
> >>>>>> mortals don't get.  I'm not sure if I'm just not able to get it to
> >>>>>> work.
> >>>>>>
> >>>>>> Nothing I have done so far relies on the symlinks going away, so if
> >>>>>> you think you can make it work, please let me know.
> >>>>>>
> >>>>>> My personal suggestion for working around the symlinks issue was not
> >>>>>> to just copy things, but rather just have a single KiCad.app that
> >>>>>> would open itself in different ways of given a different type of file,
> >>>>>> but others on the bug tracker preferred trying to copy things first.
> >>>>>>
> >>>>>> Frankly, it's exhausting spending all this time on things that users
> >>>>>> don't see, when there are so many interesting fun things we could be
> >>>>>> working on instead.
> >>>>>>
> >>>>>> In terms of what I am signing and notarizing, I have tried signing and
> >>>>>> notarizing the app, the dmg, all the apps, basically every
> >>>>>> combination.  Apple's rules are extremely fickle here, and you could
> >>>>>> even notarize unsigned things.  They explicitly say the rules about
> >>>>>> what you can notarize are hidden from developers!
> >>>>>>
> >>>>>> Adam
> >>>>>>
> >>>>>> On Mon, Feb 3, 2020, 1:08 AM Bernhard Stegmaier
> >>>>>> <stegmaier@xxxxxxxxxxxxx> wrote:
> >>>>>>
> >>>>>>> Hi Adam,
> >>>>>>>
> >>>>>>> I still don’t get it:
> >>>>>>>> Our current
> >>>>>>>> strategy of symlinking into the kicad.app bundle does not work
> >>>>>>> with
> >>>>>>>> macOS signing.
> >>>>>>>
> >>>>>>> Xcode has e.g. Instruments application in
> >>>>>>> Xcode.app/Contents/Applications/Instruments.app
> >>>>>>> If I symlink it (for example) to
> >>>>>>> /Applications/Instruments.app
> >>>>>>> It runs without any complaints when started via the symlink.
> >>>>>>>
> >>>>>>> What do you notarize?
> >>>>>>> The overall dmg with the symlink?
> >>>>>>> Have you already tried to only notarize kicad.app (no dmg, no
> >>>>>>> symlinks) and put it into the dmg with symlinks afterwards?
> >>>>>>> Another quick fix could be some script that can be run to create the
> >>>>>>> symlinks on user machine?
> >>>>>>>
> >>>>>>> A simple copy of the apps won’t work.
> >>>>>>> You need to change everything wrt shared libraries in KiCad code and
> >>>>>>> cmake script.
> >>>>>>>
> >>>>>>> In the end, you will duplicate all libraries and support stuff.
> >>>>>>> Probably not a big deal for eeschema and the other small apps, but I
> >>>>>>> guess for pcbnew.
> >>>>>>> Means duplicating all the python, nags-ice, etc. stuff.
> >>>>>>> And also, all stuff like templates, scripts, etc.
> >>>>>>> Users shouldn’t fiddle around in the .app, but could get really
> >>>>>>> messy if they now put (template, python, spice?) stuff in kicad.app
> >>>>>>> or pcbnew.app and then something doesn’t work in one or the
> >>>>>>> other...
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Bernhard
> >>>>>>>
> >>>>>>>> On 3. Feb 2020, at 02:00, Adam Wolf
> >>>>>>> <adamwolf@xxxxxxxxxxxxxxxxxxxx> wrote:
> >>>>>>>>
> >>>>>>>> Hi folks!
> >>>>>>>>
> >>>>>>>> Apple is changing how the lack of notarization looks like to users
> >>>>>>> on
> >>>>>>>> Catalina starting tomorrow.  It is not clear what will happen when
> >>>>>>>> folks download new versions of KiCad after tonight.
> >>>>>>>>
> >>>>>>>> For the past two months I've been working hard--I've got a tech
> >>>>>>> demo
> >>>>>>>> locally here that has signatures and notarization on macOS, but
> >>>>>>> it's
> >>>>>>>> not ready for primetime.  For instance, I have removed the other
> >>>>>>> .apps
> >>>>>>>> and just have kicad.app.  There's changes I made to kicad that
> >>>>>>>> probably belong in kicad-mac-builder--and, well, let's just say
> >>>>>>> it's a
> >>>>>>>> tech demo :)
> >>>>>>>>
> >>>>>>>> The main things that remain are:
> >>>>>>>> 1) Figure out a good solution for the symlinked .apps.  Our
> >>>>>>> current
> >>>>>>>> strategy of symlinking into the kicad.app bundle does not work
> >>>>>>> with
> >>>>>>>> macOS signing.  I think the current contender is to copy instead
> >>>>>>> of
> >>>>>>>> symlink.  I am not sure how much extra space that will take up but
> >>>>>>>> it's a good try.  This is definitely something I can do, but since
> >>>>>>>> it's something that can be done on its own, it's a prime contender
> >>>>>>> for
> >>>>>>>> someone looking to help out.
> >>>>>>>>
> >>>>>>>> 2) Another issue is that there are strict rules about where in the
> >>>>>>>> bundle code, data, and executable non-Mach-O files live.  For
> >>>>>>>> instance, one of the signing blockers is ngspice, because it
> >>>>>>> mingles
> >>>>>>>> scripts and Mach-O binaries and then we put them in
> >>>>>>> Contents/Plugins.
> >>>>>>>> For more details, see
> >>>>>>>>
> >>>>>>>
> >>>>>> https://developer.apple.com/library/archive/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG201.
> >>>>>>>> The big change for KiCad itself is where the Python scripts are
> >>>>>>>> stored--I've fixed this in my branch, but now I have to go through
> >>>>>>> and
> >>>>>>>> audit and fixup our partner packages, like OCE/OCC and ngspice.
> >>>>>>> If
> >>>>>>>> you want to help with this, it's going to be a big job but I'm
> >>>>>>> willing
> >>>>>>>> to put in the time to teach if you're willing to put in the time
> >>>>>>> to
> >>>>>>>> learn :)
> >>>>>>>>
> >>>>>>>> I was really hoping I could get this done before Apple turned up
> >>>>>>> the
> >>>>>>>> enforcement on notarization, but that's going to happen.  After
> >>>>>>>> tomorrow, it'll be clearer what Apple is doing.  There might be
> >>>>>>> some
> >>>>>>>> quick changes to make that will improve things for our users
> >>>>>>> without
> >>>>>>>> getting all of this done.
> >>>>>>>>
> >>>>>>>> Adam Wolf
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Mailing list: https://launchpad.net/~kicad-developers
> >>>>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> >>>>>>>> Unsubscribe : https://launchpad.net/~kicad-developers
> >>>>>>>> More help   : https://help.launchpad.net/ListHelp
> >>>
> >>
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~kicad-developers
> > Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> > Unsubscribe : https://launchpad.net/~kicad-developers
> > More help   : https://help.launchpad.net/ListHelp
>