Shipped binaries are missing some codesigning on Windows & MacOS - Causing slow first launch

[davidanthoff]

There is a delay of more than 14 seconds when I press ] to enter package mode on a new install. Here is a video that shows the timing: https://www.youtube.com/watch?v=a7PU7C1rMvw.

One strange thing is that on subsequent attempts (even after a boot of the computer) the delay is almost entirely gone. Not entirely straightforward on how to reproduce this, but uninstall 1.11 via juliaup rm 1.11 and then booting the computer mostly gets my system back into a state where the delay is present. The system where this is happening is a new desktop computer with essentially top specs for everything.
 
We (@KristofferC, @IanButterworth and I) on Slack where one hypothesis is that this is somehow related to Windows Defender scanning files on a first load, caching the results of that somehow... If that is true, maybe #54365 might help. While I'm not sure, Windows Defender might take less time to check files that have proper digital signatures on them.

[KristofferC]

Someone that can repro this could probably get it in a profiler and see if it is waiting on some syscall or something.

[KristofferC]

Also, is it really entering package mode that is slow or is it using Pkg?

[KristofferC]

I think this is too vague to be on the milestone. Also, on the video it seem to take 25 seconds to even start julia so I would not say package mode is the problematic thing here.

[IanButterworth]

On MacOS I'm noticing a lot more delay in first Pkg switch and first completion hint in pkg on nightlies via juliaup vs. julia built locally.

First pkg switch was like 4s via juliaup, ~0.3s via local build
In a new session after that they're about the same at ~0.3s

So I guess MacOS also does some system check on dylibs on first execution too.

@davidanthoff I think it might make sense for juliaup to handle this kind of first execution system check. So launch julia and load all stdlibs after installation?

---

@KristofferC pointed to this which could be the MacOS issue https://news.ycombinator.com/item?id=23273247
Perhaps a solution would be what's discussed here? https://news.ycombinator.com/item?id=23273396

[davidanthoff]

Are we stapling that notarization stuff to the files in the tarball that Juliaup is using for MacOS?

My gut sense right now is that we first should just make sure that the archives that Juliaup is using have all the code signing/notarization that can be done, then check whether we still have performance problems, and if so, we can still think about more stuff that Juliaup could do.

[IanButterworth]

Side note, I noticed that on MacOS if you have a slow but active internet connection then precompilation, including building stdlibs, is really slow because MacOS is trying and failing to do the stuff mentioned here https://news.ycombinator.com/item?id=23273247

[IanButterworth]

I've adjusted the title to cover the issue that I think we should fix before 1.11 is released. First impressions matter and on the latest 1.11 rc1 via juliaup on MacOS the Pkg switch is really slow and it sounds like it can be a lot worse on Windows

[IanButterworth]

On MacOS it seems the issue is that we don't notarize the .tar.gz archive which Juliaup uses, just the .dmg.

If I install via the .dmg I see no delay in startup nor first Pkg switch.

With the .tar.gz I see a slightly slow initial startup, then ~5-10s for first pkg switch.

https://github.com/JuliaLang/julia-buildkite/blob/dea1b33fba992d460ffae69bce54f598330fb0d3/utilities/upload_julia.sh#L45-L53

then

https://github.com/JuliaLang/julia-buildkite/blob/dea1b33fba992d460ffae69bce54f598330fb0d3/utilities/macos/build_dmg.sh#L61-L69

@davidanthoff could it be something similar for Windows?