bpo-41100: Support macOS 11 and Apple Silicon #22855

ronaldoussoren · 2020-10-21T11:23:15Z

We should consider just dropping support for macOS 10.4 here to simplify the code.

This is support for ctypes on macOS/arm64 based on PR 21249 by Lawrence D'Anna (Apple). Changes: - changed __builtin_available tests from 11.0 to 10.15 - added test to setup.py for ffi_closure_alloc and use that in malloc_closure.c - Minor change in the code path for ffi_prep_closure_var (coding style change)

The preprocessor guard in the old version doesn't work, and isn't really needed (10.4 only supported 32-bit code where unsigned long is the same as uint32_t).

… machine

This also adds an option to stop building after compiling the 3th-party dependencies, as well as a script for archiving those dependencies. This makes it easier to work on the build. There are three changes to build-installer.py related to universal2 support: 1. Add 'universal2' information to build-installer.py; 2. Building OpenSSL for arm64 requires a patch at this time; 3. For some reason I had to patch the Tcl build to avoid a build error.

Needed because my previous workaround doesn't work anymore. This uses a private API, that should be made public later...

…anch

- __builtin_available is not present in the compiler, fall back to the ancient way of testing if a symbol is usable. - Logic error code in pytime.c resulted in a compile error.

…oren/cpython into macos11-deploy-earlier-branch

The tests are not entirely done yet, and something similar needs to be done for the time module.

…on isn't present. I don't particularly like the code duplication, but this code should be pretty stable.

ned-deily · 2020-11-02T06:33:48Z

I used the current state of the PR to build and test some installer variants. Due to time constraints and my currently degraded test environment, I focused first on regressions with currently supported systems. The configurations tested were:

installer built on 10.9 (the current standard python.org variant)
installer built on 10.15 with a deployment target of 10.9 (to exercise the weak-linking code)

I tested the installer 1 on 11.0 developer preview 9. The only unexpected failure was of the modified test_ctypes:

test test_ctypes failed -- Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/test/test_macholib.py", line 51, in test_find
    self.assertIn(find_lib('pthread'),
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/test/test_macholib.py", line 43, in find_lib
    raise ValueError("%s not found" % (name,))
ValueError: pthread not found

This also failed when running installer 2 on the same 11.0 system.

I tested the installer 2 on 11.0 developer preview 9, 10.15, and 10.9. Other than the test_ctypes fallure mentioned above, there were no obvious regressions noted on 11.0, 10.15, or 10.9. I did not investigate in more detail whether the expected newer os.* functions and options were indeed available on 10.15 and not on 10.9, relying on the modified test suite to test that.

At this point, I would be comfortable saying that merging the PR as it stands should not cause regressions when building Python from source on 10.9 through 10.15 and when building the current python.org 10.9 variant. It should also allow single architecture builds from source on 11.0, assuming the test_ctypes failure is resolved. More testing is required for universal2 builds and more work is required for installer builds on 11.0 and on 10.15.

ned-deily

See comment in the discussion about current testing results.

ronaldoussoren · 2020-11-02T19:31:19Z

Great.

The test failure with installer 1 on macOS 11 is expected due to the shared library cache. It is possible to ignore that failure by detecting that we're on 11.0 (or later) and _dyld_shared_cache_contains_path is not available.

ronaldoussoren · 2020-11-05T12:12:24Z

FYI: My current plan is to merge next Saturday (Nov 7th), when I'll have time to react quickly to buildbot errors.

pradyunsg · 2020-11-05T13:22:11Z

FYI -- patchcheck is failing on this PR.

https://dev.azure.com/Python/cpython/_build/results?buildId=71072&view=logs&j=256d7e09-002a-52d7-8661-29ee3960640e&t=3d7276d3-4e8d-5309-55ad-fb0b172d9925&l=22

ronaldoussoren · 2020-11-07T10:30:15Z

I'm researching a crash that I get on a arm 11.0.1 rc1 when running test test suit (seems to be in the concurrent.future tests).

ronaldoussoren · 2020-11-07T11:08:56Z

I'm researching a crash that I get on a arm 11.0.1 rc1 when running test test suit (seems to be in the concurrent.future tests).

I noticed this crash because I got popups from the system, even though test test suite itself does not report test failures. This was with an installed framework install, I don't get these pop ups with a unix install.

These crashes are likely related to tests using the _crash function in test_concurrent_futures. This is not related to this PR and as such can be investigated more full later.

miss-islington · 2020-11-08T17:37:54Z

Thanks @ronaldoussoren for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

miss-islington · 2020-11-08T17:38:03Z

Sorry, @ronaldoussoren, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 41761933c1c30bb6003b65eef1ba23a83db4eae4 3.9

miss-islington · 2020-11-08T18:12:47Z

Thanks @ronaldoussoren for the PR 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2020-11-08T18:12:48Z

Sorry @ronaldoussoren, I had trouble checking out the 3.9 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 41761933c1c30bb6003b65eef1ba23a83db4eae4 3.9

Co-authored-by: Lawrence D’Anna <lawrence_danna@apple.com> * Add support for macOS 11 and Apple Silicon (aka arm64) As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy * Support building on recent versions of macOS while deploying to older versions This allows building installers on macOS 11 while still supporting macOS 10.9.

Backport of commit 4176193 to 3.9: ___ Co-authored-by: Lawrence D’Anna <lawrence_danna@apple.com> * Add support for macOS 11 and Apple Silicon (aka arm64) As a side effect of this work use the system copy of libffi on macOS, and remove the vendored copy * Support building on recent versions of macOS while deploying to older versions This allows building installers on macOS 11 while still supporting macOS 10.9.

(Apple Silicon being their aarch64 platform.) Patches consist of: - Upstream work: python/cpython#22855 - Fix for setup.py to find libbz2.tbd and libz.tbd now that with the shared library cache there's nothing in /usr/lib. See: https://bugs.python.org/issue41116 - Addition of __arch64__ case to fix _decimal module. A very similar fix has since been committed upstream.

maxbelanger · 2020-11-12T20:27:59Z

Is the plan still to backport this (or a part of this) to Python 3.8? We are not yet planning to migrate to Python 3.9 and the ability to build for Apple Silicon being merged thus available downlevel would be helpful (I also wouldn't expect any guarantee of support, I'm thinking of this as a "you can build for this configuration but you're on your own" situation).

dimaqq · 2020-11-13T00:25:26Z

@maxbelanger, sorry being out of the loop, though I hope this clarification may be useful to others who land on this ticket.
Who is "we" in your case?

maxbelanger · 2020-11-13T01:03:40Z

@dimaqq sorry, that wasn't clear. I was referring to Dropbox; we ship Python as part of our client application on macOS.

ronaldoussoren added 21 commits Jul 20, 2020

Add support for ARM64 to the pythonw tool

919efcc

Add support for "universal2" as a fat binary target on macOS

69c39f3

Tweak ctypes.macholib.dyld to work with the shared library cache.

Loading status checks…

3940c86

Silence compile time warning

deda5f0

We should consider just dropping support for macOS 10.4 here to simplify the code.

Fix test for macOS before 10.10

552bca8

ARM64 is a valid platform for macOS

Loading status checks…

e0c23a1

Unconditionally use uint32_t here.

Loading status checks…

e637a77

The preprocessor guard in the old version doesn't work, and isn't really needed (10.4 only supported 32-bit code where unsigned long is the same as uint32_t).

First stab at deploying to an earlier version of macOS than the build…

Loading status checks…

02fb660

… machine

Merge ctypes changes from Apple (PR 21241)

Loading status checks…

8e3b454

Needed because my previous workaround doesn't work anymore. This uses a private API, that should be made public later...

The archive-deps.py script isn't really useful

Loading status checks…

87c942b

Merge branch 'macos-deploy-to-earlier' into macos11-deploy-earlier-br…

515fbe6

…anch

Cleanup the code to check at runtime for a couple of APIs.

eee5437

Dynamicly fill posix._have_functions as well

3a1d4f2

Changes after testing with Xcode 6.2 on macOS 10.9

2f019f4

- __builtin_available is not present in the compiler, fall back to the ancient way of testing if a symbol is usable. - Logic error code in pytime.c resulted in a compile error.

Code formatting

cec3da7

Fix a number of errors in weaklinking support

86b5cf3

Merge branch 'macos11-deploy-earlier-branch' of github.com:ronaldouss…

d604cef

…oren/cpython into macos11-deploy-earlier-branch

Implement tests for weak linked symbols.

dde0ba4

The tests are not entirely done yet, and something similar needs to be done for the time module.

Finish the test for os.link behaviour

Loading status checks…

7ac26c4

ronaldoussoren added the DO-NOT-MERGE label Oct 21, 2020

ronaldoussoren requested review from abalkin, pganssle, tiran and python/macos-team as code owners Oct 21, 2020

the-knights-who-say-ni added the CLA signed label Oct 21, 2020

bedevere-bot added the awaiting core review label Oct 21, 2020

ronaldoussoren added 2 commits Oct 21, 2020

posixmodule.c uses multi-phase initialization

Loading status checks…

191a2d7

Make sure some form of prep_cif is called, even if the variadic versi…

Loading status checks…

e6d195b

…on isn't present. I don't particularly like the code duplication, but this code should be pretty stable.

ned-deily reviewed Nov 2, 2020

View changes

fxcoudert mentioned this pull request Nov 5, 2020

python@3.9: patch for Big Sur and arm64 Homebrew/homebrew-core#64169

Merged

ronaldoussoren added the needs backport to 3.9 label Nov 8, 2020

miss-islington assigned ronaldoussoren Nov 8, 2020

ronaldoussoren added needs backport to 3.9 and removed needs backport to 3.9 labels Nov 8, 2020

jibaku mentioned this pull request Nov 11, 2020

Python ThatGuySam/doesitarm#12

Closed

lawrence-danna-apple mentioned this pull request Nov 12, 2020

add support for macos 11.0, arm64, universal2 pypa/packaging#319

Open

emilmelnikov mentioned this pull request Nov 13, 2020

Add workaround for macOS Big Sur ilastik/ilastik#2345

Open

emlys mentioned this pull request Nov 13, 2020

InVEST 3.8.9 won't launch on Big Sur natcap/invest#383

Open

python / cpython

bpo-41100: Support macOS 11 and Apple Silicon #22855

bpo-41100: Support macOS 11 and Apple Silicon #22855

ronaldoussoren commented Oct 21, 2020 •

edited

ned-deily commented Nov 2, 2020 •

edited

ned-deily left a comment

ronaldoussoren commented Nov 2, 2020

ronaldoussoren commented Nov 5, 2020 •

edited

pradyunsg commented Nov 5, 2020

ronaldoussoren commented Nov 7, 2020

ronaldoussoren commented Nov 7, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

maxbelanger commented Nov 12, 2020

dimaqq commented Nov 13, 2020

maxbelanger commented Nov 13, 2020

python / cpython

Sponsor python/cpython

Join GitHub today

GitHub is where the world builds software

bpo-41100: Support macOS 11 and Apple Silicon #22855

bpo-41100: Support macOS 11 and Apple Silicon #22855

Conversation

ronaldoussoren commented Oct 21, 2020 • edited

ned-deily commented Nov 2, 2020 • edited

ned-deily left a comment

ronaldoussoren commented Nov 2, 2020

ronaldoussoren commented Nov 5, 2020 • edited

pradyunsg commented Nov 5, 2020

ronaldoussoren commented Nov 7, 2020

ronaldoussoren commented Nov 7, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

miss-islington commented Nov 8, 2020

maxbelanger commented Nov 12, 2020

dimaqq commented Nov 13, 2020

maxbelanger commented Nov 13, 2020

Essential cookies

Always active

Analytics cookies

ronaldoussoren commented Oct 21, 2020 •

edited

ned-deily commented Nov 2, 2020 •

edited

ronaldoussoren commented Nov 5, 2020 •

edited