Fix macOS pybind11 module segfault: Python_FIND_STRATEGY=LOCATION - #189
Merged
Conversation
On macOS, Python_EXECUTABLE alone doesn't stop FindPython's separate Development.Module/.Embed library lookups from independently resolving to a different Homebrew framework Python than the pinned interpreter (e.g. python@3.14 present on the macos-14 runner image, vs. the 3.12 actions/setup-python provisions). This produced pybind11 extension modules compiled against 3.12 headers but linked against 3.14's libpython, which crashed at import time inside pybind11's own version-mismatch error-reporting path. Root-caused and verified via NWChemEx/SCF's macOS CI (previously segfaulting deterministically on both compiler legs, now passing 4/4 tests including the C++ binaries that embed their own interpreter). Full investigation: scf-macos-pybind11-segfault.md in SCF's repo. Explicitly forcing Python_LIBRARY was also tried and rejected: it fixed the pybind11 module link but broke a dependency's separate Development.Embed lookup. Python_FIND_STRATEGY=LOCATION -- CMake's own documented mechanism for this exact scenario -- fixes both cleanly.
Contributor
|
🚀 [bumpr] Bumped! |
1 task
ryanmrichard
added a commit
to NWChemEx/SCF
that referenced
this pull request
Aug 11, 2026
test_pip_build's macOS gcc-14 leg fails deterministically for an unrelated, pre-existing reason (meson can't detect Homebrew gcc-14's linker while building numpy from source -- "Unable to detect linker for compiler `gcc-14 -Wl,--version`"), not the pybind11 segfault fixed by NWChemEx/.github#189. That fix only touches test_cmake_build's Configure step, so dropping allow_macos_failure from test_pip_build too was premature -- confirmed via this PR's own CI, which surfaced the gcc-14 failure again once unmasked. Keeping it dropped for test_cmake_build, which is genuinely fixed and verified (4/4 tests passing on both compiler legs).
ryanmrichard
added a commit
to NWChemEx/SCF
that referenced
this pull request
Aug 12, 2026
* Drop allow_macos_failure now that the macOS segfault is fixed The macOS pybind11 module segfault (scf-macos-pybind11-segfault.md) is fixed upstream via NWChemEx/.github#189 (Python_FIND_STRATEGY=LOCATION), verified 4/4 passing on both compiler legs. macOS should now be a real merge-blocking signal like every other platform/job. * Restore allow_macos_failure for test_pip_build only (separate bug) test_pip_build's macOS gcc-14 leg fails deterministically for an unrelated, pre-existing reason (meson can't detect Homebrew gcc-14's linker while building numpy from source -- "Unable to detect linker for compiler `gcc-14 -Wl,--version`"), not the pybind11 segfault fixed by NWChemEx/.github#189. That fix only touches test_cmake_build's Configure step, so dropping allow_macos_failure from test_pip_build too was premature -- confirmed via this PR's own CI, which surfaced the gcc-14 failure again once unmasked. Keeping it dropped for test_cmake_build, which is genuinely fixed and verified (4/4 tests passing on both compiler legs). * Scope the build-time numpy<2 cap to Linux; unblocks macOS pip_build numpy<2 was capped unconditionally to work around a manylinux2014/GCC-ABI constraint that only exists on Linux. On macOS it had the opposite effect: numpy<2's newest release (1.26.4) has no cp314 wheel, so pip fell back to building it from source, which fails distinctly there -- Homebrew's gcc-14 has no bundled linker and shells out to Apple's system ld, and meson's linker-detection probe (`gcc-14 -Wl,--version`) doesn't understand current macOS/Xcode's linker ("ld: unknown options: --version"). That's the test_pip_build (macos-14, gcc-14) failure allow_macos_failure has been masking. Scope the cap to sys_platform == 'linux' via a PEP 508 marker and leave macOS unconstrained, so it resolves a real numpy>=2 wheel instead of falling back to a source build at all -- sidesteps the linker-probe issue entirely rather than working around it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test_cmake_buildon every macOS leg since NWChemEx/SCF's CI migration (SCF#64).Python_EXECUTABLEalone pins CMake's interpreter/header search but not FindPython's separateDevelopment.Module/Development.Embedlibrary lookups, which on macOS can independently resolve to a different Homebrew framework Python (python@3.14, present on themacos-14runner image) than theactions/setup-python-provisioned 3.12 interpreter. This produced pybind11 extension modules compiled against 3.12 headers but linked against 3.14'slibpython, crashing at import time inside pybind11's own version-mismatch error-reporting path (PyErr_Format).Python_FIND_STRATEGY=LOCATION/Python3_FIND_STRATEGY=LOCATIONon macOS -- CMake's documented mechanism for tying all Development component lookups to the interpreter's own location.Python_LIBRARY) was tried and rejected: it fixed the pybind11 module link but broke a different dependency'sDevelopment.Embedlookup.FIND_STRATEGY=LOCATIONfixes both cleanly.Full investigation trail:
scf-macos-pybind11-segfault.md(in SCF's repo).Test plan
test_cmake_buildmacOS CI (throwaway PR, closed): previously segfaulted deterministically on bothclang-18andgcc-14; with this fix,100% tests passed, 0 tests failed out of 4on both legs, includingtest_unit_scf/test_integration_scf(pure C++ binaries that embed their own Python interpreter -- the case that regressed under the rejectedPython_LIBRARYalternative).pull_request.yamlwill be reverted from its temporary diagnostics-branch pin back to@master, and theallow_macos_failureworkaround dropped.🤖 Generated with Claude Code