Repository navigation
[Python] pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc at interpreter teardown (macOS arm64, py3.14); 22/23 unaffected #50428
Description
Activity
- added a commit that references this issue
on Jul 8, 2026 Could you share a gdb stacktrace?
I'm on macOS (arm64), so here is the crash data from the macOS crash reporter, which unwinds the full teardown chain:
EXC_BAD_ACCESS (SIGSEGV) KERN_INVALID_ADDRESS at 0x0000000000000017 #0 libarrow.2400.dylib _mi_theap_collect_retired + 124 #1 libarrow.2400.dylib mi_theap_collect_ex + 72 #2 libarrow.2400.dylib mi_process_done + 76 #3 libsystem_c.dylib __cxa_finalize_ranges + 480 #4 libsystem_c.dylib exit + 44 #5 libdyld.dylib dyld4::LibSystemHelpers::exit(int) const + 20 #6 dyld dyld4::LibSystemHelpersWrapper::exit(int) const + 172 #7 dyld start + 7232So libarrow's bundled mimalloc crashes in its
mi_process_doneatexit handler during__cxa_finalize, walking a thread-heap page list that contains a corrupted entry.Supporting detail from the same macOS crash report:
- Faulting instruction (lldb disasm):
ldrb w10, [x21, #0x16]with registerx21 = 0x1→ fault address0x17. The page pointer being walked is literally0x1. vmRegionInfo: "0x17 is not in any region" — a wild near-null pointer, not a mapped-but-protected page.- Register state at fault:
x19 = 0x1195300c0,x20 = 0x1,x21 = 0x1,x24 = 0x9,x25 = 0x4a— consistent with a list walk hitting a sentinel-like value (0x1) where ami_page_t*was expected. Possibly relevant: with multiple mimalloc instances in-process, one instance's static empty-page sentinel is a foreign address to the other.
Two additional data points that may help localize it:
- Under heavy allocation the same corruption also fires mid-operation rather than at exit (a second crash report shows the fault inside the co-loaded extension during object construction,
KERN_PROTECTION_FAILUREon a stack guard page). Teardown is just the most reproducible manifestation. - Version matrix: the crash requires pyarrow 24.0.0 and a co-loaded extension linking mimalloc v3, on CPython ≥3.13 (which vendors mimalloc v3 itself). pyarrow 23.0.1/22.0.0 are clean with the identical extension, and rebuilding the extension on mimalloc v2 is also clean against 24.0.0 — so whatever changed in 24.0.0's bundled allocator (or its teardown registration) interacts specifically with a second v3 instance in the process.
- Faulting instruction (lldb disasm):
PyArrow 25.0.0 will ship with mimalloc 3.3.1, so perhaps the issue will be solved then? (though it would be nice to ensure that this doesn't regress)
You can install a development wheel of PyArrow to try it out from here:
https://anaconda.org/channels/scientific-python-nightly-wheels/packages/pyarrow/overviewNightly (
25.0.0.dev257) tested: crash gone. The extension that reliably SIGSEGVs with 24.0.0 is clean across repeated runs, both import orders.The mechanism is mimalloc ≥ 3.2.6's fixed macOS TLS slots, shared by all co-resident v3 instances with no ownership check (microsoft/mimalloc#1327). 24.0.0's 3.2.7 crashes against a 3.3.1 extension because their heap layouts differ; 25.0.0's 3.3.1 coexists because the layouts happen to match. A pairing where layouts diverge again would re-expose the crash. Bundled ≥ 3.2.6 is the exposure window, which is why 23.0.0 with 3.1.5 was unaffected.
Based on microsoft/mimalloc#1327 (comment) and the following messages, it seems we could enable
MI_TLS_MODEL_LOCALwhen building Arrow's mimalloc, at least on Apple.- added a commit that references this issue
on Jul 20, 2026 - added a commit that references this issue
on Jul 21, 2026
Describe the bug, including details regarding any error messages, version, and platform.
Environment: macOS 15 arm64, CPython 3.14.3 (python.org), pyarrow from PyPI.
kgliteis a PyO3/abi3 Rust extension that statically links mimalloc v3 as its Rust global allocator (a very common Rust-wheel pattern). It links no Arrow code and exports one symbol. Import alone (no pyarrow API use, no kglite↔pyarrow data exchange) arms the crash;ARROW_DEFAULT_MEMORY_POOL=systemdoes not help;os._exit(0)avoids it (teardown-only).Crash:
EXC_BAD_ACCESSinside libarrow's bundled mimalloc (_mi_theap_collect_retired + 124, lldb). Process census via MIMALLOC_VERBOSE: CPython 3.14 vendors mimalloc v3.3.2, the extension carries v3.3.1, libarrow 2400 carries a v2-line copy. Switching the extension's copy to v2 — or downgrading pyarrow to ≤23.0.1 — eliminates the crash; so something introduced in 24.0.0 (bundled-mimalloc bump? teardown/atexit registration change?) makes the pre-existing three-allocator situation fatal.Impact: any Rust wheel shipping a mimalloc-v3 global allocator (the
mimalloccrate defaults to v3 since libmimalloc-sys 0.1.47) can now crash any Python ≥3.13 process it shares with pyarrow 24 — at exit, i.e. flaky-looking CI failures and exit-139s with no traceback.Cross-filed with microsoft/mimalloc (instance-isolation angle): microsoft/mimalloc#1327
Minimal repro wheels + lldb transcript available on request. A coexistence canary in Arrow's Python CI (import pyarrow alongside a mimalloc-v3 wheel, assert clean exit) would catch this class before release; we've added the mirror-image canary on our side.
Component(s)
Python