fix(py): pin mimalloc v2 — v3 corrupts a co-resident pyarrow heap

[niko86]

✅ Not stacked — branches off main (ec66197), full CI applies. Closes #294; re-points #122.

What it is

import pyarrow then import laterite was enough to corrupt pyarrow's buffers:

import pyarrow as pa      # pyarrow's bundled mimalloc initialises first
import laterite           # our cdylib brought a second mimalloc v3

vals = ["A" * 32, "B" * 32, "C" * 32, "D" * 32]
pa.table({"c": vals}).column(0).chunk(0).to_pylist()
# ['\x00' * 32, '\x00' * 32, 'CCCC…', 'DDDD…']   <- first 64 bytes zeroed

No laterite call. No Arrow FFI. 100% reproducible, and green with the pin.

pyarrow bundles its own mimalloc as its default memory pool, and libmimalloc-sys defaults to v3 from 0.1.47. Two co-resident mimalloc v3 instances hand out overlapping memory — microsoft/mimalloc#1327.

This is known upstream and already fixed on Arrow's side. apache/arrow GH-50428 — "pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc" — milestone 25.0.1, fixed by #50549. It independently names switching the extension's mimalloc copy to v2 as the workaround, which is exactly this PR.

So why pin at all, if pyarrow 25.0.1 fixes it? Because we do not control which pyarrow a user installs — [pyarrow] resolves to whatever their environment allows, and 24.0.0/25.0.0 are affected. GH-50428 also records that CPython 3.14 vendors its own mimalloc v3, so pyarrow is not the only co-residency partner in the process.

Measured across 50 array shapes on our v3 build: pyarrow 25.0.0 corrupts 44 of 50, pyarrow 25.0.1 corrupts 0 of 50. Under the v2 pin: 0 of 50 on both.

The diagnosis it replaces

#122 attributed this to an arrow-rs use-after-free consuming a pyarrow-produced ArrowArrayStream (apache/arrow-rs#10439) and landed a pl.from_pandas guard. That is not what is happening, which is why #294 kept reproducing through the guarded path.

claim	measured
pandas-specific	no — a raw pyarrow.Table in a live local corrupts identically
needs the Arrow FFI	no — the minimal repro calls nothing
arrow-rs frees a foreign pointer	no — an allocator shim watching pyarrow's buffer address never fired
only under a non-system allocator	yes — and it needs both mimallocs

Cross-matrix (ours × pyarrow's pool): mimalloc × mimalloc → corrupt; mimalloc × system, system × mimalloc, system × system → clean. Corruption needs a string buffer >64 bytes with 2+ rows, in the first pyarrow allocations after our module loads.

Why not just drop the allocator

lib.rs sets mimalloc because the read hot-path is allocation-bound, and speed is the headline claim — so the cheap fix was the wrong one. On the 25 MB fixture (median of 7, ×3):

build	read	validate
v3 (was shipping)	115–118 ms	223–227 ms
v2 (this PR)	122–125 ms	229–233 ms
no mimalloc	150–152 ms	269–272 ms

v2 keeps ~18–19% read / ~14–15% validate over the system allocator. Removing mimalloc gives that back; the pin costs ~6% read against v3.

Scope

pyarrow only — polars and duckdb are clean, so the base pip install laterite was never exposed. Affected: [pyarrow], [compat,pyarrow], [all], and dev/CI (--all-extras).

laterite-node and laterite-cli take the same pin in a second commit. Neither is a proven exposure and neither is a fix: the addon loads no pyarrow but is dlopen'd beside addons we do not choose, and lat owns its whole process so the fault cannot reach it. Version hygiene — one workspace, one mimalloc major, so a future surface cannot inherit v3 by default.

Tests

The regression test runs in a subprocess: the fault depends on which allocator initialises first, so it is a property of process startup, and no in-process assertion can express it — by the time a test module imports, laterite is already loaded.

That is also the honest answer to "why did the #122 regression never catch this": neither of its assertions can. The texts match with or without the guard, and one native write cannot detect a corrupted heap. Its docstring now says so instead of claiming a guarantee it cannot check. I did not manufacture a second test at that seam — I tried, and both public pandas paths stay green even on the faulty build, so a test there would have been more of the same theatre.

• test_pyarrow_buffers_survive_a_coresident_native_module — red on v3, green on v2, verified by swapping the two .so builds under the same test run.
• Payload shape is load-bearing and commented as such, so nobody shrinks the fixture and silently disarms it.

Verification

• pytest packages/laterite/tests — 1026 passed
• ci.yml's python job invocation, 8×: 1241 passed, 99.11%, no crashed workers (test(python): intermittent heap corruption in an in-process native call fails the coverage gate ~1 run in 4 #294 reported ~1 in 4 failing at a depressed 93.39%)
• ruff check · ruff format --check · ty check — clean

The guard stays

build_ags4's pl.from_pandas guard is no longer what makes the pandas path memory-safe — the unguarded path is clean after the pin. But it is still load-bearing for a different reason: on a pyarrow-free [compat] install, pandas' __arrow_c_stream__ calls import_optional_dependency("pyarrow") and raises. Measured with pyarrow blocked: guarded → OK, unguarded → ImportError: Missing optional dependency 'pyarrow'. Comment corrected; retiring it is a separate, deliberate change.

Confirmed against the upstream reproducer

apache/arrow-rs#10439 has been withdrawn and closed, on evidence rather than inference. I rebuilt its reproducer verbatim and ran it at the versions originally reported, changing only the mimalloc major:

reproducer's Cargo.toml	result
mimalloc = "0.1" (v3, as filed)	SIGSEGV rc=139, 2/2
+ features = ["v2"]	survives 500 iters, 3/3

arrow-rs identical across both. Worth recording two corrections to my own earlier reasoning:

• It did not reproduce on my first attempt, because the resolver gave me pandas 3.0.5 instead of the reported 2.3.3. At the reported versions it segfaults reliably. A non-reproduction at the wrong versions would have been a bad reason to close it.
• In that reproducer the arrow-rs consume call is necessary — a control keeping co-residency and heap churn but never calling consume() survives. I read that as arrow-rs being the workload that grows the Rust heap into the colliding region, not a defect in its consume path, and said so upstream so maintainers can disagree.

That prompted me to re-check the guard claim below under iteration rather than single-shot, since the upstream crash needs ~200 iterations: 500 iterations of the unguarded pandas → native emit survive on v2 and fail immediately on v3. The claim holds.

Not done here

• Verified on macOS arm64 only. Whether Linux CI is exposed is untested.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!