Repository navigation
Heap corruption / SIGSEGV consuming a pyarrow-produced ArrowArrayStream under a non-system global allocator #10439
Description
Describe the bug
Consuming an Arrow C stream produced by pyarrow (e.g. a pandas ≥2.2 DataFrame's
__arrow_c_stream__) via arrow::ffi_stream::ArrowArrayStreamReader corrupts the
process heap when the extension uses a non-system global allocator (mimalloc). The
damage surfaces on a later, unrelated allocation as either silently-wrong bytes or a
hard SIGSEGV.
Key isolation:
- Reproduces via arrow-rs's own
ArrowArrayStreamReader::from_pyarrow_bound— i.e.
no pyo3-arrow involved. (It also reproduces viapyo3_arrow::PyTable, confirming
that wrapper is just a passthrough and the fault is in the arrow-rs consume path.) - A polars-produced (arrow-rs-native) stream through the identical code path is
never affected — only pyarrow-produced streams. - It only manifests under mimalloc; the system allocator masks it (consistent
with an invalid/double free the system allocator tolerates but mimalloc does not). - Deterministic: SIGSEGV on every run (3/3), typically within ~200 iterations.
To Reproduce
Cargo.toml:
[package]
name = "arrowrs_stream_repro"
version = "0.0.0"
edition = "2021"
[lib]
crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.29", features = ["extension-module", "abi3-py312"] }
arrow = { version = "59", features = ["pyarrow"] }
mimalloc = "0.1"src/lib.rs:
use arrow::ffi_stream::ArrowArrayStreamReader;
use arrow::pyarrow::FromPyArrow;
use arrow::util::display::{ArrayFormatter, FormatOptions};
use pyo3::exceptions::PyRuntimeError;
use pyo3::prelude::*;
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
/// Consume a C stream from any object exposing __arrow_c_stream__, read every cell, drop.
#[pyfunction]
fn consume(obj: &Bound<'_, PyAny>) -> PyResult<usize> {
let reader = ArrowArrayStreamReader::from_pyarrow_bound(obj)?;
let opts = FormatOptions::default();
let mut n = 0usize;
for batch in reader {
let batch = batch.map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
let fmts: Vec<ArrayFormatter> = batch
.columns()
.iter()
.map(|c| ArrayFormatter::try_new(c.as_ref(), &opts))
.collect::<Result<_, _>>()
.map_err(|e| PyRuntimeError::new_err(e.to_string()))?;
for row in 0..batch.num_rows() {
for f in &fmts {
n += f.value(row).to_string().len();
}
}
}
Ok(n)
}
/// Allocate + integrity-check on the (mimalloc) heap — trips if a prior free corrupted it.
#[pyfunction]
fn canary() -> PyResult<u64> {
let sizes = [16u32, 48, 96, 256, 1024, 4096, 7000, 8192, 16384, 65536];
let mut live: Vec<Vec<u8>> = Vec::new();
let mut sum = 0u64;
for _ in 0..64 {
for &sz in &sizes {
let v: Vec<u8> = (0..sz).map(|i| (i % 251) as u8).collect();
for (i, &b) in v.iter().enumerate() {
if b != (i % 251) as u8 {
return Err(PyRuntimeError::new_err("mimalloc heap corrupted"));
}
sum = sum.wrapping_add(b as u64);
}
live.push(v);
if live.len() > 32 {
live.remove(0);
}
}
}
Ok(sum)
}
#[pymodule]
fn arrowrs_stream_repro(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(consume, m)?)?;
m.add_function(wrap_pyfunction!(canary, m)?)?;
Ok(())
}repro.py (maturin develop --release && python repro.py pandas):
import sys
import pandas as pd, polars as pl
import arrowrs_stream_repro as r
producer = sys.argv[1] if len(sys.argv) > 1 else "pandas"
for i in range(500):
frame = (pd if producer == "pandas" else pl).DataFrame(
{"a": [f"x{i}", "yy", "zzz"], "b": ["p", "qq", "rrr"]}
)
r.consume(frame)
r.canary()
print(f"survived (producer={producer})")python repro.py pandas→ SIGSEGV (exit 139), reliably within ~200 iters.python repro.py polars→ survives 500 iters clean.
Expected behavior
Consuming a valid pyarrow-produced ArrowArrayStream should be memory-safe and leave
the process heap intact — identical to a polars/arrow-rs-native stream. (The pyarrow
capsule is valid: it re-imports cleanly via
pyarrow.RecordBatchReader._import_from_c_capsule(...).)
Additional context
Versions: arrow 59.1.0 · pyarrow 25.0.0 · pandas 2.3.3 · pyo3 0.29 ·
mimalloc 0.1 · Python 3.14.3 (also 3.12) · macOS arm64.
What I ruled out:
- pyo3-arrow — arrow-rs's own
from_pyarrow_boundreproduces it, so the wrapper isn't
the cause. - Capsule double-release —
import_stream_pycapsule/FFI_ArrowArrayStream::from_raw
correctly nulls the source release callback. - Read path — deep-copying the imported batches (
concat_batches) before use does
not help; the corruption is at release/drop of the pyarrow-produced
stream/arrays, not at read.
Hypothesis: the stream is drained (eager Vec<RecordBatch>), then the stream is
released while the arrays are still held. Per the C Data Interface, stream-produced
arrays are lifetime-independent of the stream; if pyarrow's exported arrays are instead
coupled to the stream's lifetime, releasing the stream first is a use-after-free.
polars' arrays are genuinely independent → unaffected. This may ultimately be a pyarrow
(apache/arrow C++) protocol issue surfaced by arrow-rs's consume/release ordering — but
arrow-rs is the consumer, and the crash is memory-unsafe from safe Rust, so filing here.
An ASAN/valgrind run over the reproducer would pinpoint the exact double-free — happy to
help if useful.
Activity
Python coverage batch (P3/P4/P5) + build_ags4 pandas heap-UAF guard (#…
niko86 commentedon Aug 13, 2026
Withdrawing this — the fault is not in arrow-rs. Sorry for the noise, and thanks to anyone who spent time on it.
Root cause
pyarrow bundles its own mimalloc as its default memory pool (
pa.default_memory_pool().backend_name→mimalloc). The reproducer above links a second mimalloc via themimalloccrate, andlibmimalloc-sys0.1.49 defaults to mimalloc v3. Two co-resident v3 instances hand out overlapping memory — microsoft/mimalloc#1287, fixed ondev3but not in any stable release. apache/datafusion-python#1607 is the same fault in the same shape.Controlled experiment
The reproducer in the issue body, unchanged, at the versions originally reported (
pandas2.3.3,pyarrow25.0.0, Python 3.12, macOS arm64) — now onarrow59.2.0. The only variable is the mimalloc major:Cargo.tomlmimalloc = "0.1"(→ v3.3.2, as filed)mimalloc = { version = "0.1", features = ["v2"] }(→ v2.3.2)arrow-rs is byte-identical across those two runs. A
polarsproducer survives in both.Where my isolation reasoning went wrong
Every "key isolation" bullet in the report is equally explained by allocator co-residency, which I never tested:
mimalloc × system,system × mimallocandsystem × systemare all clean.from_pyarrow_bound, so pyo3-arrow is exonerated" — correct, but it exonerates arrow-rs by the same argument. What both paths have in common is that pyarrow is imported, not which API consumed the stream.Two further checks in my own project, where this originally surfaced: an instrumented
GlobalAllocwrappingMiMallocand watching pyarrow's buffer address never fired while corruption still occurred — so the Rust allocator never receives the foreign pointer, consistent withDeallocation::Custom's drop arm being a no-op. And the corruption reproduces there with no arrow-rs code on the path at all —import pyarrowthen importing the extension is sufficient to corrupt a subsequently-builtpa.table.One honest caveat
In this reproducer the arrow-rs consume call is necessary to trigger the crash: a control that keeps the co-residency and the heap churn but never calls
consume()survives 500 iterations. I read that as arrow-rs being the workload that grows the Rust heap into the colliding region while pyarrow buffers are live, rather than as a defect in the consume path — the v2/v3 A/B above is what I'd weight, since it changes nothing else. Flagging it so maintainers can reach their own conclusion; happy to reopen with better evidence if anyone reads it differently.No action needed on arrow-rs. Closing.
niko86 commentedon Aug 13, 2026
Correction to my previous comment, and a better citation — the conclusion is unchanged but two details in it were wrong.
1. The version that matters is pyarrow, not pandas. I said the reproducer stopped reproducing under pandas 3.0.5. Isolating the variables properly, pandas is irrelevant:
2. The right upstream issue is microsoft/mimalloc#1327, not #1287. I cited #1287 second-hand without reading it; it is "mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads" — a threading bug, not this. #1327 is the co-residency report: "Two independently-linked mimalloc v3 instances … SIGSEGV in
_mi_theap_collect_retiredat process exit".Better still, Arrow already has this from its own side: apache/arrow GH-50428 — "pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc" — milestone 25.0.1, fixed by #50549 "Better mimalloc configuration on macOS". That issue independently names switching the extension's mimalloc copy to v2 as the workaround, which is what the A/B in my previous comment measured, and notes CPython 3.14 vendors its own mimalloc v3 as a third instance.
So this was a pyarrow-side regression with an arrow-rs-shaped symptom, and it is fixed in pyarrow 25.0.1. Confirming across 50 array shapes on an unchanged mimalloc-v3 consumer: pyarrow 25.0.0 corrupts 44/50, pyarrow 25.0.1 corrupts 0/50.
Staying closed — nothing here for arrow-rs. Apologies for the noisy trail; leaving the corrected record for anyone who lands on this from a search.
fix(py): pin mimalloc v2 — v3 corrupts a co-resident pyarrow heap (#301)
fix(refs): the compliance crate's #458 and #446 now point here, not a…