Skip to content

Track Apache Arrow mimalloc allocations - #943

Merged
godlygeek merged 1 commit into
bloomberg:mainfrom
pablogsal:arrow_support
Aug 12, 2026
Merged

godlygeek merged 1 commit into
bloomberg:mainfrom
pablogsal:arrow_support

Conversation

@pablogsal

@pablogsal pablogsal commented May 24, 2026 •

Copy link
Copy Markdown
Collaborator

Hook the arrow_mi_malloc_aligned, arrow_mi_realloc_aligned, and arrow_mi_free symbols introduced in Apache Arrow 25 so Memray can attribute Arrow logical allocations instead of seeing only mimalloc mmap arenas.

Scope: Linux Arrow 25+ builds using mimalloc with interposable arrow_mi_* wrapper symbols. Official PyArrow 25 Linux wheels satisfy these requirements.

The hooks are re-resolved when libraries are loaded after Tracker initialization. To avoid crossing independent statically linked mimalloc instances, Memray patches only the ELF object that defines the selected wrapper; if multiple providers are loaded, later providers are left untracked rather than routed through the wrong allocator.

Tests cover loading Arrow before tracking, during tracking, and between Tracker sessions, plus isolation between multiple providers.

@codecov-commenter

codecov-commenter commented May 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.42%. Comparing base (f70db81) to head (22a7ed1).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #943      +/-   ##
==========================================
- Coverage   92.45%   92.42%   -0.03%     
==========================================
  Files         101      102       +1     
  Lines       13211    13260      +49     
  Branches      477      477              
==========================================
+ Hits        12214    12256      +42     
- Misses        997     1004       +7     
Flag Coverage Δ
cpp 92.42% <100.00%> (-0.03%) ⬇️
python_and_cython 92.42% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread src/memray/_memray/tracking_api.cpp Outdated
@godlygeek

Copy link
Copy Markdown
Contributor

Looks like apache/arrow#41128 has landed. Should we move forward with this, @pablogsal ?

@pablogsal
pablogsal marked this pull request as ready for review August 9, 2026 19:20
@pablogsal

Copy link
Copy Markdown
Collaborator Author

Ready for rock and roll

@pablogsal
pablogsal force-pushed the arrow_support branch 3 times, most recently from 7d42df0 to 8af240b Compare August 9, 2026 20:52
Comment thread .github/workflows/coverage.yml Outdated
Comment thread src/memray/_memray/elf_shenanigans.cpp
Comment thread src/memray/_memray/hooks.cpp
Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>

@godlygeek godlygeek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I've fixed the coverage to require arrow>=25.0, and I tweaked the wording on the news item to something I think users will understand better (from "Track logical allocations" to "Track individual allocations", to make it clearer that what we were seeing previously was the allocations of the arenas and not the individual allocations from those arenas).

Other than that, this LGTM. Thanks @pablogsal!

@godlygeek

Copy link
Copy Markdown
Contributor

I do imagine we might eventually want to be able to interpose these on macOS as well, but if Arrow doesn't support that yet, that's a problem for another day.

@godlygeek
godlygeek enabled auto-merge (rebase) August 10, 2026 17:02
@godlygeek
godlygeek merged commit b126da3 into bloomberg:main Aug 12, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants