Repository navigation
Track Apache Arrow mimalloc allocations - #943
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #943 +/- ##
==========================================
- Coverage 92.45% 92.42% -0.03%
==========================================
Files 101 102 +1
Lines 13211 13260 +49
Branches 477 477
==========================================
+ Hits 12214 12256 +42
- Misses 997 1004 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Looks like apache/arrow#41128 has landed. Should we move forward with this, @pablogsal ? |
f7d234a to
0ce0c24
Compare
|
Ready for rock and roll |
7d42df0 to
8af240b
Compare
Signed-off-by: Pablo Galindo Salgado <pablogsal@gmail.com>
8af240b to
22a7ed1
Compare
godlygeek
left a comment
There was a problem hiding this comment.
LGTM. I've fixed the coverage to require arrow>=25.0, and I tweaked the wording on the news item to something I think users will understand better (from "Track logical allocations" to "Track individual allocations", to make it clearer that what we were seeing previously was the allocations of the arenas and not the individual allocations from those arenas).
Other than that, this LGTM. Thanks @pablogsal!
|
I do imagine we might eventually want to be able to interpose these on macOS as well, but if Arrow doesn't support that yet, that's a problem for another day. |
Hook the
arrow_mi_malloc_aligned,arrow_mi_realloc_aligned, andarrow_mi_freesymbols introduced in Apache Arrow 25 so Memray can attribute Arrow logical allocations instead of seeing only mimalloc mmap arenas.Scope: Linux Arrow 25+ builds using mimalloc with interposable
arrow_mi_*wrapper symbols. Official PyArrow 25 Linux wheels satisfy these requirements.The hooks are re-resolved when libraries are loaded after Tracker initialization. To avoid crossing independent statically linked mimalloc instances, Memray patches only the ELF object that defines the selected wrapper; if multiple providers are loaded, later providers are left untracked rather than routed through the wrong allocator.
Tests cover loading Arrow before tracking, during tracking, and between Tracker sessions, plus isolation between multiple providers.