Repository navigation
perf: Improve efficiency of array_has_any on sliced lists - #25427
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #25427 +/- ##
==========================================
+ Coverage 82.33% 82.36% +0.02%
==========================================
Files 1137 1137
Lines 432049 432937 +888
Branches 432049 432937 +888
==========================================
+ Hits 355723 356573 +850
- Misses 54826 54834 +8
- Partials 21500 21530 +30 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jayzhan211
approved these changes
Sep 18, 2026
jayzhan211
left a comment
Contributor
There was a problem hiding this comment.
Thanks @neilconway ! I left a few minor comments
neilconway
enabled auto-merge
September 18, 2026 14:27
haohuaijin
pushed a commit
to haohuaijin/arrow-datafusion
that referenced
this pull request
Sep 19, 2026
…5427) ## Which issue does this PR close? - Closes apache#25426. ## Rationale for this change When one argument of `array_has_any` is a scalar and the element type is not a string, the fast path `array_has_any_with_scalar_general` row-converts the list column's full child array, which does a lot of unnecessary work for sliced lists. Benchmarks: - array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs, −96.0% - array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs, −96.9% - array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs, −93.9% - array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs, −96.6% Results on unsliced inputs were unchanged, within run-to-run noise. ## What changes are included in this PR? * Improve `array_has_any_with_scalar_general` to only touch the visible portion of the input list * Add a benchmark * Add unit tests for correct behavior on sliced lists ## What is the testing strategy for this PR? Existing tests pass, new tests added. Benchmark confirms that the inefficiency has been addressed. ## Are there any user-facing changes? No.
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…5427) ## Which issue does this PR close? - Closes apache#25426. ## Rationale for this change When one argument of `array_has_any` is a scalar and the element type is not a string, the fast path `array_has_any_with_scalar_general` row-converts the list column's full child array, which does a lot of unnecessary work for sliced lists. Benchmarks: - array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs, −96.0% - array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs, −96.9% - array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs, −93.9% - array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs, −96.6% Results on unsliced inputs were unchanged, within run-to-run noise. ## What changes are included in this PR? * Improve `array_has_any_with_scalar_general` to only touch the visible portion of the input list * Add a benchmark * Add unit tests for correct behavior on sliced lists ## What is the testing strategy for this PR? Existing tests pass, new tests added. Benchmark confirms that the inefficiency has been addressed. ## Are there any user-facing changes? No.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
array_has_anywith a scalar argument is inefficient for sliced lists #25426.Rationale for this change
When one argument of
array_has_anyis a scalar and the element type is not a string, the fast patharray_has_any_with_scalar_generalrow-converts the list column's full child array, which does a lot of unnecessary work for sliced lists.Benchmarks:
Results on unsliced inputs were unchanged, within run-to-run noise.
What changes are included in this PR?
array_has_any_with_scalar_generalto only touch the visible portion of the input listWhat is the testing strategy for this PR?
Existing tests pass, new tests added. Benchmark confirms that the inefficiency has been addressed.
Are there any user-facing changes?
No.