Skip to content

perf: Improve efficiency of array_has_any on sliced lists - #25427

Merged
neilconway merged 2 commits into
apache:mainfrom
neilconway:neilc/perf-array-has-sliced
Sep 18, 2026
Merged

neilconway merged 2 commits into
apache:mainfrom
neilconway:neilc/perf-array-has-sliced

Conversation

@neilconway

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When one argument of array_has_any is a scalar and the element type is not a string, the fast path array_has_any_with_scalar_general row-converts the list column's full child array, which does a lot of unnecessary work for sliced lists.

Benchmarks:

  • array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs, −96.0%
  • array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs, −96.9%
  • array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs, −93.9%
  • array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs, −96.6%

Results on unsliced inputs were unchanged, within run-to-run noise.

What changes are included in this PR?

  • Improve array_has_any_with_scalar_general to only touch the visible portion of the input list
  • Add a benchmark
  • Add unit tests for correct behavior on sliced lists

What is the testing strategy for this PR?

Existing tests pass, new tests added. Benchmark confirms that the inefficiency has been addressed.

Are there any user-facing changes?

No.

@github-actions github-actions Bot added the functions Changes to functions implementation label Sep 17, 2026
@codecov-commenter

codecov-commenter commented Sep 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.63014% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.36%. Comparing base (98b26c5) to head (7b200f3).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/functions-nested/src/array_has.rs 98.63% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25427      +/-   ##
==========================================
+ Coverage   82.33%   82.36%   +0.02%     
==========================================
  Files        1137     1137              
  Lines      432049   432937     +888     
  Branches   432049   432937     +888     
==========================================
+ Hits       355723   356573     +850     
- Misses      54826    54834       +8     
- Partials    21500    21530      +30     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neilconway ! I left a few minor comments

Comment thread datafusion/functions-nested/src/array_has.rs
@neilconway
neilconway added this pull request to the merge queue Sep 18, 2026
Merged via the queue into apache:main with commit 64871d9 Sep 18, 2026
41 checks passed
@neilconway
neilconway deleted the neilc/perf-array-has-sliced branch September 18, 2026 15:01
haohuaijin pushed a commit to haohuaijin/arrow-datafusion that referenced this pull request Sep 19, 2026
…5427)

## Which issue does this PR close?

- Closes apache#25426.

## Rationale for this change

When one argument of `array_has_any` is a scalar and the element type is
not a string, the fast path `array_has_any_with_scalar_general`
row-converts the list column's full child array, which does a lot of
unnecessary work for sliced lists.

Benchmarks:

- array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs,
−96.0%
- array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs,
−96.9%
- array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs,
−93.9%
- array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs,
−96.6%

Results on unsliced inputs were unchanged, within run-to-run noise.

## What changes are included in this PR?

* Improve `array_has_any_with_scalar_general` to only touch the visible
portion of the input list
* Add a benchmark
* Add unit tests for correct behavior on sliced lists

## What is the testing strategy for this PR?

Existing tests pass, new tests added. Benchmark confirms that the
inefficiency has been addressed.

## Are there any user-facing changes?

No.
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Oct 11, 2026
…5427)

## Which issue does this PR close?

- Closes apache#25426.

## Rationale for this change

When one argument of `array_has_any` is a scalar and the element type is
not a string, the fast path `array_has_any_with_scalar_general`
row-converts the list column's full child array, which does a lot of
unnecessary work for sliced lists.

Benchmarks:

- array_has_any_scalar/i64_no_match_sliced/1: 4.14 ms -> 167.4 µs,
−96.0%
- array_has_any_scalar/i64_no_match_sliced/10: 4.11 ms -> 125.9 µs,
−96.9%
- array_has_any_scalar/i64_no_match_sliced/100: 4.25 ms -> 257.8 µs,
−93.9%
- array_has_any_scalar/i64_no_match_sliced/1000: 4.12 ms -> 141.1 µs,
−96.6%

Results on unsliced inputs were unchanged, within run-to-run noise.

## What changes are included in this PR?

* Improve `array_has_any_with_scalar_general` to only touch the visible
portion of the input list
* Add a benchmark
* Add unit tests for correct behavior on sliced lists

## What is the testing strategy for this PR?

Existing tests pass, new tests added. Benchmark confirms that the
inefficiency has been addressed.

## Are there any user-facing changes?

No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

array_has_any with a scalar argument is inefficient for sliced lists

4 participants