Skip to content

fix: clear_shrink(0) on rows should reset to original size + single group by fixes in clear_shrink - #25914

Merged
rluvaton merged 4 commits into
apache:mainfrom
rluvaton:release-rows-on-clear-shrink
Oct 5, 2026
Merged

rluvaton merged 4 commits into
apache:mainfrom
rluvaton:release-rows-on-clear-shrink

Conversation

@rluvaton

Copy link
Copy Markdown
Member

Which issue does this PR close?

None

Rationale for this change

For the Row change: clear_shrink 0 should release all memory and return to the initial size so we can return to a memory footprint that we know we can hold (this was found in #24015)

For the rest of the single group by fixes, the clear shrink left the struct in inconsistent state

What changes are included in this PR?

clear memory in rows and reset forgotten fields

What is the testing strategy for this PR?

Unit tests since for the group by it is not reached by our code but still leave in consistent state and it is a public trait
for the rows, it came from other test start failing in #24015 so I just added unit test here as well

Are there any user-facing changes?

No

…roup by fixes in clear_shrink

the clear_shrink was only called after emit, so this is why the bug was not shown before
@codecov-commenter

codecov-commenter commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.65%. Comparing base (9547b09) to head (760ef21).

Files with missing lines Patch % Lines
...gregates/group_values/single_group_by/primitive.rs 88.88% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25914      +/-   ##
==========================================
- Coverage   82.65%   82.65%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      446087   446179      +92     
  Branches   446087   446179      +92     
==========================================
+ Hits       368721   368783      +62     
- Misses      54980    55002      +22     
- Partials    22386    22394       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @rluvaton

left one comment also can you resolve the conflict

Comment thread datafusion/physical-plan/src/aggregates/group_values/row.rs
kosiew
kosiew previously requested changes Oct 2, 2026

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rluvaton,

Thanks for working on these clear_shrink fixes. I found one remaining issue in the row-backed path that still prevents clear_shrink(0) from returning to the initial memory footprint.

Comment thread datafusion/physical-plan/src/aggregates/group_values/row.rs
# Conflicts:
#	datafusion/physical-plan/src/aggregates/group_values/row.rs
@rluvaton

rluvaton commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

@kumarUjjawal @kosiew can you approve? I resolved your comments

@kumarUjjawal kumarUjjawal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @rluvaton

Looks good 👍

@rluvaton
rluvaton added this pull request to the merge queue Oct 5, 2026
Merged via the queue into apache:main with commit 5741773 Oct 5, 2026
42 checks passed
@rluvaton
rluvaton deleted the release-rows-on-clear-shrink branch October 5, 2026 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants