Skip to content

Fix retained-capacity accounting for grouped byte MIN/MAX - #26154

Open
kosiew wants to merge 2 commits into
apache:mainfrom
kosiew:memcalc-13-23393
Open

kosiew wants to merge 2 commits into
apache:mainfrom
kosiew:memcalc-13-23393

Conversation

@kosiew

@kosiew kosiew commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Grouped byte MIN/MAX accumulators can underreport memory usage because their accounting is based on the current data length and number of group slots rather than the memory allocations they retain.

For example, replacing a 128-byte string with a one-byte string reuses the existing vector allocation, but the previous accounting reports only the new string length. Similarly, unused capacity in the outer group vector is not counted.

This change makes memory accounting reflect retained allocations, providing more accurate memory usage estimates for grouped MIN/MAX operations.

What changes are included in this PR?

  • Add total_data_capacity to MinMaxBytesState to track retained inner Vec<u8> capacities independently of total_data_bytes, which remains responsible for output preallocation.
  • Update set_value() to maintain capacity accounting when values are inserted, replaced, or cause an existing vector to grow.
  • Update size() to account for the outer min_max vector's capacity and the cached inner vector capacities, without double-counting the inline Option<Vec<u8>> descriptors.
  • Keep size() O(1) by maintaining the inner capacity total incrementally.
  • Update emit_to() to reset capacity accounting on full emission and subtract emitted inner vector capacities on partial emission.

Are these changes tested?

Three regression tests are added:

  • size_counts_outer_capacity: Verifies that memory accounting includes allocated outer vector capacity, even when fewer group slots are initialized.
  • size_retains_inner_capacity: Tests both MIN and MAX accumulators, verifying that replacing a long string with a shorter one preserves retained-capacity accounting and that partial and full emission update it correctly.
  • size_tracks_growth_and_emit_reuse: Verifies capacity accounting when an inner vector grows, groups are partially emitted, and the state is reused after full emission.

Are there any user-facing changes?

No changes to SQL MIN/MAX results or public APIs are intended.

The change corrects internal memory accounting for grouped byte MIN/MAX accumulators, allowing retained allocations to be reported more accurately.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

kosiew added 2 commits October 9, 2026 16:59
- Counts retained outer/inner capacities; size() stays O(1).
- Adds 3 regressions: capacity retention, growth, emit/reuse.
- Existing regression now checks evaluate(EmitTo::First(1)) output + remaining size, both min/max.
- Kept direct full-emission byte-length checks.
@github-actions github-actions Bot added the functions Changes to functions implementation label Oct 9, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.40230% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.77%. Comparing base (6a31aa8) to head (e7a07f4).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
...n/functions-aggregate/src/min_max/min_max_bytes.rs 95.40% 0 Missing and 4 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26154      +/-   ##
==========================================
+ Coverage   82.76%   82.77%   +0.01%     
==========================================
  Files        1147     1147              
  Lines      450354   450850     +496     
  Branches   450354   450850     +496     
==========================================
+ Hits       372714   373179     +465     
- Misses      54932    54947      +15     
- Partials    22708    22724      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kosiew
kosiew marked this pull request as ready for review October 9, 2026 14:28
@kosiew
kosiew requested a review from comphead October 9, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants