Repository navigation
Sequence try grow - #6948
Open
nat-openai wants to merge 8 commits into
Open
Sequence try grow#6948nat-openai wants to merge 8 commits into
nat-openai wants to merge 8 commits into
Conversation
nat-openai
force-pushed
the
sequence-try-grow
branch
from
October 14, 2026 20:35
88bd7e5 to
f147d0d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Addresses part 2 of #5712: memory accounting for integral
sequenceconstruction.Rationale for this change
Integral
sequencecan allocate a large Arrow values buffer without consulting Comet’s memory pool. Fallible allocation handles allocator refusals, but does not enforce the task’s memory budget.This change reserves construction memory through the task’s existing pool before allocating buffers. The reservation lasts only for the invocation; downstream buffering operators remain responsible for accounting for the output they retain.
What changes are included in this PR?
SessionState, preserving the session’s exact memory pool and tracking wrappers.try_growreservations for column inputs: reserve offsets, validity buffers and length scratch first, then reserve values after validating and caching each row’s length once.The check follows the existing pool policy and introduces no new memory setting. A refusal can occur before a downstream native operator reaches its spill step; this does not add native reclamation or guarantee a process-memory ceiling. On-heap mode retains its existing unbounded pool policy.
How are these changes tested?
sequence_pooledCriterion group exercises the UDF with a tracked, bounded pool across short, long, nullable and scalar inputs.Preliminary diagnostics retain approximately 5–7% overhead for the two-element column/null cases and 12.5% for default singleton rows; performance parity is not claimed. Validation against the pinned DataFusion 55.2, JVM test execution and production JNI performance measurements remain pending.