Skip to content

Sequence try grow - #6948

Open
nat-openai wants to merge 8 commits into
apache:mainfrom
nat-openai:sequence-try-grow
Open

nat-openai wants to merge 8 commits into
apache:mainfrom
nat-openai:sequence-try-grow

Conversation

@nat-openai

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Addresses part 2 of #5712: memory accounting for integral sequence construction.

Rationale for this change

Integral sequence can allocate a large Arrow values buffer without consulting Comet’s memory pool. Fallible allocation handles allocator refusals, but does not enforce the task’s memory budget.

This change reserves construction memory through the task’s existing pool before allocating buffers. The reservation lasts only for the invocation; downstream buffering operators remain responsible for accounting for the output they retain.

What changes are included in this PR?

  • Construct the sequence UDF through the scalar-function factory using SessionState, preserving the session’s exact memory pool and tracking wrappers.
  • Use staged try_grow reservations for column inputs: reserve offsets, validity buffers and length scratch first, then reserve values after validating and caching each row’s length once.
  • Retain a single reservation request for scalar, empty and metadata-proven all-null inputs. Construct complete scalar batches within the reservation so native constant folding cannot bypass admission.
  • Release reservations and consumer registrations on return, errors and unwinding. Preserve Spark’s null handling, overflow checks and error precedence, while propagating non-resource pool errors unchanged.
  • Add pooled benchmarks alongside the existing kernel benchmarks and document the reservation’s scope and limitations.

The check follows the existing pool policy and introduces no new memory setting. A refusal can occur before a downstream native operator reaches its spill step; this does not add native reclamation or guarantee a process-memory ceiling. On-heap mode retains its existing unbounded pool policy.

How are these changes tested?

  • Native tests cover both reservation stages, refusal and recovery, concurrent admission, scalar expansion, null and sliced inputs, error precedence, interruption propagation, and cleanup while returned arrays or FFI exports remain alive.
  • Factory and planner tests verify session-pool wiring. Spark tests cover bounded-pool refusal, recovery with smaller batches, and sorting and collecting sequence output.
  • The new sequence_pooled Criterion group exercises the UDF with a tracked, bounded pool across short, long, nullable and scalar inputs.

Preliminary diagnostics retain approximately 5–7% overhead for the two-element column/null cases and 12.5% for default singleton rows; performance parity is not claimed. Validation against the pinned DataFusion 55.2, JVM test execution and production JNI performance measurements remain pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:expressions Expression evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant