Skip to content

Account for retained sort-key capacity #26099

Description

@kosiew

Parent: #23393

Context: PR comment

Goal

Make partial group-ordering memory accounting include the retained allocation
for its Vec<ScalarValue> sort key.

Invariant / contract

GroupOrderingPartial::heap_size() reports the retained heap allocations owned
by its state. For an in-progress state, this includes the sort-key vector's
allocated capacity and heap allocations owned by its initialized scalar
elements, each exactly once. It does not add an inline descriptor already
charged by GroupOrdering::size().

Tasks

  • Replace the live-element-only sort-key accounting in
    State::heap_size() with capacity-aware DFHeapSize accounting, following
    the grouped COUNT accumulator's established approach.
  • Add a deterministic test with spare Vec<ScalarValue> capacity and a
    heap-owning scalar, proving spare slots and initialized scalar payloads are
    accounted for without relying on allocator measurements.
  • Preserve the existing ownership boundary: do not charge inline
    descriptors or allocations owned by other ordering state again.

Acceptance criteria

  • An in-progress sort key with spare vector capacity reports that capacity
    even when the vector's live length is smaller.
  • Heap allocations owned by initialized ScalarValues remain included.
  • Existing start/complete and ordering-state accounting behavior remains
    unchanged.

Tests / verification

  • Add or update the focused GroupOrderingPartial tests in
    datafusion/physical-plan/src/aggregates/order/partial.rs.
  • Run cargo test -p datafusion-physical-plan.
  • Run ./ci/scripts/doc_prettier_check.sh --write --allow-dirty for this
    Markdown change.

Dependencies / blockers

Keep the descriptor accounting established by the GroupOrdering owner
separate from this retained-capacity fix.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions