CrossJoinExec buffers the entire left input without spilling (OOM under memory pressure) #26383

Is your feature request related to a problem or challenge?

CrossJoinExec buffers the entire left input in memory with no spill path, so a large left side can exhaust the memory pool even when the right side (and the useful working set) is tiny.

In datafusion/physical-plan/src/joins/cross_join.rs, load_left_input folds all left batches into a Vec, growing a MemoryReservation per batch:

async fn load_left_input(..., reservation: MemoryReservation, ...) {
    let (batches, _metrics, reservation) = stream
        .fold((Vec::new(), metrics, reservation), |(mut batches, metrics, reservation), batch| async {
            let batch_size = batch.get_array_memory_size();
            reservation.try_grow(batch_size)?;   // grow-or-error; no spill
            batches.push(batch);
            ...
        }).await?;
}

When the pool refuses try_grow, execution fails with Resources exhausted: Additional allocation failed for CrossJoinExec ... can spill: false. Unlike SortExec (and the recent sort-merge-join spill work), CrossJoinExec has no SpillManager fallback.

Reproduction (observed via datafusion-comet, which delegates Spark CartesianProductExec to CrossJoinExec): a 1,000,000-row left range crossed with a 1-row right range under a ~4 MiB pool fails at its fair share, while engines that spill the buffered side (e.g. Spark's UnsafeCartesianRDD) complete and return the result. DataFusion 55.2.0's load_left_input confirms unconditional accumulation.

Describe the solution you'd like

Add a spill path to CrossJoinExec so the buffered left input can spill to disk when the reservation cannot grow, then re-read it from disk during probing — mirroring the approach SortExec and the sort-merge-join operator already use (SpillManager + on-disk runs). This would let cross joins over a large left input complete under memory pressure instead of erroring.

Describe alternatives you've considered

• Streaming the left and buffering/spilling the right instead (closer to Spark's model), though that changes which side is materialized.
• Keeping the current behavior but documenting the unbounded-left-buffer limitation.

Additional context

Found while adding native cross-join support in apache/datafusion-comet (apache/datafusion-comet#6938); Comet currently gates that path as opt-in specifically because of this non-spillable buffering. Happy to work on this.