Repository navigation
Decouple operator statistics propagation from traversal/caching via statistics_from_inputs #22958
Description
Is your feature request related to a problem or challenge?
The current statistics_with_args / StatisticsArgs design (#21815) embeds cache lookup
and child traversal directly inside each operator's statistics_with_args override via args.compute_child_statistics(...). This means:
- Each operator must be aware of caching mechanics, coupling local propagation logic to the traversal strategy
- Evolving the traversal or cache model requires touching every operator implementation
Describe the solution you'd like
Introduce a stateless statistics_from_inputs method on ExecutionPlan (defaulting to Statistics::new_unknown) that expresses only local propagation logic from pre-computed child statistics:
fn statistics_from_inputs(
&self,
input_stats: &[Arc<Statistics>],
partition: Option<usize>,
) -> Result<Arc<Statistics>> {
Ok(Arc::new(Statistics::new_unknown(self.schema().as_ref())))
}The external StatisticsContext owns traversal and cache management, calling statistics_from_inputs after resolving child statistics. statistics_with_args remains the public API and is unchanged.
Benefits
- Non-breaking:
statistics_from_inputshas a safe default - Operators that override
statistics_from_inputsautomatically benefit from any future improvements to the traversal/caching strategy without code changes - Operators become easier to test in isolation (no need to construct
StatisticsArgsor a plan tree)
Describe alternatives you've considered
Keep the current statistics_with_args design as-is. Each operator handles caching via args.compute_child_statistics(...). Works correctly but tightly couples propagation logic to traversal mechanics, making the cache model hard to evolve.
Additional context
Suggested by @2010YOUY01 in this comment as a follow-up for #21815
Activity
alamb commentedon Jun 16, 2026
Maybe we could add the children directly to StatisticsArgs?
We would have to add the traversal somehow -- perhaps as a visitor or something 🤔
asolimando commentedon Jun 17, 2026
Took a pass at this to make the discussion more concrete (currently stacked on #21815): asolimando/datafusion@asolimando/22958-statistics-from-inputs
Went with the shape @2010YOUY01 proposed, children passed as a dedicated arg rather than folded into
StatisticsArgs:StatisticsContextowns the traversal + cache (the "visitor" piece), which was previously part ofStatisticsArgsin the latest version of #21815.There are two choices I'd like your read on:
input_statsarg vs. insideStatisticsArgs, as I felt it was goo to keep "data vs config" separate (but it's not hard to merge them).statistics_with_argsoutright (it's unreleased so probably OK, we can keep that method name instead ofstatistics_from_inputsif you both agree on that).partition_statisticsstays deprecated as per Add StatisticsContext parameter to partition_statistics #21815.This can be a separate PR or fold into #21815, whichever you prefer.
refactor(physical-plan): externalize statistics traversal into Statis…
refactor(physical-plan): externalize statistics traversal into Statis…
refactor(physical-plan): externalize statistics traversal into Statis…