Describe the bug
An accurate filter estimate becomes a 1.45-million-fold error at aggregation:
input rows are reused as the number of groups when grouping keys lack explicit
NDVs. Grouping by a derived year has the same fallback.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT l_returnflag, l_linestatus, COUNT(*) FROM lineitem WHERE l_shipdate <= DATE '1998-09-02' GROUP BY l_returnflag, l_linestatus;
SELECT EXTRACT(YEAR FROM l_shipdate) AS y, COUNT(*) FROM lineitem GROUP BY EXTRACT(YEAR FROM l_shipdate);
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT |
Node |
Estimated groups |
Actual groups |
| Return flag/status |
AggregateExec, 0.0 |
5,787,395 |
4 |
| Shipping year |
AggregateExec, 0.0 |
6,001,215 |
7 |
Expected behavior
Use available group-domain or expression statistics instead of immediately
falling back to input rows. Keep uncertain range bounds distinct from measured
NDVs.
Additional context
The first aggregate’s filter input estimates 5,787,395 rows versus 5,916,591
actual, within 3%. Related:
NDV epic #20766.
Part of #25610.
Describe the bug
An accurate filter estimate becomes a 1.45-million-fold error at aggregation:
input rows are reused as the number of groups when grouping keys lack explicit
NDVs. Grouping by a derived year has the same fallback.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
Observed with
tpchgen-cli1.1.1 at6c320561b5.
Inspect the SELECT reports; ignore the empty
SETreports.AggregateExec,0.0AggregateExec,0.0Expected behavior
Use available group-domain or expression statistics instead of immediately
falling back to input rows. Keep uncertain range bounds distinct from measured
NDVs.
Additional context
The first aggregate’s filter input estimates 5,787,395 rows versus 5,916,591
actual, within 3%. Related:
NDV epic #20766.
Part of #25610.