Describe the bug
Adding an unsupported AND condition discards the useful estimate for the
supported condition. The extra restriction increases the estimate tenfold
because the whole predicate falls back to 20% selectivity.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
cargo install tpchgen-cli --version 1.1.1 --locked # if not already installed
repro_dir=$(mktemp -d)
tpchgen-cli --scale-factor 1 --format parquet \
--parquet-compression 'ZSTD(1)' --parts 1 --output-dir "$repro_dir/data"
cat > "$repro_dir/repro.sql" <<'SQL'
SET datafusion.execution.target_partitions = 1;
SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
SELECT p_partkey FROM part WHERE p_size = 15;
SELECT p_partkey FROM part WHERE p_size = 15 AND p_type LIKE '%BRASS%';
SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed with tpchgen-cli 1.1.1 at
6c320561b5.
Inspect the SELECT reports; ignore the empty SET reports.
| SELECT |
FilterExec node |
Estimated rows |
Actual rows |
p_size = 15 |
0 |
4,000 |
3,907 |
Add AND p_type LIKE '%BRASS%' |
0 |
40,000 |
747 |
Expected behavior
Retain supported conjunct estimates and combine unsupported conjuncts
conservatively. Adding an AND condition should not increase estimated rows; OR
requires different treatment.
Additional context
Related:
filter selectivity #14237.
Part of #25610.
Describe the bug
Adding an unsupported AND condition discards the useful estimate for the
supported condition. The extra restriction increases the estimate tenfold
because the whole predicate falls back to 20% selectivity.
To Reproduce
From the repository root, with the CLI fix from
PR #25570 applied:
Observed with
tpchgen-cli1.1.1 at6c320561b5.
Inspect the SELECT reports; ignore the empty
SETreports.p_size = 150AND p_type LIKE '%BRASS%'0Expected behavior
Retain supported conjunct estimates and combine unsupported conjuncts
conservatively. Adding an AND condition should not increase estimated rows; OR
requires different treatment.
Additional context
Related:
filter selectivity #14237.
Part of #25610.