You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use null counts when estimating IS NULL and IS NOT NULL #25620
A bare null check falls back to 20% selectivity even when input column
statistics contain a null count. IS NULL and IS NOT NULL receive the same
estimate.
cargo build --profile ci --locked -p datafusion-benchmarks --bin dfbench
repro_dir=$(mktemp -d)
mkdir -p "$repro_dir/data"
curl --fail --location \
https://raw.githubusercontent.com/apache/datafusion-benchmarks/32f67477f692453616d2fa98a05a37c5eb4cae49/tpcds/data/sf1/store_sales.parquet \
-o "$repro_dir/data/store_sales.parquet"
cat >"$repro_dir/repro.sql"<<'SQL'SET datafusion.execution.target_partitions = 1;SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NULL;SELECT ss_item_sk FROM store_sales WHERE ss_addr_sk IS NOT NULL;SQL
target/ci/dfbench statistics \
--path "$repro_dir/data" --query_path "$repro_dir/repro.sql"
Observed at this revision
with the pinned SF1 data above. Inspect the SELECT reports in order; ignore the
empty SET reports.
Predicate
FilterExec node
Estimated rows
Actual rows
IS NULL
0
576,081
129,975
IS NOT NULL
0
576,081
2,750,429
Expected behavior
Use the input null count for IS NULL and subtract it from the input row count
for IS NOT NULL, retaining the appropriate precision. Here those estimates would
be 129,975 and 2,750,429.
Additional context
Reduced from TPC-DS Q76. The input has 2,880,404 rows; a diagnostic capture
confirms its ss_addr_sk null count is available as Inexact(129975). Related: #24942 concerns removing
provably redundant null checks; this report concerns estimating checks that
actually remove rows.
Activity
fix: preserve uncertainty in singleton filter statistics (apache#25863)
fix: apply pushed-down fetch to filter statistics (apache#25862)
fix: preserve uncertainty in singleton filter statistics (apache#25863)
fix: apply pushed-down fetch to filter statistics (apache#25862)
fix: preserve uncertainty in singleton filter statistics (apache#25863)
fix: apply pushed-down fetch to filter statistics (apache#25862)