feat: let the Parquet scan skip optional filters that are not worth their cost - #25682

Draft

adriangb wants to merge 16 commits into

apache:mainfrom

pydantic:optional-filter-parquet

Commits on Sep 24, 2026

feat: add OptionalFilterPhysicalExpr to mark filters not needed for correctness

adriangb
and
claude
committed
c839c76
feat: add filter_stats (FilterCost, Clock) for filter measurements

adriangb
and
claude
committed
0337936
feat: add OptionalFilterGate to pause optional filters that cost more than they save

adriangb
and
claude
committed
62a11a7

Commits on Sep 25, 2026

feat(gate): add a measured overhead for each evaluated row to the cost

adriangb
and
claude
committed
3b8994a
feat(gate): share pauses between the gates of one plan site

adriangb
and
claude
committed
9755fd1
feat(gate): decide only on windows of at least MIN_OBSERVED_ROWS rows

adriangb
and
claude
committed
3ccb85e
feat(gate): use the measured work of the filter producer as the saving

adriangb
and
claude
committed
0c024cc
feat: add optional_filter_mode config

adriangb
and
claude
committed
b8fb779
feat: let the Parquet scan skip optional filters that are not worth their cost

adriangb
and
claude
committed
5becb4e

Commits on Sep 26, 2026

fix(parquet): count only skippable rows in the saving of an optional row filter

adriangb
and
claude
committed
9fdef55
feat(parquet): share the pauses of optional filter gates in a scan

adriangb
and
claude
committed
ee93043
test(parquet): gate windows of MIN_OBSERVED_ROWS rows in row filter tests

adriangb
and
claude
committed
7caf968
feat(parquet): order row filter predicates by measured rows removed per nanosecond

adriangb
and
claude
committed
9f36c9d
test(parquet): size the optional filter integration tests for gate windows

adriangb
and
claude
committed
f68a421

Commits on Sep 28, 2026

Merge apache/main into optional-filter-parquet

adriangb
and
claude
committed
214446d
fix(parquet): no scan equivalences from optional conjuncts

adriangb
and
claude
committed
38cf3e7