feat: adaptive placement of Parquet scan filters (row filter, post-scan, skip) - #25727

Draft

adriangb wants to merge 45 commits into

apache:mainfrom

pydantic:parquet-adaptive-placement-v2

Commits on Sep 24, 2026

feat(parquet-datasource): always accept pushable filters, run rejected conjuncts post-scan

adriangb
and
claude
committed
fcb0470
Update tests added on main for always-accepted parquet filters

adriangb
and
claude
committed
22ce8bf

Commits on Sep 25, 2026

fix: stop the Parquet push decoder stream after the final flush

adriangb
and
claude
committed
31199fc

Commits on Sep 26, 2026

refactor(physical-plan): share the round-robin row check of EnforceDistribution

adriangb
and
claude
committed
c2d8f85
fix(datasource): keep a filter above a scan that cannot give the target partitions

adriangb
and
claude
committed
9ff1146
feat: add OptionalFilterPhysicalExpr to mark filters not needed for correctness

adriangb
and
claude
committed
fe6aa04
feat: keep optional filters pruning-only in the Parquet post-scan path

adriangb
and
claude
committed
2246afc
feat: add filter_stats (FilterCost, Clock) for filter measurements

adriangb
and
claude
committed
ab2b6e2
feat: add OptionalFilterGate to pause optional filters that cost more than they save

adriangb
and
claude
committed
ab74ffa
feat(gate): add a measured overhead for each evaluated row to the cost

adriangb
and
claude
committed
6ae18bc
feat(gate): share pauses between the gates of one plan site

adriangb
and
claude
committed
ac838c4
feat(gate): decide only on windows of at least MIN_OBSERVED_ROWS rows

adriangb
and
claude
committed
36c73a0
feat(gate): use the measured work of the filter producer as the saving

adriangb
and
claude
committed
a352cb7
feat: add optional_filter_mode config

adriangb
and
claude
committed
ded1dd6
feat: let the Parquet scan skip optional filters that are not worth their cost

adriangb
and
claude
committed
0a51a93
test: end-to-end tests for optional filters in the Parquet post-scan path

adriangb
and
claude
committed
de1b263
feat(parquet): adaptive placement of scan filter conjuncts

adriangb
and
claude
committed
e616293
test: sqllogictests for adaptive filter placement

adriangb
and
claude
committed
1fbda42
feat(parquet): carry row selections across placement rebuilds

adriangb
and
claude
committed
b0088dc
feat(parquet): measure skippable rows on the stage input

adriangb
and
claude
committed
98259e5
feat(parquet): start filters post-scan and add a row filter stage cost

adriangb
and
claude
committed
ed19bdc
fix(parquet): hand coalesced rows downstream at row group boundaries

adriangb
and
claude
committed
9b24e9b
fix(parquet): count the actual rows in optional_filter_rows_skipped

adriangb
and
claude
committed
824dc27
fix(parquet): count only skippable rows in the saving of an optional row filter

adriangb
and
claude
committed
49b7bb1
feat(parquet): share the pauses of optional filter gates in a scan

adriangb
and
claude
committed
9af896f
feat(parquet): place and order required and optional conjuncts with one model

adriangb
and
claude
committed
e3fb0f0
feat(parquet): add hysteresis to the placement changes of a file

adriangb
and
claude
committed
aa00419
feat(parquet): measure unmeasured conjuncts first and forget old filters

adriangb
and
claude
committed
30ec574
test(parquet): gate windows of MIN_OBSERVED_ROWS rows in row filter tests

adriangb
and
claude
committed
3f9098e
refactor(parquet): use the shared MIN_OBSERVED_ROWS in the placement model

adriangb
and
claude
committed
15f1b67
feat(parquet): let measured evidence decide the placement, drop the statistics prior

adriangb
and
claude
committed
33a188a
feat(parquet): order row filter predicates by measured rows removed per nanosecond

adriangb
and
claude
committed
86531c1
refactor(parquet): share the ranking and the hysteresis with the row filter order

adriangb
and
claude
committed
0322fd6
feat(parquet): add the measured post-scan copy to the benefit of a row filter

adriangb
and
claude
committed
102627d
test(parquet): size the optional filter integration tests for gate windows

adriangb
and
claude
committed
b964d74
feat(parquet): order the post-scan conjuncts by their measurements at each batch

adriangb
and
claude
committed
48be981
fix(parquet): fix rustdoc links of the adaptive filter placement

adriangb
and
claude
committed
e6a798e
refactor(physical-expr): make PRE_SELECTION_THRESHOLD public

adriangb
and
claude
committed
4a3820e
fix(parquet): compact the post-scan batch with the pre-selection threshold of AND

adriangb
and
claude
committed
2627130

Commits on Sep 28, 2026

Merge apache/main into parquet-post-scan-filter

adriangb
and
claude
committed
8590f0c
fix(parquet): no scan equivalences from a pruning-only predicate

adriangb
and
claude
committed
a9aa655
Merge the current #22384 into parquet-post-scan-optional

adriangb
and
claude
committed
07096a0
fix(parquet): no scan equivalences from optional conjuncts

adriangb
and
claude
committed
4cd30d7
Merge the current #25722 into parquet-adaptive-placement-v2

adriangb
and
claude
committed
a11aa41
docs(parquet): optional conjuncts of the placement are not exact

adriangb
and
claude
committed
bd867a0