Repository navigation
Conversation
Owner
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600" |
github-actions
Bot
added
documentation
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
SIMULATE_LATENCY: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Owner
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600" |
Owner
Author
|
run benchmark tpch tpcds clickbench_partitioned env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpch
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark tpcds
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing combine-24086-25752 (07caaa3) to 1ae6b87 (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"
changed:
env:
DATAFUSION_EXECUTION_PARQUET_READ_AHEAD_BYTES: "104857600"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Each file of a scan made its own gates, thus each file paid for a window of each optional filter before the gate could pause it, and each gate probed on its own (TPC-H Q9: five join filters that remove no rows and a CASE routing filter at 83 ns for each row, in each of 12 files). `ParquetSource` now keeps an `OptionalFilterSites` map (reset when the predicate changes, like the decode speed): one `SharedGateVerdict` for each optional filter, keyed by the expression id of the filter. All gates of the filter in all files and partitions of the scan use it. A filter without an expression id does not share its pauses. PR: apache#25682 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne model The adaptive filter placement started each required conjunct in the post-scan filter, but it put each optional filter in the row filter. The row filter runs before the post-scan filter, thus the optional filters saw all rows first (TPC-DS Q13: a CASE routing filter on 2.69M rows, where the required filters leave 89K rows; ClickBench Q25: the TopK filter on the 87% of rows that `SearchPhrase <> ''` removes). Now required and optional conjuncts use the same model and the same pooled measurements. The only difference is the action set: an optional conjunct is skipped while its gate is paused. * Placement: each conjunct starts in the post-scan filter and moves to the row filter on measured evidence (skippable rows times unread output bytes against the stage cost, the fetch latency and the second decode). A conjunct that reads all output columns has no unread bytes, thus no decode saving. * Order: in each stage, the conjuncts run by measured rows removed for each nanosecond (`FilterCost::rows_removed_per_nano`, the key of the adaptive conjunct order of `FilterExec`). Unmeasured conjuncts run first in the written order, so that they are measured on all rows of their stage. The pooled statistics now include the evaluation time. The statistics prior is recorded for optional conjuncts too. * The post-scan filter asks the gate of an optional conjunct for each batch. The measured saving and stage cost of the gate apply only while the conjunct is a row filter predicate. * A placement change rebuilds the decoder only when its projection mask or its `RowFilter` changes. When only the post-scan filter changes (for example an optional conjunct on output columns that its gate pauses and probes), the stream swaps the post-scan filter. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The measurements are noisy near a decision boundary of the placement model, and each change can rebuild the decoder (128 placement changes in ClickBench Q25). After the n-th change of a file, the next change now waits for 2^(n-1) - 1 row group boundaries, thus a file whose decision flips again and again changes at most log2(row groups) + 1 times. This applies to all decisions of the model (placement and evaluation order) in the same way. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three measurement gaps of the placement model: * A conjunct in the post-scan filter sees only the rows that the row filter lets pass. When the row filter removes most rows, the conjunct never gets enough measurements for a better placement or order (ClickBench Q23: the TopK filter on `EventTime` behind the `URL LIKE` row filter saw about 1K of 100M rows, thus it stayed behind the expensive `LIKE`). A conjunct without enough measurements is now the first predicate of the row filter when the row filter is used, thus it is measured on all rows. This costs one row filter stage until the pooled measurements have `MIN_OBSERVED_ROWS` rows. * The pooled measurements of a conjunct with a dynamic filter described all generations of the filter together (for example the `true` placeholder of a TopK filter). They are now cleared when the generation of the filter increases. * For the same reason, the row group pruning result (the statistics prior, which overrides the measured evidence) is not recorded for a conjunct whose dynamic filters can still change: it describes the filter at file open. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ests The gate decides only on windows of at least `MIN_OBSERVED_ROWS` rows. PR: apache#25682 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…model The filter placement and the optional filter gates now use the same sample size (`filter_stats::MIN_OBSERVED_ROWS`). The tests of gated conjuncts use windows of this size. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tatistics prior The statistics prior (a conjunct that the row group statistics pruned in at least half of the row groups stays in the post-scan filter) overrode the measured evidence. A prior must yield to evidence. Before a conjunct has `MIN_OBSERVED_ROWS` measured rows, it is in the post-scan filter anyway, thus a prior that yields to evidence never changes a decision. This removes the prior from the placement model (`CLUSTERED_PRUNED_FRACTION`) and the pruning counts from the pooled statistics. The model does not need the per-conjunct pruning statistics of apache#25726 any more. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er nanosecond `order_candidates` put the gated optional predicates after all required predicates, and `prebuild_row_filter_candidates` built them in that order too. This is a rule for one kind of filter, not a measurement. Now all predicates of the `RowFilter` (without adaptive filter placement) use one rule: before the measurements, the written order of the predicate (or by `required_bytes` with `reorder_predicates`); once a predicate has `MIN_OBSERVED_ROWS` measured rows, by rows removed for each nanosecond (`FilterCost::rows_removed_per_nano`, the key of the adaptive conjunct order of `FilterExec`). An unmeasured predicate comes first, so that it is measured on all rows. Each predicate measures its rows and time in the file. At each row group boundary the scan rebuilds the `RowFilter` when the order changes, with hysteresis: after the n-th change, the next change waits for 2^(n-1) - 1 row group boundaries. `row_filter_cost` now has `rank`, `evaluation_order` and `ChangeHysteresis`, for the adaptive filter placement too. PR: apache#25682 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…filter order The adaptive filter placement and the `RowFilter` order without placement now use the same `row_filter_cost::rank`, `evaluation_order` and `ChangeHysteresis`, thus the same rule and the same sample size. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w filter The placement model compared the decode time that a row filter saves with the cost of a row filter stage, but the post-scan filter has a cost that a row filter does not have: after a selective conjunct it copies the working batch to the rows that pass (the compaction). The model did not see it, thus it kept selective conjuncts in the post-scan filter where a row filter was faster (TPC-DS Q72 and Q38 `catalog_sales`). The post-scan filter now times each copy and adds it to the measurements of the conjunct after which it copies. The model adds this time, for each row of the input batches of the post-scan filter, to the benefit of a row filter. There is no new constant: the term is measured. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ndows A gate decides only on windows of at least `MIN_OBSERVED_ROWS` rows, thus the tests use batches of 4096 rows (and more row groups for the test with 15-row row groups). The row group size is a multiple of the period of `a` and of the write batch size, so that the page index cannot prune a page with only a tail of the row group. PR: apache#25682 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… each batch The post-scan filter evaluated its conjuncts in the order of the placement, which changes only at a row group boundary. A file range with one or two row groups (TPC-DS SF1 `store_sales` has one file for 12 partitions) kept the written order until its end, before any pooled measurement: in TPC-DS Q24 the join filter on `ss_ticket_number` (a hash lookup, 12 ns for each row) ran on all rows before the filter on `ss_store_sk` (2 ns for each row) that removes all of them. Changing the post-scan order needs no decoder rebuild, thus the post-scan filter now orders its conjuncts at each batch with the same rule as the placement (`evaluation_order`: rows removed for each nanosecond, from the pooled measurements, unmeasured conjuncts first in their order). PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cargo doc` with `-D warnings` rejected a link to the removed `ConjunctSiteKey` and two redundant explicit link targets. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shold of AND The post-scan filter copied the working batch (all its columns) to the surviving rows when a conjunct kept at most 80% of them. The caller then copies the surviving rows again when it applies the final mask. For a cheap range predicate that keeps about half of the rows, the first copy costs much more than the evaluation that it saves on the next conjunct. TPC-DS Q82, `inventory` scan with `inv_quantity_on_hand BETWEEN 100 AND 500`, SF1, all 11.7M rows (EXPLAIN ANALYZE, 3 runs): | | post-scan filter eval | scan compute | |---|---|---| | threshold 0.8 | 28.4 to 29.2 ms | 104 to 106 ms | | threshold 0.2 | 4.5 to 4.6 ms | 74 to 78 ms | | main, `FilterExec` above the scan | 11.7 to 14.1 ms (`FilterExec`) | 64 to 66 ms | The threshold is now `PRE_SELECTION_THRESHOLD` (0.2), the threshold of `AND` in `BinaryExpr` that a `FilterExec` uses. Thus the post-scan filter makes the same copy decision as the `FilterExec` that it replaces. The old comment gave a `CASE` dynamic filter as the reason for 0.8; the partitioned hash join dynamic filters are no longer `CASE` expressions, and optional filters have gates and a measured order. Without a compaction, the working batch also has the rows that the earlier conjuncts removed. The measurements of a later conjunct (its placement statistics and its gate) now count these rows as passing, as they do after a compaction. Before, a later conjunct got the credit for rows that it did not remove. Wall time, min of 16 runs, pushdown on: TPC-DS Q82 1.04x -> 0.96x of main. TPC-H Q14 1.08x -> 0.94x, Q15 1.09x -> 1.00x, Q12 0.94x -> 0.84x. No other TPC-H, TPC-DS or ClickBench query changed outside the A/A noise. A threshold of 0 (never compact) is slower on TPC-H Q20 (1.14x) and TPC-DS Q42, Q52, Q55 (1.2x), thus the compaction stays. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests of apache#25729 (page index select-all) and the test of apache#25727 for the coalesced rows at row group boundaries show the TopK dynamic filter in the scan predicate. With apache#25681 the pushed filter is `Optional(DynamicFilter [...])`. Integration of apache#25729, apache#25727 and apache#25681 in the final state branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
apache#25780 on main added `FileSource::exact_filter`: the part of the filter that every output row satisfies, the only part that the scan derives equivalences from. Its `ParquetSource` version returns the pushable conjuncts when `pushdown_filters` is on, and nothing otherwise. This PR changes both cases: - A pruning-only predicate (a filter that stays in a `FilterExec` above a scan that cannot give the target partitions) is used only to prune, also with `pushdown_filters = true`. `exact_filter` returned it, thus the scan claimed that `a` is constant for `a = 5`, the order-preserving repartition merged on `b` only, and `ORDER BY b LIMIT 1` returned 2 instead of 1. It now returns `None`. - With `pushdown_filters = false` the scan applies the accepted conjuncts in the post-scan filter. They are exact, thus `exact_filter` returns them. This keeps the plans of this PR (for example no `SortExec` for `ORDER BY b` with `b = 2`). Tests: a new case in `push_down_filter_parquet.slt` (a plan pin and two results that were wrong: `2` for `LIMIT 1`, and `5 2 / 5 1` for the order) and `exact_filter` checks in the pruning-only unit test. The plan of the apache#25780 case with `pushdown_filters = false` changes: the scan applies `a = 5` and there is no `FilterExec`. PR: apache#22384 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ParquetSource::exact_filter` (apache#25780) returns the conjuncts that every output row satisfies. An optional conjunct is not one of them: the scan does not evaluate it after the decode (this PR), and it drops it when the `RowFilter` cannot evaluate it. `exact_filter` now skips optional conjuncts, so that the scan claims no equivalence from them. Test: `exact_filter_excludes_optional_conjuncts` (failed before this change: `a@0 = 1 AND Optional(b@1 = 2)` with `pushdown_filters = false`). PR: apache#25722 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ParquetSource::exact_filter` (apache#25780) returns the conjuncts that every output row satisfies. An optional conjunct is not one of them: in the `adaptive` mode its gate can skip it, in the `pruning_only` mode the scan does not evaluate it, and the scan drops it when the `RowFilter` cannot evaluate it. `exact_filter` now skips optional conjuncts in all modes, so that the scan claims no equivalence from them. Test: `exact_filter_excludes_optional_conjuncts` (failed before this change: `a@0 = 1 AND Optional(b@1 = 2)`). PR: apache#25682 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With adaptive filter placement, an optional conjunct can be in the post-scan filter (its gate asks for each batch) or skipped. Say so in the documentation of `ParquetSource::exact_filter`, which already excludes optional conjuncts. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new `parquet_statistics.slt` case of apache#25795 on main pins a `FilterExec` above the scan. With apache#22384 the scan accepts the filter (one file of two rows keeps the filter in the scan), thus the plan is the scan alone. Its statistics are still `Rows=Inexact(2)`, not empty, and the query result does not change. Integration of apache#22384 and main in the final state branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e default Change two defaults: - `datafusion.execution.optional_filter_mode` = `adaptive` (was `always`). The Parquet scan and `FilterExec` pause optional filters (hash join, TopK and aggregate dynamic filters) when they remove no rows or cost more than they save. - `datafusion.execution.adaptive_filter_placement` = `true` (was `false`). With `pushdown_filters = true`, each filter conjunct starts after the decode, and the Parquet scan makes it a row filter only when the measurements show that this saves more than it costs. The Parquet scan uses the two options only when `datafusion.execution.parquet.pushdown_filters` is true. `ParquetSource::new` keeps adaptive filter placement off, because the source has no setter for it: only `try_pushdown_filters` applies the session value. Tests that check exact row filter metrics (predicate cache, runtime row group pruning, optional filters in the row filter, EXPLAIN ANALYZE categories) set the previous values, because the adaptive decisions use time measurements, and because adaptive filter placement coalesces small batches, which delays TopK dynamic filters on tiny row groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The previous commit made the scan hand the coalesced rows to the TopK at each row group boundary. Thus the runtime row group pruning tests do not need the previous defaults any more: - `dynamic_row_group_pruning.slt` runs with the defaults. A new case checks that the scan prunes the same row groups without adaptive filter placement. - The Parquet test harness keeps `optional_filter_mode = 'always'`, because its tests check exact metrics, but uses the default adaptive filter placement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the adaptive filter placement, an optional filter starts in the post-scan filter like a required conjunct, thus the `EXPLAIN ANALYZE` metrics move from `pushdown_rows_*` and `predicate_cache_*` to `post_scan_rows_*`. The results do not change. PR: apache#25727 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
combine-24086-25752
branch
from
October 5, 2026 02:05
07caaa3 to
2e04ab8
Compare
… read-ahead Opt-in with datafusion.execution.parquet.read_ahead_bytes (default unset). If set, the scan builds the same ParquetPushDecoder as the default path with FetchGranularity::Batch and drives it with try_decode. ReadAhead fetches ranges from scan_plan() in the background within that many bytes. datafusion.execution.parquet.read_ahead_conditional (default false) also reads ahead ranges that a pushed-down filter can make unnecessary. Filtered scans, runtime row-group pruning and the per-row-group filter toggle use the same path as main. Squashed from the history in branch claude/datafusion-rowgroup-buffering-history (pydantic/datafusion). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ranges Skipping ranges that a pushed-down filter can make unnecessary made filtered scans on object storage 2-16x slower. Read-ahead now fetches them, bounded by the read-ahead window. Proto field 40 is reserved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each read-ahead stream registers a `ParquetReadAhead[partition]` consumer. The reservation follows the bytes the decoder holds plus the bytes in flight. Bytes the decoder asks for are always reserved (grow). Speculative read-ahead takes only what the pool can grant (try_grow); ranges that do not fit stay pending. `FileSource::create_morselizer_with_context` gives the source the scan's `TaskContext`. The default delegates to `create_morselizer`. `ParquetSource` uses it to get the memory pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # datafusion/datasource-parquet/src/opener/mod.rs # datafusion/datasource-parquet/src/push_decoder.rs # datafusion/datasource-parquet/src/source.rs
adriangb
force-pushed
the
combine-24086-25752
branch
from
October 5, 2026 02:11
2e04ab8 to
1201e86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines apache#24086 (batch-granular Parquet scans with
read_ahead_bytes) and apache#25752 (optional filters stack) for benchmarking. Do not merge.The merge resolves conflicts in
datafusion/datasource-parquet/src/push_decoder.rs: the read-ahead path (transition_streaming) and the default path share the post-scan filter, coalescer, limit and row group boundary logic (pending_output,push_decoded_batch,handle_row_group_boundary).Local checks:
datafusion-datasource-parquetunit tests pass, parquet sqllogictests pass, and the full sqllogictest suite withread_ahead_bytesforced to 64 KiB returns the same query results (only predicate cache metrics differ).🤖 Generated with Claude Code