Repository navigation
Conversation
github-actions
Bot
added
documentation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #26276 +/- ##
==========================================
+ Coverage 82.87% 82.89% +0.01%
==========================================
Files 1153 1155 +2
Lines 457101 458241 +1140
Branches 457101 458241 +1140
==========================================
+ Hits 378818 379838 +1020
- Misses 55268 55292 +24
- Partials 23015 23111 +96 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
adriangb
force-pushed
the
mcte-auto-materialize
branch
2 times, most recently
from
October 16, 2026 03:46
2490f15 to
2de017a
Compare
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 16, 2026 14:58
2de017a to
26d5165
Compare
Replace the `MaterializedCte` and `MaterializedCteScan` extension nodes with the variants `LogicalPlan::MaterializedCte` and `LogicalPlan::MaterializedCteScan`. Each rule now states what it does with the nodes in its `match`, and the downcasts are removed from `PushDownFilter`, `PullUpCorrelatedExpr`, `head_output_expr` and the physical planner. `MaterializedCte::try_new` checks that the scans have the schema of the body, and `MaterializedCteScan::find_in` finds the scans of one CTE, also in subquery expressions. The unparser, the Substrait producer and the protobuf serializer return a "not implemented" error for the two variants. Plans and results do not change. Closes apache#26238. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 16, 2026 16:57
26d5165 to
644524b
Compare
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 16, 2026 20:39
644524b to
8575777
Compare
Add the option `datafusion.optimizer.enable_automatic_cte_materialization`, which is off by default. With the default, each query plans as before. When the option is on, the SQL planner plans each non-recursive CTE that has no directive as a `MaterializedCte` with `automatic = true`, in each dialect. The new optimizer rule `InlineAutomaticCtes` replaces an automatic node with `MaterializedCte::inline` when the CTE has zero or one reference, or when its body is volatile, has an outer reference, or is only projections and filters on one table, or when a `LIMIT` with a fetch is above all references in the chain of one-input nodes from the root of the query. All other automatic CTEs stay materialized and are computed one time. `AS MATERIALIZED` and `AS NOT MATERIALIZED` override the rule. The physical planner inlines an automatic CTE that has an unbounded body, because such a body cannot be buffered and the query worked before. Closes apache#26237.
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 16, 2026 21:20
8575777 to
4f0729c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Stacked on #26269. Only the last commit is new.
Rationale for this change
A CTE that has no directive is inlined at each reference, so its body runs one time for each reference:
With the option off, which is the default,
tis aggregated two times. With the option on, the body is computed one time:This works in each SQL dialect. A CTE with one reference, or with a cheap body, plans exactly as before.
What changes are included in this PR?
datafusion.optimizer.enable_automatic_cte_materialization, defaultfalse. With the default, each query plans as before.datafusion.optimizer.automatic_cte_materialization_keeps_volatile, defaultfalse. See rule 2 below.MaterializedCtehas the new fieldautomatic. When the option is on, the SQL planner plans each non-recursive CTE that has no directive as an automaticMaterializedCte.InlineAutomaticCtes, the first rule of the optimizer. It replaces an automatic node with a copy of the body in place of each scan when one of these conditions is true:datafusion.optimizer.automatic_cte_materialization_keeps_volatileisfalse(the default). Each reference then evaluates the expression again, as with the main option off (Avoid inlining non deterministic CTE #10337). With the second optiontrue, the CTE stays materialized and each reference reads the same rows. This is what PostgreSQL and DuckDB do.VALUESlist or empty relation, without a subquery expression.AS MATERIALIZEDandAS NOT MATERIALIZEDoverride the rule.WITH RECURSIVEblock and a CTE with a correlated body are inlined by the SQL planner, as before.SELECT INTOwith an automatic CTE inlines the CTE.create_initial_plancatches; a different mechanism is possible.Known limits:
EXPLAINshows theMaterializedCtenode in the logical plan and the inlined query in the physical plan, and the inlined copies do not get logical filter pushdown.MaterializedCtenodes for each CTE. The unparser and the Substrait producer return a "not implemented" error for these nodes.The rule also inlines an automatic CTE when a
LIMITwith a fetch is above all of its references. The rule finds such a limit by following the nodes with one input from the root of the query, and it stops at a node with a subquery expression. A materialized body is computed completely before any reference reads a row, so a limit above the references cannot stop it early. Inlined copies stop when the limit has its rows. This follows theContainsLimitcheck of DuckDB, with one difference: a sort with a fetch (TopK) does not count, because TopK reads all of its input, and inlining would then compute the body more than one time. A limit above only some references, for example above one side of a join, keeps the CTE materialized. A limit inside the CTE body also keeps it materialized.Comparison with PostgreSQL and DuckDB. PostgreSQL inlines a CTE with no directive only when it has one reference and no volatile functions, and it does not use cost (subselect.c). DuckDB materializes by default since v1.4.0. It keeps a CTE with a volatile body or an aggregate body materialized, and it inlines a cheap body (cte_inlining.cpp). The rules of this PR are closest to DuckDB's. Two things differ: a volatile body is inlined unless
automatic_cte_materialization_keeps_volatileis set, and a "cheap" body also includes filters and projections on one table, because DataFusion pushes those into the scan.Are these changes tested?
Yes. New file
cte_auto_materialize.slt: two references to an aggregate body, one reference, no reference, cheap body, volatile body (with each value ofautomatic_cte_materialization_keeps_volatile), scan in a subquery expression, CTE read by another CTE, correlated body, recursive CTE,SELECT INTO, the two directives, the MySQL dialect, an unbounded body, and aLIMITabove all references, above one side of a join, and none. Unit tests forInlineAutomaticCtesand forMaterializedCte::inline. The full sqllogictest suite also passes with the default of the option set totrue, except for the tests that show the default value.Are there any user-facing changes?
Yes. There are two new configuration options, which are off by default, and a new optimizer rule.
MaterializedCtehas the new public fieldautomaticand the new methodswith_automatic,is_automaticandinline. The upgrade guide for 56.0.0 and the configuration documentation are updated.