Skip to content

feat: decide automatically when to materialize a CTE #26276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adriangb wants to merge 2 commits into
base: main
Choose a base branch
from
+2,296 −356
Draft

feat: decide automatically when to materialize a CTE #26276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
37c5b98
feat: make the materialized CTE nodes a first-class LogicalPlan variant
adriangb Oct 16, 2026
4f0729c
feat: decide automatically when to materialize a CTE
adriangb Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions datafusion/common/src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1664,6 +1664,28 @@ config_namespace! {
/// past window functions, if possible
pub enable_window_limits: bool, default = true

/// When set to true, the SQL planner plans each non-recursive CTE that
/// has no `MATERIALIZED` or `NOT MATERIALIZED` directive as a
/// materialized CTE, and the optimizer rule `inline_automatic_ctes`
/// decides which of them stay materialized. A CTE stays materialized,
/// and is computed one time, when it has more than one reference and
/// its body is more than projections and filters on one table, and no
/// `LIMIT` is above all of its references. A CTE
/// with a volatile body is inlined, unless
/// `automatic_cte_materialization_keeps_volatile` is true. All other
/// CTEs are inlined at each reference.
/// When set to false, a CTE is materialized only with `AS MATERIALIZED`.
pub enable_automatic_cte_materialization: bool, default = false

/// When set to true, together with
/// `enable_automatic_cte_materialization`, a CTE with a volatile body
/// and more than one reference stays materialized, so that each
/// reference reads the same rows. PostgreSQL and DuckDB do the same.
/// When set to false, such a CTE is inlined at each reference, and
/// each reference evaluates the volatile expressions again, as with
/// `enable_automatic_cte_materialization` set to false.
pub automatic_cte_materialization_keeps_volatile: bool, default = false

/// When set to true, the optimizer will replace
/// Filter(rn<=K) → Window(ROW_NUMBER) → Sort patterns with a
/// PartitionedTopKExec that maintains per-partition heaps, avoiding
Expand Down
53 changes: 27 additions & 26 deletions datafusion/core/src/optimizer_rule_reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,32 +37,33 @@ Rule order matters. The default pipeline may change between releases.

| order | rule | summary |
| ----- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| 1 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |
| 2 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |
| 3 | `unions_to_filter` | Merges `UNION DISTINCT` branches that share the same source into a single filtered branch with a disjunctive predicate. |
| 4 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |
| 5 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |
| 6 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |
| 7 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |
| 8 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |
| 9 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |
| 10 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |
| 11 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |
| 12 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |
| 13 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |
| 14 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |
| 15 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |
| 16 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |
| 17 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |
| 18 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits. |
| 19 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
| 20 | `eliminate_aggregate_distinct` | Drops the `DISTINCT` modifier from aggregates whose result cannot change, such as `min`, `max` and `bit_or`. |
| 21 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
| 22 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
| 23 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
| 24 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
| 25 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
| 26 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
| 1 | `inline_automatic_ctes` | Inlines the automatic materialized CTEs that have at most one reference, or a volatile or cheap body. |
| 2 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |
| 3 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |
| 4 | `unions_to_filter` | Merges `UNION DISTINCT` branches that share the same source into a single filtered branch with a disjunctive predicate. |
| 5 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |
| 6 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |
| 7 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |
| 8 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |
| 9 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |
| 10 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |
| 11 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |
| 12 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |
| 13 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |
| 14 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |
| 15 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |
| 16 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |
| 17 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |
| 18 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |
| 19 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits. |
| 20 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
| 21 | `eliminate_aggregate_distinct` | Drops the `DISTINCT` modifier from aggregates whose result cannot change, such as `min`, `max` and `bit_or`. |
| 22 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
| 23 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
| 24 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
| 25 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
| 26 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
| 27 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |

### Physical Optimizer Rules

Expand Down
Loading
Loading