Repository navigation
Internal error: GROUP BY over UNION ALL fails when a branch projects coalesce(<nullable bool>, FALSE) (CSE rewrite is physically nullable) #25978
Description
Describe the bug
A GROUP BY aggregate over a UNION ALL fails to plan when one branch projects
coalesce(<nullable boolean expression>, FALSE):
Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences:
- field nullability at index 1 [f]: (physical) true vs (logical) false.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
The optimizer rewrites coalesce(x = 'y', FALSE) into __common_expr_1 IS NOT NULL AND __common_expr_1
(plan below). The rewritten expression is physically nullable, while the Union's logical schema keeps
the non-nullable type of coalesce(..., FALSE), so the aggregate's schema check fails.
Reproduces on datafusion-cli 55.1.0 (latest crates.io release) and 53.1.0, and on the
datafusion 54.0.0 Python wheel.
To Reproduce
CREATE TABLE a (k VARCHAR NOT NULL, x VARCHAR) AS VALUES ('k1', 'y'), ('k2', NULL);
CREATE TABLE b (k VARCHAR NOT NULL) AS VALUES ('k3');
SELECT k, bool_or(f) AS f FROM (
SELECT k, coalesce(x = 'y', FALSE) AS f FROM a
UNION ALL
SELECT k, TRUE AS f FROM b
) u GROUP BY k;datafusion-cli -f repro.sql raises the internal error above.
Variants, measured on 55.1.0:
| Variant | Result |
|---|---|
| As above | Internal error |
| The two branches swapped | Internal error |
No UNION (the coalesce branch alone), same GROUP BY k |
OK |
No aggregate above the UNION |
OK |
Global aggregate (count(f) without GROUP BY) |
OK |
coalesce(n, 0) on an INT column, or coalesce(x, 'z') on the text column |
OK |
x IS NOT NULL instead of the coalesce |
OK |
The plan without the aggregate (EXPLAIN SELECT k, f FROM (...) u, datafusion.explain.format = 'indent'):
logical_plan SubqueryAlias: u
Union
Projection: a.k, __common_expr_1 IS NOT NULL AND __common_expr_1 AS f
Projection: a.x = Utf8View("y") AS __common_expr_1, a.k
TableScan: a projection=[k, x]
Projection: b.k, Boolean(true) AS f
TableScan: b projection=[k]
physical_plan UnionExec
ProjectionExec: expr=[k@1 as k, __common_expr_1@0 IS NOT NULL AND __common_expr_1@0 as f]
ProjectionExec: expr=[x@1 = y as __common_expr_1, k@0 as k]
DataSourceExec: partitions=1, partition_sizes=[1]
ProjectionExec: expr=[k@0 as k, CAST(true AS Boolean) as f]
DataSourceExec: partitions=1, partition_sizes=[1]
With SET datafusion.optimizer.max_passes = 0 the same aggregate query fails differently:
Internal error: coalesce should have been simplified to case.
Expected behavior
The query plans and returns k1 → true, k2 → false, k3 → true.
Additional context
- Possibly the same family as Internal error: CSE-extracted
__common_expr_1has mismatched nullability between logical and physical plans when the expression is under an aggregate #25369 (a CSE-extracted expression with mismatched nullability
under an aggregate), but the trigger differs: the SQL contains no duplicated expression, the
failure needs aUNIONand aGROUP BY, and the mismatched field is the projected columnf
rather than__common_expr_1. The fix in progress there (fix: plan aggregates over CASE conditions with multiple columns #25684) addressesCASEconditions over
several columns. - Found while generating aggregate queries over unions of per-table rows. We work around it by
materializing the union before aggregating.
Activity
siddubakka commentedon Oct 8, 2026
take
siddubakka commentedon Oct 9, 2026
I have created a PR to resolve this issue: #26164.
cc @paul-heyse
fix: correct nullability of AND/OR expressions to resolve plan mismat…
fix: revert nullability changes and update test for issue apache#25978