You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native element_at and map_from_arrays skip a shared subexpression on NULL rows under ANSI #6970
With ANSI mode on and subexpression elimination on (its default), Spark's projection evaluates a subexpression that several of its expressions share for every row, before any of their NULL checks. The native element_at on a map and the native map_from_arrays wrap their work in a CASE WHEN <input> IS NOT NULL guard, so on a row whose map or key array is NULL the shared subexpression is never evaluated. When that subexpression raises, Spark fails the query and Comet returns rows.
The guards are in CometElementAt.convert (the needsNullGuard branch in arrays.scala) and CometMapFromArrays.convert (maps.scala). Neither consults QueryPlanSerde.canShortCircuitNulls, which withNullShortCircuit uses to keep the arguments eager when a later one holds an operator-shared subexpression. GetMapValue goes through withNullShortCircuit and already matches Spark.
Found in #5867 while answering #5867 (comment), which reported the same gap for the dispatched lookups that PR adds.
-- Config: spark.sql.ansi.enabled=true-- Config: spark.sql.subexpressionElimination.enabled=true
statement
CREATETABLEt(m1 map<int, int>, m2 map<int, int>, k1 array<int>, k2 array<int>, x double, i int)
USING parquet
statement
INSERT INTO t VALUES
(CAST(NULLAS MAP<INT, INT>), CAST(NULLAS MAP<INT, INT>), CAST(NULLAS ARRAY<INT>),
CAST(NULLAS ARRAY<INT>), 0.0D, 0),
(map(1, 5), map(1, 6), array(1), array(2), 1.0D, 1)
query expect_error(DIVIDE_BY_ZERO)
SELECT element_at(m1, 1 div i), element_at(m2, 1 div i) FROM t
query expect_error(DIVIDE_BY_ZERO)
SELECT element_at(m1, CAST(1.0D / x ASINT)), element_at(m2, CAST(1.0D / x ASINT)) FROM t
query expect_error(DIVIDE_BY_ZERO)
SELECT map_from_arrays(k1, array(1.0D / x)), map_from_arrays(k2, array(1.0D / x)) FROM t
Run one per file (a file stops at its first failure), each of the three queries fails with "Expected Comet to throw an error matching 'DIVIDE_BY_ZERO' but query succeeded". The same shape with m1[1 div i], m2[1 div i] passes.
Expected behavior
Comet raises DIVIDE_BY_ZERO like Spark. One way is to build these guards through withNullShortCircuit, or to check canShortCircuitNulls before adding them, so a shared subexpression stays evaluated for every row.
Additional context
The dispatched paths that #5867 adds decline the same shapes and fall back to Spark. Existing JVM codegen dispatch serdes on main may have a similar gap. I have not reproduced that yet.
element_at over a nullable array has the same gap, for example element_at(a1, CAST(1 div i AS INT)), element_at(a2, CAST(1 div i AS INT)).
Evaluating the key eagerly through withNullShortCircuit, as GetMapValue does, fixes these shapes but makes map_from_arrays raise where Spark does not when the arguments share only a harmless piece. The existing map_from_arrays short-circuits null keys test in CometMapExpressionSuite fails that way: the keys and values share k = 0, Spark hoists only that, and still skips 1 / k on the rows whose keys are NULL. It also raises with subexpression elimination off, which is Native null short-circuit evaluates shared arguments eagerly when subexpression elimination is off #6972.
So the plan is to decline these shapes and fall back to Spark instead, using the check #5867 adds for dispatched expressions, in the same change as #6971 once #5867 is in.
Activity
dwsmith1983 commentedon Oct 14, 2026
take
dwsmith1983 commentedon Oct 14, 2026
More from working on this:
element_atover a nullable array has the same gap, for exampleelement_at(a1, CAST(1 div i AS INT)), element_at(a2, CAST(1 div i AS INT)).withNullShortCircuit, asGetMapValuedoes, fixes these shapes but makesmap_from_arraysraise where Spark does not when the arguments share only a harmless piece. The existingmap_from_arrays short-circuits null keystest inCometMapExpressionSuitefails that way: the keys and values sharek = 0, Spark hoists only that, and still skips1 / kon the rows whose keys are NULL. It also raises with subexpression elimination off, which is Native null short-circuit evaluates shared arguments eagerly when subexpression elimination is off #6972.So the plan is to decline these shapes and fall back to Spark instead, using the check #5867 adds for dispatched expressions, in the same change as #6971 once #5867 is in.