Skip to content

Native element_at and map_from_arrays skip a shared subexpression on NULL rows under ANSI #6970

Open

Description

@dwsmith1983

Describe the bug

With ANSI mode on and subexpression elimination on (its default), Spark's projection evaluates a subexpression that several of its expressions share for every row, before any of their NULL checks. The native element_at on a map and the native map_from_arrays wrap their work in a CASE WHEN <input> IS NOT NULL guard, so on a row whose map or key array is NULL the shared subexpression is never evaluated. When that subexpression raises, Spark fails the query and Comet returns rows.

The guards are in CometElementAt.convert (the needsNullGuard branch in arrays.scala) and CometMapFromArrays.convert (maps.scala). Neither consults QueryPlanSerde.canShortCircuitNulls, which withNullShortCircuit uses to keep the arguments eager when a later one holds an operator-shared subexpression. GetMapValue goes through withNullShortCircuit and already matches Spark.

Found in #5867 while answering #5867 (comment), which reported the same gap for the dispatched lookups that PR adds.

Steps to reproduce

On main (7d8d9b1), as a SQL file test:

-- Config: spark.sql.ansi.enabled=true
-- Config: spark.sql.subexpressionElimination.enabled=true

statement
CREATE TABLE t(m1 map<int, int>, m2 map<int, int>, k1 array<int>, k2 array<int>, x double, i int)
USING parquet

statement
INSERT INTO t VALUES
  (CAST(NULL AS MAP<INT, INT>), CAST(NULL AS MAP<INT, INT>), CAST(NULL AS ARRAY<INT>),
   CAST(NULL AS ARRAY<INT>), 0.0D, 0),
  (map(1, 5), map(1, 6), array(1), array(2), 1.0D, 1)

query expect_error(DIVIDE_BY_ZERO)
SELECT element_at(m1, 1 div i), element_at(m2, 1 div i) FROM t

query expect_error(DIVIDE_BY_ZERO)
SELECT element_at(m1, CAST(1.0D / x AS INT)), element_at(m2, CAST(1.0D / x AS INT)) FROM t

query expect_error(DIVIDE_BY_ZERO)
SELECT map_from_arrays(k1, array(1.0D / x)), map_from_arrays(k2, array(1.0D / x)) FROM t

Run one per file (a file stops at its first failure), each of the three queries fails with "Expected Comet to throw an error matching 'DIVIDE_BY_ZERO' but query succeeded". The same shape with m1[1 div i], m2[1 div i] passes.

Expected behavior

Comet raises DIVIDE_BY_ZERO like Spark. One way is to build these guards through withNullShortCircuit, or to check canShortCircuitNulls before adding them, so a shared subexpression stays evaluated for every row.

Additional context

The dispatched paths that #5867 adds decline the same shapes and fall back to Spark. Existing JVM codegen dispatch serdes on main may have a similar gap. I have not reproduced that yet.

Activity

  1. dwsmith1983 commented on Oct 14, 2026

    @dwsmith1983
    ContributorAuthor

    take

  2. dwsmith1983 commented on Oct 14, 2026

    @dwsmith1983
    ContributorAuthor

    More from working on this:

    • element_at over a nullable array has the same gap, for example element_at(a1, CAST(1 div i AS INT)), element_at(a2, CAST(1 div i AS INT)).
    • Evaluating the key eagerly through withNullShortCircuit, as GetMapValue does, fixes these shapes but makes map_from_arrays raise where Spark does not when the arguments share only a harmless piece. The existing map_from_arrays short-circuits null keys test in CometMapExpressionSuite fails that way: the keys and values share k = 0, Spark hoists only that, and still skips 1 / k on the rows whose keys are NULL. It also raises with subexpression elimination off, which is Native null short-circuit evaluates shared arguments eagerly when subexpression elimination is off #6972.

    So the plan is to decline these shapes and fall back to Spark instead, using the check #5867 adds for dispatched expressions, in the same change as #6971 once #5867 is in.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions