Skip to content

DuckDB IEJoin silently drops a catalog/schema qualifier on a projection/GROUP BY/ORDER BY column (miscompiles a query the naive plan rejects) #206

Description

@conradbzura

Description

Under dialect="duckdb", a column-to-column INTERSECTS join whose SELECT list, aggregate argument, GROUP BY, or ORDER BY carries a catalog/schema-qualified column (memory.main.a.name, db1.a.name) silently miscompiles: the IEJoin engages and returns rows, but the rewrite keys only on the column's table alias (a) and drops the catalog/schema qualifier. The naive plan (dialect=None / "datafusion") — and raw DuckDB — reject the same query with a binder error, so dialect="duckdb" invents results for a query every other backend refuses.

This is the exact hazard the extra-predicate path already guards against (_validate_extra_qualifiers raises for a mycat.myschema.a.col extra predicate, "the alias rewriter would leave the catalog/schema qualifier intact in the inner subquery, where it references a different relation than intended"), but the projection / aggregate / GROUP BY / ORDER BY surfaces have no equivalent guard.

Reproduction (executed on DuckDB 1.5.4; peaks/genes have columns chrom, start, end, name, score, strand):

from giql import transpile
import duckdb

J = "FROM peaks a JOIN genes b ON a.interval INTERSECTS b.interval"
# all four ENGAGE the IEJoin and return rows; the naive plan raises
#   BinderException: Referenced table "memory.main.a" not found
q1 = f"SELECT memory.main.a.name {J}"                              # -> dd:[('p1',),('p4',)]  naive:error
q2 = f"SELECT SUM(memory.main.a.score) AS s {J}"                   # -> dd:[(37,)]            naive:error
q3 = f"SELECT a.chrom, COUNT(*) AS c {J} GROUP BY memory.main.a.chrom"  # -> dd rows          naive:error
q4 = f"SELECT a.start {J} ORDER BY memory.main.a.start"           # -> dd rows              naive:error

Expected behavior

dialect="duckdb" should be consistent with the naive plan for a catalog/schema-qualified column: either honor the qualifier, or reject/decline. The cleanest fix mirrors the recently-added projection declines (#204/#205): treat a column bearing a db/catalog qualifier as one the rebuild cannot attribute to a join side, so it either declines the IEJoin to the naive plan (surfacing the same binder error) or raises a clean ValueError like the extras path — rather than silently engaging and dropping the qualifier. The fix must cover all four surfaces: the projection pre-scan, render_aggregate, and the GROUP BY / HAVING / ORDER BY pre-allocation.

Root cause

IntersectsDuckDBIEJoinTransformer._projection_declines_to_naive and the dispatch/allocate path (src/giql/expanders/intersects_duckdb.py) key on _normalize_alias(col.table) alone and never inspect col.args["db"] / col.args["catalog"]. A catalog/schema-qualified column is therefore treated as a plain qualified column (table == "a"), engages the rebuild, and the catalog/schema is dropped. The GROUP BY / HAVING / ORDER BY pre-allocation loops and render_aggregate have the same blind spot.

Pre-existing (predates the #204/#205 fold-in; not a regression — identical before). Surfaced by the PR #203 adversarial review.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions