Repository navigation
Substrait consumer rejects multi-needle InPredicate for row-valued IN subqueries #25720
Description
Describe the bug
DataFusion rejects a spec-valid Substrait InPredicate when it contains more than one needle expression. This represents row-valued SQL IN predicates like:
SELECT 1 AS result
WHERE (1, 2) IN (SELECT 1, 2)The Substrait spec allows this shape. InPredicate.needles is repeated, and the proto comment explicitly includes (x, y) IN (SELECT a, b FROM t): https://github.com/substrait-io/substrait/blob/b0341cea26fd0a83eb2dfd8e9d4f64df1b8e1759/proto/substrait/algebra.proto#L1818-L1820
To Reproduce
Run DataFusion's Substrait consumer on this minimal plan, generated with substrait-go:
{
"version": {
"minorNumber": 29,
"producer": "substrait-go v9.0.0-alpha.0 linux/arm64"
},
"relations": [
{
"root": {
"input": {
"project": {
"common": {
"direct": {}
},
"input": {
"filter": {
"common": {
"direct": {}
},
"input": {
"read": {
"common": {
"direct": {}
},
"baseSchema": {
"struct": {
"nullability": "NULLABILITY_REQUIRED"
}
},
"virtualTable": {
"expressions": [
{}
]
}
}
},
"condition": {
"subquery": {
"inPredicate": {
"needles": [
{
"literal": {
"i64": "1",
"nullable": true
}
},
{
"literal": {
"i64": "2",
"nullable": true
}
}
],
"haystack": {
"project": {
"common": {
"direct": {}
},
"input": {
"read": {
"common": {
"direct": {}
},
"baseSchema": {
"struct": {
"nullability": "NULLABILITY_REQUIRED"
}
},
"virtualTable": {
"expressions": [
{}
]
}
}
},
"expressions": [
{
"literal": {
"i64": "1",
"nullable": true
}
},
{
"literal": {
"i64": "2",
"nullable": true
}
}
]
}
}
}
}
}
}
},
"expressions": [
{
"literal": {
"i64": "1",
"nullable": true
}
}
]
}
},
"names": [
"result"
]
}
}
]
}DataFusion returns:
Substrait error: InPredicate Subquery type must have exactly one Needle expression
The immediate rejection is in the Substrait consumer:
Expected behavior
DataFusion should accept this Substrait plan and return one row:
result
------
1
Additional context
This likely needs broader support beyond removing the Substrait guard. The logical expression model currently describes InSubquery as one expression compared against a single-column subquery:
datafusion/datafusion/expr/src/expr.rs
Lines 1384 to 1388 in a98f304
The SQL planner also validates IN subqueries as single-column:
datafusion/datafusion/sql/src/expr/subquery.rs
Lines 73 to 78 in a98f304
There is related SQL-side work in #19857, but I did not find an issue tracking Substrait multi-needle InPredicate support specifically.
Activity
mohitgurav20 commentedon Sep 25, 2026
Hi @benbellick! This is a great find. Row-valued IN subqueries are a very useful feature.
As you correctly pointed out, simply removing the guard in the Substrait consumer isn't enough. To do this properly, we'll need to thread this support through the entire stack:
Upgrade the underlying logical expression model (InSubquery in datafusion/expr/src/expr.rs) to accept a list of expressions (Vec) rather than just a single Box.
Update the SQL planner (datafusion/sql/src/expr/subquery.rs) to remove the single-column validation and map the row-valued syntax to the updated logical expression.
Finally, adjust the Substrait consumer (datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs) to remove the length guard and map multiple needles correctly.
I'd love to take this on! I have been working heavily in DataFusion's expression and planning layers recently (including logical/physical boundaries) and have the context needed to implement this full-stack change cleanly.
Could you assign this to me?
mohitgurav20 commentedon Sep 25, 2026
take.