fix(substrait): roundtrip conditionless joins

[bvolpato]

Which issue does this PR close?

• Closes [substrait] Scalar subquery in select not supported #18066.

Rationale for this change

Queries with conditionless joins can fail when their optimized logical plans roundtrip through Substrait, with Plan("join condition should not be empty"). This affects LEFT JOIN ... ON true and uncorrelated WHERE EXISTS, as well as scalar-subquery projections when the legacy scalar-subquery rewrite is enabled.

The optimizer can remove a constant true join filter. The producer then omitted JoinRel.expression, which Substrait requires.

What changes are included in this PR?

• Serialize conditionless inner joins as CrossRel and other conditionless joins with a literal true condition.
• Handle overlapping input names when producing the synthetic condition for semi, anti, and mark joins.
• Add SQL result coverage for outer joins and uncorrelated EXISTS / NOT EXISTS, including empty inputs.
• Add an opt-in --substrait-optimize test mode and run the new fixture through it in CI. This exercises the optimized plans that trigger the bug.

What is the testing strategy for this PR?

joins_conditionless.slt checks query results through ordinary SQL execution and optimized Substrait roundtripping. Existing producer and DataFrame tests cover conditionless inner joins, scalar-subquery projections, and overlapping input names.

Local validation passed: formatting, Clippy with all targets and features, the extended workspace suite, and ./dev/rust_lint.sh. The extended suite ran in an isolated PID namespace so the RSS tests could sample a valid baseline on this host. Removing the producer fix makes seven of the new optimized roundtrip cases fail with join condition should not be empty; all pass with the fix restored.

Are there any user-facing changes?

Queries containing conditionless joins can roundtrip through Substrait successfully. No breaking production API changes.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.85950% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.72%. Comparing base (7f0ab42) to head (db73510).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...es/datafusion_substrait_roundtrip_engine/runner.rs	0.00%	11 Missing ⚠️
...on/substrait/src/logical_plan/producer/rel/join.rs	89.41%	2 Missing and 7 partials ⚠️
datafusion/sqllogictest/bin/sqllogictests.rs	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23469      +/-   ##
==========================================
- Coverage   82.72%   82.72%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      448992   449193     +201     
  Branches   448992   449193     +201     
==========================================
+ Hits       371439   371598     +159     
- Misses      54918    54941      +23     
- Partials    22635    22654      +19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bvolpato]

@neilconway, would you have some time to review this one?

it fixes scalar-subquery projection round-trips when the rewritten join has no condition. the producer now emits a valid CrossRel or literal-true condition without loosening consumer validation.

[neilconway]

@bvolpato Sure! I'm traveling for a few days but I'm happy to take a look next week.

[bvolpato]

@neilconway, thanks again for offering to take a look. I pushed a follow-up fix for conditionless joins with overlapping input names, and CI is green on the latest head. Could you review when you have a chance?

[neilconway]

Thanks @bvolpato ! Sorry for the delay on this.

Overall this looks good, but as far as I can tell, it isn't specific to scalar subqueries? e.g., the same issue arises for queries containing x LEFT JOIN y ON true or WHERE EXISTS (select 1 from x). The problem is really that condition-less joins didn't roundtrip correctly. If you agree, can you update the PR description accordingly?

I think it would also make sense to test some of these cases via SLT -- since if/when we remove the old-style approach to evaluating scalar subqueries, the SLT we're adding for that case will be removed then.

[bvolpato]

thanks for the review, Neil. agreed, the fix is about conditionless joins more broadly. I updated the title and description accordingly.

I added SQL tests for outer joins and uncorrelated EXISTS / NOT EXISTS, including empty inputs. They're also run in optimized Substrait round-trip mode in CI, so they exercise the join shapes created by the optimizer. Removing the producer fix makes seven of the new cases fail with join condition should not be empty; they pass with the fix restored.

[neilconway]

Thanks @bvolpato !