Repository navigation
Conversation
Spark gives a join the type Cross when the query asks for a cross join explicitly, with CROSS JOIN in SQL or Dataset.crossJoin, and its join operators run Cross exactly as they run Inner. Comet's sort-merge, hash and nested loop join serdes matched only Inner, so these joins fell back with "Unsupported join type Cross". Map InnerLike to the native Inner join type in all three. Spark's SPARK-26138 test takes LIMIT 3 of a cross join with no ORDER BY. That join now runs as a native nested loop join, which emits rows in a different order than Spark, so ignore the test under Comet in every Spark diff. Closes apache#6903.
Times Spark, Comet with the join left in Spark, and Comet for sort-merge, shuffled hash, broadcast hash and broadcast nested loop joins of type Cross, at 10 and 100 matches per probe row. Before timing, each case checks that every arm planned the join it claims, that the Comet arm is fully native, and that the arms return the same sum.
|
The benchmark shows that when rows match many rows, the native sort-merge join is slower than what Comet does today with the join left in Spark: twice the time at 100 matches per row, and 2.7x at 1000. That isn't specific to Profiling put most of the native time in DataFusion's sort-merge join recording its matches one buffered row at a time, so I opened apache/datafusion#26277 to append them in bulk. With a prototype of that change on DataFusion 55.2.0, measured in the same runs:
That cuts the native join's CPU time by 39% at 100 matches per row and 46% at 1000. What's left at very high fan-out is mostly building the output batches. The join gathers every column of both sides because it has no built-in projection, and its batch coalescer copies them again (apache/datafusion#25515). I'll keep this PR in draft until the fix reaches Comet, then rerun |
1 participant
Which issue does this PR close?
Closes #6903.
Rationale for this change
Spark gives a join the type
Crosswhen the query asks for a cross join explicitly, withCROSS JOINin SQL orDataset.crossJoin, and Spark's join operators runCrossexactly as they runInner. Comet's join serdes matched onlyInner, so a cross join that Spark planned as a sort-merge, shuffled hash, broadcast hash or broadcast nested loop join fell back withUnsupported join type Cross, while the same join written asJOIN ... ONran natively.A production job on Spark 3.5 hit this. A cross join with an equi-key filter became a
SortMergeJoinof typeCrossthat produced 110 billion rows in Spark, and its stage took two thirds of the job's task time.What changes are included in this PR?
InnerLike(InnerandCross) to the nativeInnerjoin type. The keys and condition carry over unchanged, so the native plan is the one the equivalent inner join gets.InnerLikewith either build side, in its support check and its conversion, as it already did forInner.Cross, andLeftSinglefrom Spark 4.0.SPARK-26138 Pushdown limit through InnerLike when condition is emptytestIgnoreCometin all five Spark diffs, regenerated from the Spark tags after checking that each committed diff round-trips unchanged. The test takesLIMIT 3oft1 CROSS JOIN t2with noORDER BYand expects the rows Spark returns. The join now runs as a native nested loop join, which emits rows in a different order, so Comet returns(0,0), (1,0), (2,0)where Spark returns(0,0), (0,1), (0,2). Both are valid answers.mainalready returns Comet's rows forFROM t1, t2 LIMIT 3, anInnerjoin; the test passed only because theCrossjoin fell back.CometCrossJoinBenchmark, which times each of the four join strategies on a join of typeCrossin three arms: Spark, Comet with the join left in Spark (the plan Comet made before this change), and Comet.Native cross nested loop joins now also have the problem described in #6898, which inner ones already have: the join reports the streamed side's order without keeping it.
CartesianProductExec, a cross join with no broadcast side, still falls back (#199).How are these changes tested?
New tests in
CometJoinSuite:CROSS JOIN ... WHERE tbl_a._2 = tbl_b._1, with and without a residual condition, underSHUFFLE_HASH,MERGEandBROADCASThints.CROSS JOINwith a broadcast hint on either side, with no condition and with a non-equi condition, which Spark plans as a broadcast nested loop join.Each test checks the answer against Spark, checks that the plan is fully native, and checks that the native join has the type
Cross, so a plan in which Spark choseInnercan't make it pass. With the serde change reverted, all four fail withUnsupported join type CrossorJoin type Cross is not supported.Locally,
CometJoinSuitepasses on Spark 3.4, 3.5 and 4.1. The other Comet tests that use a cross join pass on Spark 3.4 and 4.1:CometFloatSemanticsSuite, the AQE tests inCometExecSuite, aCometCodegenSuitetest, and two SQL file tests.For Spark's SQL tests, I went through every cross join in the
sql/coreandsql/hivetests of Spark 3.5.9 and 4.1.3 with the diffs applied, and compared the cross joins in 3.4.3, 4.0.4 and 4.2.0 against 4.1.3.SPARK-26138is the only test whose outcome changes. The others already accept Comet's join classes, are in suites ignored under Comet, or check only results. I haven't run Spark's SQL suites locally.Benchmark
CometCrossJoinBenchmarkjoins a 1M-row probe table in which every row meets 10 or 100 rows on the other side, 10.5M or 105M joined rows, and sums a column from each side so that only one row leaves the native plan. Before timing, each case checks that every arm planned the join it claims with the join typeCross, that the Comet arm is fully native, that Spark's join produced the expected number of rows, and that the three arms return the same sum.Best time in ms, the median of three runs with the range in brackets. Apple M3 Max with a load average of 5 to 10 from other work,
local[5], Spark 4.1, release build:Comet is 1.1x to 1.8x faster than Spark in every case except the sort-merge join at fan-out 100. Most of that comes from the native scans and shuffles: at fan-out 10, the two Comet arms are within 13% of each other.
At fan-out 100, the native shuffled and broadcast hash joins take 6% and 19% less time than the joins left in Spark. The native nested loop join takes 14% longer, and the native sort-merge join twice as long. Spark fuses the merge and the aggregate into one generated loop and never materializes the 105M joined rows, while the native join builds Arrow batches for all of them. This cost isn't specific to
Cross: anInnersort-merge join of the same shape took 410 ms natively, against 427 ms forCrossin the same run. It is the pattern described in #4614. Cross joins with an equi-key filter are often many-to-many, so they are likely to hit it, and the job that prompted #6903 had a join of this kind.