[Spark 4.1.0] MAX_BROADCAST_TABLE_BYTES constant removed from BroadcastExchangeExec

[res-life]

Description

In Spark 4.1.0, the MAX_BROADCAST_TABLE_BYTES constant was removed from BroadcastExchangeExec object.

Spark 4.0.x

// org.apache.spark.sql.execution.exchange.BroadcastExchangeExec
val MAX_BROADCAST_TABLE_BYTES = 8L << 30  // 8GB

Spark 4.1.0

The constant was removed and replaced with a configurable value via conf.maxBroadcastTableSizeInBytes.

Impact

Code that imports BroadcastExchangeExec.MAX_BROADCAST_TABLE_BYTES will fail to compile against Spark 4.1.0:

object MAX_BROADCAST_TABLE_BYTES is not a member of object org.apache.spark.sql.execution.exchange.BroadcastExchangeExec

Affected Files

• sql-plugin/src/main/scala/org/apache/spark/sql/rapids/execution/GpuBroadcastExchangeExec.scala

Solution

Create a shim BroadcastExchangeShims that:

• For Spark <= 4.0.x: references BroadcastExchangeExec.MAX_BROADCAST_TABLE_BYTES
• For Spark 4.1.0+: defines the constant directly as 8L << 30 (the original hardcoded value)

Update GpuBroadcastExchangeExec.scala to import from the shim:

// Before
import org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.MAX_BROADCAST_TABLE_BYTES

// After  
import com.nvidia.spark.rapids.shims.BroadcastExchangeShims.MAX_BROADCAST_TABLE_BYTES

References

• Spark 4.0.1: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala line 271
• Spark 4.1.0: Constant removed, now uses conf.maxBroadcastTableSizeInBytes

[res-life]

I'll finish the shim(the original hardcoded value ), but need this issue to track the impact:
Do we need to get it from conf.maxBroadcastTableSizeInBytes

[res-life]

I'll finish the shim and make Spark 410 shim building pass, but need more investigations about potential impacts, tests.

[res-life]

Will be closed by #14120