[Spark 4.1.0] SpecializedGetters trait added getGeography and getGeometry methods

[res-life]

Summary

In Spark 4.1.0, the SpecializedGetters trait added new abstract methods getGeography and getGeometry for spatial data types.

Details

• Spark Version: 4.1.0
• Change Type: Trait API addition

Spark ≤4.0.x

trait SpecializedGetters {
  def getBoolean(ordinal: Int): Boolean
  def getByte(ordinal: Int): Byte
  // ... other getters
  def getVariant(ordinal: Int): VariantVal  // Added in 4.0
}

Spark 4.1.0+

trait SpecializedGetters {
  def getBoolean(ordinal: Int): Boolean
  def getByte(ordinal: Int): Byte
  // ... other getters
  def getVariant(ordinal: Int): VariantVal
  def getGeography(ordinal: Int): GeographyVal  // NEW
  def getGeometry(ordinal: Int): GeometryVal    // NEW
}

Impact

Classes implementing SpecializedGetters that don't implement the new methods will fail to compile:

class CudfUnsafeRow needs to be abstract. Missing implementations for 2 members of trait SpecializedGetters.
  def getGeography(x$1: Int): org.apache.spark.unsafe.types.GeographyVal = ???
  def getGeometry(x$1: Int): org.apache.spark.unsafe.types.GeometryVal = ???

Affected Files

• sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/CudfUnsafeRow.scala

Solution

Add the missing method implementations in CudfUnsafeRow:

final class CudfUnsafeRow(...) extends CudfUnsafeRowBase(...) {
  def getVariant(ordinal: Int) = 
    throw new UnsupportedOperationException("VariantVal is not supported")
  
  // Add these for Spark 4.1.0 compatibility
  override def getGeography(ordinal: Int): GeographyVal =
    throw new UnsupportedOperationException("GeographyVal is not supported")
  
  override def getGeometry(ordinal: Int): GeometryVal =
    throw new UnsupportedOperationException("GeometryVal is not supported")
}

Note: RAPIDS Accelerator doesn't support spatial types, so these throw UnsupportedOperationException.

References

• Spark 4.1.0 added spatial data types (Geography, Geometry) support
• Related types: GeographyVal, GeometryVal in org.apache.spark.unsafe.types

[res-life]

I'll finish the shim and make Spark 410 shim building pass, but need more investigations about potential impacts, tests.

[res-life]

Will be closed by #14120