You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
date_trunc SECOND/MILLISECOND wraps on Spark 4.2, where Spark raises long overflow #6739
expressions/datetime/trunc_timestamp.sql fails on the Spark 4.2 profile in both dictionary variants:
Error executing SQL 'SELECT date_trunc('SECOND', ts), date_trunc('MILLISECOND', ts)
FROM test_trunc_ts_overflow' Job aborted due to stage failure: ...
java.lang.ArithmeticException: long overflow
at java.base/java.lang.Math.subtractExact(Math.java:945)
at org.apache.spark.sql.catalyst.util.DateTimeUtils$.truncTimestamp(DateTimeUtils.scala:542)
The query came in with #5956. Its fixture expects Spark to wrap the Long subtraction when SECOND or MILLISECOND truncation of timestamp_micros(-9223372036854775808) falls below Long.MinValue. Spark 4.1.3 and earlier do wrap. Spark 4.2.0 uses Math.subtractExact there and raises long overflow, like every other unit (SPARK-56663, apache/spark@da095f9e63). The pull request tier only runs Spark 4.1, so this first showed up on #6664, which has run-all-spark-profiles. The nightly will fail the same way.
The fixture is the visible half. Comet's native kernel wraps on every Spark version (timestamp_trunc_fine_boundary in native/spark-expr/src/kernels/temporal.rs), so on Spark 4.2 Comet returns a timestamp in the year 294247 where Spark raises.
Steps to reproduce
./mvnw test -Pspark-4.2 -Dtest=none -DwildcardSuites=none \
-Dsuites="org.apache.comet.CometSqlFileTestSuite trunc_timestamp.sql"
Expected behavior
On Spark 4.2, Comet raises long overflow when SECOND or MILLISECOND truncation falls below the smallest timestamp. On earlier versions it keeps wrapping, as Spark does.
Activity