You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException #6966
Spark parses a string to decimal with new java.math.BigDecimal(str) and then rescales it to the target type with BigDecimal.setScale. For extreme exponents that rescale fails with a raw java.lang.ArithmeticException in every eval mode (ANSI, legacy and try_cast), while the native cast returns a value or NULL. Found while reviewing #6826.
Two shapes:
Positive target scale.'1e-2147483647' cast to DECIMAL(10,2): the parsed scale (2147483647) fits in an int, so the string is valid, but the scale difference to the target is about 2.1e9. Spark fails with BigInteger would overflow supported range. The native cast returns 0.00, because parse_string_to_decimal returns zero whenever the value has to be divided by more than 10^38.
Negative target scale (spark.sql.legacy.allowNegativeScaleOfDecimal=true). '1e-2147483647' cast to DECIMAL(10,-2): the scale difference does not fit in an int, and Spark fails with Underflow. Since fix: do not fail planning on a cast of a literal that fails to evaluate #6826 the native cast reports CAST_INVALID_INPUT under ANSI and NULL otherwise, so both engines fail under ANSI but with different exceptions, and they differ outside ANSI.
On JDK 17 and 21, setScale returns zero for scale differences up to about 1e8 (after computing a very large power of ten) and throws from about 6e8. The exact threshold depends on the JDK's BigInteger limits, so the native cast cannot reproduce it exactly.
Steps to reproduce
CREATETABLEt(s string) USING parquet;
INSERT INTO t VALUES ('1e-2147483647');
SELECT try_cast(s ASDECIMAL(10,2)) FROM t;
Spark: java.lang.ArithmeticException: BigInteger would overflow supported range. Comet: 0.00.
Expected behavior
To be decided. Spark raising a raw ArithmeticException even from try_cast looks like a Spark-side issue. Comet should at least not silently return a value where Spark fails.
Activity
Link the remaining string to decimal gap to apache#6966
fix: do not fail planning on a cast of a literal that fails to evalua…