Skip to content

fix(spark): decode unbase64 input like Spark's MIME Base64 decoder - #26345

Open
ozenyunus35 wants to merge 1 commit into
apache:mainfrom
ozenyunus35:spark-unbase64-mime
Open

ozenyunus35 wants to merge 1 commit into
apache:mainfrom
ozenyunus35:spark-unbase64-mime

Conversation

@ozenyunus35

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Spark's unbase64 decodes with Java's MIME Base64 decoder, which accepts missing padding and skips characters outside the Base64 alphabet. datafusion-spark rewrote unbase64 to the core decode(..., 'base64pad'), so inputs like '123', 'YQ' or 'YW Jj' failed here while Spark decodes them.

What changes are included in this PR?

unbase64 no longer simplifies to decode; it decodes each value with a small mime_decode function:

  • characters outside the standard alphabet (A-Z a-z 0-9 + /) are skipped, and padding is optional;
  • a final unit of one character is an error ("last unit does not have enough valid bits");
  • = must complete the current unit (xx== or xxx=), and only non-alphabet characters may follow it; anything else is an "invalid padding" error.

Binary, LargeBinary and BinaryView inputs are handled; the return type is unchanged (LargeBinary for LargeBinary, Binary otherwise). base64 is unchanged.

I took the expected results from running java.util.Base64.getMimeDecoder().decode(...) on JDK 21 for each input in the tests, including the edge cases around padding (YQ=, YQ= =, YQ===, YQ==x, YWJj=, YQ==YQ==).

What is the testing strategy for this PR?

  • base64.slt: unbase64('123') (string and binary) now returns d76d instead of an error, as the issue asks, plus cases for missing padding, skipped characters, an input with only skipped characters, xxx=, a LargeBinary input, and the three error cases. Six of these fail on main.
  • Unit tests for mime_decode cover the accepted inputs and the error cases above.
  • cargo test -p datafusion-spark --lib string::base64, the spark/string/base64 sqllogictests and cargo clippy -p datafusion-spark --all-targets -- -D warnings pass.

Are there any user-facing changes?

Yes: unbase64 now accepts the inputs Spark accepts (missing padding, characters outside the alphabet), instead of failing.

I used an AI assistant to help draft parts of the code and tests; I reviewed the changes and ran the tests myself.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] unbase64 rejects input that Spark's MIME decoder accepts

1 participant