Repository navigation
fix(spark): decode unbase64 input like Spark's MIME Base64 decoder - #26345
Open
ozenyunus35 wants to merge 1 commit into
Open
ozenyunus35 wants to merge 1 commit into
ozenyunus35 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Spark's
unbase64decodes with Java's MIME Base64 decoder, which accepts missing padding and skips characters outside the Base64 alphabet.datafusion-sparkrewroteunbase64to the coredecode(..., 'base64pad'), so inputs like'123','YQ'or'YW Jj'failed here while Spark decodes them.What changes are included in this PR?
unbase64no longer simplifies todecode; it decodes each value with a smallmime_decodefunction:A-Z a-z 0-9 + /) are skipped, and padding is optional;=must complete the current unit (xx==orxxx=), and only non-alphabet characters may follow it; anything else is an "invalid padding" error.Binary, LargeBinary and BinaryView inputs are handled; the return type is unchanged (LargeBinary for LargeBinary, Binary otherwise).
base64is unchanged.I took the expected results from running
java.util.Base64.getMimeDecoder().decode(...)on JDK 21 for each input in the tests, including the edge cases around padding (YQ=,YQ= =,YQ===,YQ==x,YWJj=,YQ==YQ==).What is the testing strategy for this PR?
base64.slt:unbase64('123')(string and binary) now returnsd76dinstead of an error, as the issue asks, plus cases for missing padding, skipped characters, an input with only skipped characters,xxx=, a LargeBinary input, and the three error cases. Six of these fail onmain.mime_decodecover the accepted inputs and the error cases above.cargo test -p datafusion-spark --lib string::base64, thespark/string/base64sqllogictests andcargo clippy -p datafusion-spark --all-targets -- -D warningspass.Are there any user-facing changes?
Yes:
unbase64now accepts the inputs Spark accepts (missing padding, characters outside the alphabet), instead of failing.I used an AI assistant to help draft parts of the code and tests; I reviewed the changes and ran the tests myself.