Repository navigation
bug: datafusion-spark string literals don't interpret escape sequences like Spark #21516
Description
Describe the bug
DataFusion-spark treats \t and \n in SQL string literals as literal backslash characters, while Apache Spark interprets them as escape sequences (tab and newline). This affects any function that receives string arguments containing these sequences.
To Reproduce
PySpark (Spark behavior):
SELECT soundex('\thello'); -- returns tab + "hello" (soundex passes through non-alpha input)
SELECT soundex('\nhello'); -- returns newline + "hello"
SELECT length('\thello'); -- 6 (tab is one character)
SELECT length('\nhello'); -- 6 (newline is one character)DataFusion-spark (current behavior):
SELECT soundex('\thello'); -- returns literal "\thello" (backslash-t-hello)
SELECT soundex('\nhello'); -- returns literal "\nhello" (backslash-n-hello)
SELECT length('\thello'); -- 7 (\t is two characters: backslash and t)
SELECT length('\nhello'); -- 7 (\n is two characters: backslash and n)Expected behavior
DataFusion-spark should interpret \t, \n, and other escape sequences in string literals the same way Spark does, for Spark compatibility.
Additional context
This is a string literal parsing issue, not specific to soundex. It affects all string functions. The .slt tests at string/soundex.slt lines 83 and 193 have expected values that match DataFusion's literal interpretation rather than Spark's escape interpretation.
This was discovered by running a PySpark validation script against the .slt test files (see #17045, #21508).
Activity
SamAya21 commentedon Apr 9, 2026
Hello, I would like to try working on this bug. I haven't worked on an issue for this project before do you think this bug is fine as a first issue or would you recommend any of the other 4 bug reports you've posted recently.
Abaydoun54 commentedon Apr 12, 2026
take (working with SamAya21)
SamAya21 commentedon Apr 13, 2026
take
fix: address escape literal issue apache#21516
SamAya21 commentedon Apr 15, 2026
Is there a suggested way to rewrite test cases related to this change for example there are some test cases in sqlllogictest/test_files/spark/string/soundex.stl. There are two test cases with \nhello and \thello, but editing them have a new line or tab never matches the expected or just breaks the test file. There is also a simliar issue in spark/array/array.stl where a new lines gets added in a middle an array.
SamAya21 commentedon Jun 15, 2026
untake