fix: handle surrogate pairs in non-unicode regex patterns

[amrkhaled104]

Overview

This PR fixes test262 failures related to RegExp matching when surrogate pairs (like 𠮷) are used without the u or v flags.

The Problem

I found that in Non-Unicode mode, Boa was compiling the regex pattern using full 32-bit Code Points. However, at runtime, the engine uses find_from_ucs2 which looks at raw 16-bit Code Units.

For example:

• The character 𠮷 was compiled as one atom: 0x20BB7.
• But the input string in memory is two units: [0xD842, 0xDFB7].
  This mismatch caused the regex to fail even if the character was clearly there.

What I Changed

1. In core/engine/src/builtins/regexp/mod.rs:

• I updated compile_native_regexp to check if the Unicode flag is missing.
• If there is no u or v flag, I manually "flatten" the pattern. I iterate through each code point and decompose it into its individual UTF-16 units (surrogates) before passing them to the regress matcher.
• This makes the compiled pattern match the 16-bit structure of the input string.

How I Tested

I ran the boa_tester with the following command:

cargo run --bin boa_tester -- run -s test/built-ins/String/prototype/match/regexp-prototype-match-v-u-flag.js

Result: All tests passed

and exec test

---

[github-actions]

Test262 conformance changes

Test result	main count	PR count	difference
Total	52,862	52,862	0
Passed	49,497	49,504	+7
Ignored	2,261	2,262	+1
Failed	1,104	1,096	-8
Panics	0	0	0
Conformance	93.63%	93.65%	+0.01%

Fixed tests (8):

test/staging/sm/RegExp/unicode-raw.js (previously Failed)
test/staging/sm/RegExp/unicode-class-raw.js (previously Failed)
test/built-ins/String/prototype/replace/regexp-prototype-replace-v-u-flag.js (previously Failed)
test/built-ins/String/prototype/matchAll/regexp-prototype-matchAll-v-u-flag.js (previously Failed)
test/built-ins/String/prototype/search/regexp-prototype-search-v-flag.js (previously Failed)
test/built-ins/String/prototype/search/regexp-prototype-search-v-u-flag.js (previously Failed)
test/built-ins/String/prototype/match/regexp-prototype-match-v-u-flag.js (previously Failed)
test/built-ins/RegExp/prototype/exec/regexp-builtin-exec-v-u-flag.js (previously Failed)

[amrkhaled104]

@jedel1043 why cI not work ?

[amrkhaled104]

@jedel1043 All 8 tests are now passing 💪

[codecov]

Codecov Report

❌ Patch coverage is 81.25000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.08%. Comparing base (6ddc2b4) to head (8f51053).
⚠️ Report is 674 commits behind head on main.

Files with missing lines	Patch %	Lines
core/engine/src/builtins/regexp/mod.rs	81.25%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4700      +/-   ##
==========================================
+ Coverage   47.24%   57.08%   +9.84%     
==========================================
  Files         476      549      +73     
  Lines       46892    60152   +13260     
==========================================
+ Hits        22154    34338   +12184     
- Misses      24738    25814    +1076     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[amrkhaled104]

Are there any issues you'd suggest I pick up ? @jedel1043

[jedel1043]

I kind of don't understand why we need this. IIRC the only things that influence if a regex is parsed as UTF16 or UCS2 are the u and v flags, right? And that's being taken care of by the full_unicode check above.

[amrkhaled104]

@jedel1043 I want to explain the full picture. Sorry, I forgot to update the PR description after my last changes
In the first version of the PR, I found that many Prototype tests were failing. The reason was how we handle Unicode and Non-Unicode modes
In Non-Unicode mode, JavaScript treats 'heavy' characters (like emojis or special math symbols) as two 16-bit units, not one character. But our engine was treating them as one unit. This made the search index wrong
I used flat_map to split these characters into two units. This fixed the indexing and made the Prototype tests pass.

After that, the test test\built-ins\RegExp\named-groups\non-unicode-property-names-valid.js failed. I realized this is a special case Even in Non-Unicode mode, Named Groups (?) need the character to stay as one 'Identity' (Code Point). If we split the name into two units, the Regex engine cannot find the group name because the name becomes 'broken' So, I added a check:
let has_named_groups = !full_unicode && p.to_std_string_escaped().contains("(?<");
or

            matches!(
                (w[0], w[1], w[2]),
                (
                    CodePoint::Unicode('('),
                    CodePoint::Unicode('?'),
                    CodePoint::Unicode('<')
                )
            )
        });

This code checks for the (?< sequence If the pattern has named groups, we keep the characters as Code Points (like Unicode mode). .

[jedel1043]

Then this edge case is not our responsibility. It should be the responsibility of regress to parse group names as unicode even if parsing without unicode support.

I would suggest opening a bug on their side reporting this, and removing the hack. The test should pass afterwards without having to hack around this.

[amrkhaled104]

You are right,I will update the PR

[amrkhaled104]

@jedel1043 I have updated the code and removed the hack. To keep the CI green while we wait for a fix in regress, should I add this specific test to the test262_config.toml ignore list?

[jedel1043]

Yep, please do! And if you can, add a TODO on the config file pointing to regress' issue.

[amrkhaled104]

Got it! I will add the test to test262_config.toml with a TODO note in this PR.

[amrkhaled104]

@jedel1043 Anything else?

[jedel1043]

Nope, everything looks good!