Fix PCRE2 UTF-8 validation errors for Qwen tokenizers

[Copilot]

Qwen tokenizer patterns contain byte sequences that aren't valid UTF-8, causing PCRE2 compilation to fail with "UTF-8 error: isolated byte with 0x80 bit set". This prevents loading tokenizers for Qwen2.5 and OuteTTS models.

Changes

• pcre2_regex.cpp: Detect UTF-8 validation errors (codes -3 to -23) and retry compilation without PCRE2_UTF flag
• std_regex.cpp: Catch std::regex_error in find_all() to prevent crashes from pattern complexity errors
• test_regex.cpp: Add test for invalid UTF-8 pattern handling

Fallback chain

Pattern compilation now follows: RE2 → PCRE2 (with UTF) → PCRE2 (without UTF) → std::regex

Example

// Previously failed: "UTF-8 error: overlong 2-byte sequence"
const std::string pattern = "test\xC0\x80pattern";
Pcre2Regex regex;
regex.compile(pattern);  // Now succeeds via non-UTF retry

Original prompt

Look at pytorch/executorch#14432 and pytorch/executorch#16631

Can you fix on this repo?

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[larryliu0820]

Remove?