Performance optimizations for non-ASCII input

[junegunn]

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

---

Acknowledgement

• I confirm that this PR meets the above expectations and reflects my own understanding and real-world context.

[Copilot]

Pull request overview

Optimizes fuzzy matching and ingestion for non-ASCII input using rune-array prefiltering and cached folding metadata.

Changes:

• Adds portable and architecture-optimized rune scanners.
• Tracks foldable Unicode input and prefilters rune-mode matches.
• Adds differential tests, fuzzing, and release notes.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/util/chars.go	Adds rune counting and folding metadata.
src/util/chars_test.go	Tests conversion, flags, and struct size.
src/algo/algo.go	Enables rune-array prefiltering.
src/algo/runeprefilter_test.go	Adds equivalence and fuzz tests.
src/algo/runeindex_x86.go	Adds optimized little-endian rune scanners.
src/algo/runeindex_ref.go	Adds portable reference scanners.
src/algo/runeindex_others.go	Selects reference scanners on other architectures.
Makefile	Adds the new fuzz target.
CHANGELOG.md	Documents performance improvements.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/algo/algo.go:388

• This byte-scanner branch also accepts negative runes because it only checks the upper bound. Once isAscii correctly sends such patterns through the rune prefilter, this condition must select indexRune; otherwise byte(-1) searches for U+00FF and falsely rejects an exact rune(-1) match.

		if last < utf8.RuneSelf {

src/util/chars.go:56

• This merged range also marks U+1F00–U+1FFF (Greek Extended), plus punctuation and symbols up to U+2184, as foldable even though they cannot become ASCII. Any item containing one of those runes therefore sets flagMayFold, and an ASCII query bypasses the new prefilter entirely. Split the Latin blocks and isolated compatibility characters so Greek Extended receives the optimization promised above.

	{0x1D00, 0x2184},

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/util/chars.go:56

• This merged interval also marks Greek Extended (U+1F00–U+1FFF) and unrelated blocks such as General Punctuation as foldable, even though those runes do not lowercase or normalize to ASCII. As a result, polytonic-Greek or punctuation-heavy lines set flagMayFold and bypass the new prefilter, contradicting the stated Greek exclusion; the script test only covers the basic Greek block. Split this interval into the actual Latin/subscript/stray ranges (or generate the exact bitmap) and cover Greek Extended in the test.

	{0x1D00, 0x2184},

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.