Repository navigation
fix(search): tokenize diacritical text via NFD normalisation - #31
Open
ashishbhateja wants to merge 1 commit into
Open
ashishbhateja wants to merge 1 commit into
ashishbhateja wants to merge 1 commit into
Conversation
Words like "Śrī" were silently reduced to just "aurobindo" (losing "sri" entirely) because the regex split on the non-ASCII code-points. Applying NFD normalisation followed by stripping the U+0300–U+036F combining-mark block lets diacritical variants like Śrī/ā/é match their ASCII equivalents, which is how a reader would naturally type a search query. Adds three new checks in smoke.mjs: per-token diacritics stripping for Śrī → sri and Pondichéry → pondichery, and an end-to-end search test that finds a diacritical title via an ASCII query. Advances #3.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
tokenizeinsrc/search.jsnow applies NFD normalisation and strips combining diacritical marks (U+0300–U+036F) before splitting on non-word characters. Two lines added, no logic changed for ASCII input.Why
Words containing diacritical characters were silently mangled by the search index. For example:
"Śrī Aurobindo"→ only["aurobindo"]was indexed;"sri"was lost entirely becauseŚandīare non-ASCII and caused the regex to split around single-character fragments that were then filtered out."Pondichéry"→ indexed as["pondich", "ry"](two useless fragments) instead of["pondichery"].This matters for a magazine whose content routinely uses Sanskrit transliterations and proper nouns with French/Indian diacriticals. A reader typing "Sri" to find articles about Śrī Aurobindo got no results from any title or field that used the Unicode form.
The fix uses
String.prototype.normalize('NFD')(standard in all engines since 2014) to decompose accented characters into a base letter plus a combining mark, then strips the combining marks. ASCII text is completely unaffected.Closes #3 (partial — improves index fidelity for multilingual content).
Checklist
node scripts/smoke.mjspasses (22 checks, up from 19; three new diacritics tests added)Generated by Claude Code