Conversation
Implements a length-preserving, settings-driven text cleanup pipeline so users can strip anti-scraper watermarks and fix LN name pronunciations before text reaches the native Android TTS engine, across all playback modes and paths (initial queue, WebView DOM refills, fallback speak). Addresses the ISSUE #17 root cause verified in PR.md: the proposed window.tts.getTextNodes() synchronous hook was rejected because that mechanism does not exist and cannot affect the RN-parsed initial audio queue, which is the only source in the default background mode. The fix instead applies a declarative pipeline in the React Native layer. Pipeline (per paragraph): 1. Optional NFD unicode normalization + combining-mark strip 2. Ordered find/replace rules (literal or regex + flags, strip via empty replacement) 3. Phonetic dictionary (whole-word, unicode-aware boundaries) Key properties: - Length-preserving: never drops/merges paragraphs, keeping the RN <-> WebView paragraph index and utterance-ID contract intact - No eval/Function of user code in RN; declarative rules only - No hardcoded site regexes; everything is user-configurable - Invalid regexes are skipped without crashing TTS - Missing settings on legacy installs degrade to a no-op Wiring covers every audio-feeding point (verified by audit): - useTTSController.ts: 7 RN-parse sites + 'tts-queue' refill + 'speak' fallback (both speak paths) - useTTSUtilities.ts: restart/seek/drift path - WebViewReader.tsx: live-settings-change restart path Settings UI: AccessibilityTab (global) and ReaderTTSTab quick access, with a new TtsTextCleanupModal editor (rules + phonetic dictionary). Tests: 20 new unit tests for the cleanup engine; all affected TTS suites updated; full suite 1216 passing (baseline 1196).
Closes the ISSUE #17 audit findings on the cleanup engine: - ReDoS guard: reject regex patterns longer than TTS_CLEANUP_MAX_REGEX_LENGTH (200) or matching narrow catastrophic-backtracking shapes ((a+)+, (?:a*)*, (?:a+){2,}, (a|a)+). Heuristics were validated empirically against the issue thread's own patterns and 20 realistic safe patterns (zero false positives). Save-time validation in the cleanup modal surfaces the rejection; runtime skip keeps TTS from ever freezing. - Literal replacement semantics: regex rules now use a callback so $&, $', \$` , $$ and $n in the replacement stay literal. - Flag normalization: dedupe + strip invalid flags + always force g; drop the sticky y flag which silently no-ops on mid-string matches (verified /y and /gy do not scan past index 0). - NFD pattern matching: literal rule patterns are NFD-normalized when unicode normalization is on, so precomposed chars like é still match normalized text (regex patterns documented as matched verbatim). - Defensive replaceWholeWord: try/catch around RegExp construction with a bounded word-keyed cache (avoids recompiling per paragraph across 2000-paragraph queue builds). - String type-guard in cleanTtsText for non-string runtime inputs. Tests: +10 (heuristic detection/safe-pattern regression, length cap, catastrophic skip ordering, literal $ semantics, sticky-as-global, precomposed matching, flag normalization). 30/30 cleanup tests pass.
Closes the ISSUE #17 audit MAJOR finding: ttsTextCleanup was global-only while the PR design promised per-novel support 'like other TTS settings' (voice/rate/pitch via NovelTtsSettings). - novelTtsSettings.ts: optional ttsTextCleanup field on NovelTtsSettings (backward compatible — absent for legacy objects) + exported resolveEffectiveTtsCleanup(global, novelId) returning the per-novel override when per-novel mode is enabled AND a cleanup override was saved, else the global settings. Any MMKV read failure degrades to global. - WebViewReader.tsx: chapterGeneralSettingsRef now carries the EFFECTIVE cleanup. A syncEffectiveTtsCleanup callback re-resolves after every wholesale ref assignment (ref-sync effect, MMKV CHAPTER_GENERAL_SETTINGS listener) plus a new effect reacting to novel/novelTtsSettings/global changes. novelIdRef mirrors novel.id for the mount-once listener. All 10 existing cleanup call sites (useTTSController/useTTSUtilities/WebViewReader) automatically read the effective value with zero controller changes. - ReaderTTSTab.tsx: per-novel cleanup state + effectiveCleanup memo; Text Cleanup entry shows per-novel badge; modal save routes to setNovelTtsSettings when per-novel mode is on (writing a frozen per-novel copy) else to global. All four setNovelTtsSettings writers now preserve an existing ttsTextCleanup so it is never dropped. - Tests: +6 resolver unit tests (undefined novelId, absent settings, disabled mode, saved override, enabled-without-override, MMKV throw); WebViewReader test mocks upgraded with a resolver mirroring the real one against the mocked getNovelTtsSettings. Per-novel semantics: enabling per-novel TTS (voice/rate/pitch) also activates per-novel cleanup when one has been saved; otherwise global cleanup applies. AccessibilityTab (global editor) is unaffected.
…itor Closes two ISSUE #17 audit MINOR findings: - CJK phonetic matching: whole-word boundaries never fire between adjacent CJK characters (both sides are \p{L}), making the pronunciation dictionary a silent no-op for unspaced CJK prose. TtsPhoneticPair gains an optional matchMode ('whole-word' default | 'substring'); substring mode replaces every occurrence via split/join. The pair editor exposes a toggle. Legacy persisted pairs without the field default to whole-word (backward compatible). - Rule reorder UI: rules are applied in array order, so the editor now supports move up/down (arrow icons, disabled at boundaries) for both find/replace rules and phonetic pairs, per the PR design's ordered rule list. Tests: +2 (substring CJK replacement; absent-matchMode defaults to whole-word preserving legacy behavior). 32/32 cleanup tests pass.
Closes the ISSUE #17 audit MINOR finding: applyTtsTextCleanup and cleanTtsText were identity-mocked in the controller tests but never asserted, so removing the wiring from the 'speak' or 'tts-queue' handlers would not have failed any test. - useTTSController.integration.test.ts: the tts-queue handler test now asserts applyTtsTextCleanup is called with the queue texts + effective settings before addToBatch; a new test asserts cleanTtsText is called with the DOM text + settings on the 'speak' fallback path. - mediaNav/progressSync suites do not dispatch speak/tts-queue messages (mediaNav mocks useTTSUtilities; progressSync is a self-contained simulation), and the WebViewReader restart branch is guarded by a live TTS reading state the harness cannot reach — both documented as covered elsewhere (useTTSUtilities.test.ts already asserts the restart/seek path).
Suppresses a react-hooks/exhaustive-deps warning introduced by the per-novel cleanup wiring: syncEffectiveTtsCleanup is a stable useCallback ([] deps), so referencing it in the mount-once MMKV listener is safe and the dependency array can list it explicitly.
- fix(tts): include chapterGeneralSettingsRef in controller effect deps (resolves 3 exhaustive-deps warnings introduced by cleanup wiring) - fix(tts): guard cleanup modal against partial stored settings objects (normalize rules/phoneticPairs at source + defensive ?? [] in entry points) - docs(tts): add unicode normalization hint to regex rule editor - docs(tts): add specs/tts-text-cleanup/PRD.md and update AGENTS.md (Current Task, Recent Fixes, TTS File Map)
- useTTSController.ts: include chapterName in 4 effect/callback dep arrays and restoreSavedEngine (stable useCallback, [] deps) in the dialog handlers callback — behavior-neutral, no new effect re-runs - useTTSUtilities.ts: include chapter.name in restart-from-index callback - WebViewReader.tsx warnings intentionally left untouched (documented design: adding offset/haptic deps caused WebView reload regressions)
- Preset Library: 7 curated one-tap templates (Novelight spaced watermark,
u2014 corruption, '(Official version)' tags, 'Do not rehost' spam,
math-bold lookalikes, LN name pronunciations, CJK substring pairs)
shipped as DATA only - applied by copying into the user's editable
rule list (MMKV); the core pipeline stays site-agnostic (PR_17 §10)
- Import/Export: versioned JSON envelope ('lnreader-tts-cleanup' v1) via
native share sheet; import sanitizes + regenerates ids, validates
regexes (length cap, compile check), coerces matchMode/flags, skips
invalid entries with a summary; replace-in-draft (Cancel discards)
- Pure logic in ttsCleanupPresets.ts (applyPresetToSettings,
serializeCleanupSettings, parseCleanupSettingsImport) + 21 unit tests
- docs(README): add TTS Text Cleanup feature table row, showcase
subsection with pipeline diagram, What's New entries, Getting Started
guide, and TOC links
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1. Summary
Implements a declarative, length-preserving TTS text-cleanup pipeline so users can strip anti-scraper watermarks (e.g. Novelight) and fix LN name pronunciations before text reaches the native Android TTS engine — across all playback modes and paths, including background playback. Includes per-novel overrides, regex-safety hardening, a full rule/phonetic editor, one-tap curated presets, and JSON import/export for sharing/backing up rule sets.
The issue's proposed mechanism (
window.tts.getTextNodes()+executeCustomUserReaderJSsynchronous hook) was investigated and rejected — see §3. The chosen approach applies a settings-driven rule pipeline in the React Native layer instead.2. Issue context
snthsh2)u2014string corruption, "Do not rehost this novel" / "(Official version)" phrases) that bypass user customJS cleaning and get read aloud by TTS.Xianxia→ "Shee-an-shah",Qing→ "Ching",Ainz→ "Ownz") so system engines stop mispronouncing LN names.2.1 Why the proposed hook was rejected (verified against codebase)
window.tts.getTextNodes()"getTextNodesdoesn't exist incore.js; nativespeakBatchreceivesList<String>onlyextractParagraphs), never reads the DOMgetTextNodeswill fix watermarks"tts-queue) and fallback single-speak do use DOMtextContent2.2 Actual TTS audio data flow (the crux)
extractParagraphs(sanitizedHtml)el.textContentviatts-queuetextToSpeakviaspeakextractParagraphsBackground playback is the default mode (whole chapter queued from RN in one
speakBatch), so a WebView-only mechanism could never clean background audio. Cleanup must happen RN-side — which is what this PR does.3. What this PR does
3.1 New declarative cleanup engine —
src/utils/htmlParagraphExtractor.tsPure functions + settings types (no
eval/Function, no hardcoded site regexes):TtsTextCleanupSettings— masterenabledswitch,normalizeUnicode(NFD + strip combining marks), orderedrules[],phoneticPairs[]TtsCleanupRule— literal find/replace or regex (pattern+flags+replacement; empty replacement strips)TtsPhoneticPair— whole-word, case-sensitive pronunciation swaps (unicode-aware boundaries)cleanTtsText(text, settings)— pipeline: unicode normalization → ordered rules → phonetic dictionaryapplyTtsTextCleanup(paragraphs, settings)— length-preserving map (never drops/merges entries → RN↔WebView paragraph index contract intact)3.2 Wiring — every audio-feeding point (coverage map)
src/screens/reader/hooks/useTTSController.tsapplyTtsTextCleanup(extractParagraphs(...), ref)on all RN-parse pathssrc/screens/reader/hooks/useTTSController.tstts-queue)addToBatchsrc/screens/reader/hooks/useTTSController.tsspeakcase)cleanTtsTextontextToSpeak(covers both fallback speaks) + cleaned re-extractsrc/screens/reader/hooks/useTTSUtilities.tssrc/screens/reader/components/WebViewReader.tsxsrc/hooks/persisted/useSettings.tsChapterGeneralSettingsttsTextCleanupfield +DEFAULT_TTS_CLEANUP_SETTINGSSettings flow via
chapterGeneralSettingsRef(kept fresh by MMKV listener +useEffect). Legacy persisted settings without the new key degrade safely (optional chaining + destructure defaults + no-op whenenabled: false).3.3 Settings UI
TtsTextCleanupModal(new): master toggle, unicode normalization toggle, ordered Find & Replace rule editor (literal or regex + flags, reorderable, save-time regex safety validation with inline errors), phonetic dictionary editor (whole-word or substring match mode for CJK, reorderable); both lists support add/edit/delete and per-item enableu2014corruption, "(Official version)" tags, "Do not rehost" spam, math-bold lookalikes, LN name pronunciations, CJK substring pairs). Presets are UI data only — never executed by the pipeline; applying merges them into the user's editable rule list (deduped), keeping the core site-agnosticlnreader-tts-cleanupv1) shared via the native share sheet; import validates (id regeneration, regex length-cap + compile check,matchMode/flags coercion, invalid-entry skip with summary) and replaces the draft (Cancel discards)4. Design constraints honored
5. Audit results (3 independent reviewers)
The implementation was audited by three independent fresh-context reviewers (core utility / TTS wiring / settings+UI), plus
type-check, ESLint, and the full test suite. No blockers.5.1 Findings addressed in this PR
extractParagraphscall sites +tts-queue+ bothspeakfallbacks) — verified end-to-end by tracing everyTTSHighlight.speak/speakBatch/addToBatchcall\p{L},\p{N}) andString.normalize('NFD')verified supported by the bundled Hermes runtime (RN 0.82)ttsTextCleanupcannot crash any consumer5.2 Audit follow-ups — ALL RESOLVED in follow-up commits (2026-08-02)
NovelTtsSettingsextended with optionalttsTextCleanup;resolveEffectiveTtsCleanup()writes the effective value intochapterGeneralSettingsRef(re-synced after every ref assignment + on novel/settings change); per-novel save routing + field preservation in all 4setNovelTtsSettingswritersaeec7abb6TTS_CLEANUP_MAX_REGEX_LENGTH(200) +isPotentiallyCatastrophic()(3 narrow structural heuristics, empirically validated — zero false positives on issue-thread patterns) runtime skip; save-time validation with inline error in the modal1a1ffb00d$&/$nreplacement interpolation1a1ffb00dmatchMode: 'substring'(split/join) + editor toggle; legacy pairs default to whole-wordb0b56f333tts-queuehandler test assertsapplyTtsTextCleanupargs; newspeakfallback test assertscleanTtsTextargsb47f527c0yflag silent no-opnormalizeRegExpFlags()stripsy(verified/yand/gynever scan past index 0) + dedupes/validates flags, always forcesg1a1ffb00d1a1ffb00db0b56f333replaceWholeWordunguarded / per-paragraph recompile1a1ffb00dcleanTtsTexttypeof text !== 'string'early return1a1ffb00dTests after follow-ups: 1256 passing (baseline 1196 + 60 new: 32 cleanup engine + 6 resolver + 1 controller wiring + 21 presets/import-export).
6. Tests
src/utils/__tests__/ttsTextCleanup.test.ts(32 tests) — literal/regex rules, invalid-regex skip, unicode normalization + lookalike mapping, phonetic whole-word swaps (incl. no-match-inside-words), pipeline ordering, length-preservation, null/undefined/disabled passthrough, regex-safety hardeningsrc/screens/settings/SettingsReaderScreen/Modals/__tests__/ttsCleanupPresets.test.ts(21 tests) — preset shape + regex safety, apply/merge/dedupe/immutability, serialize round-trip, import validation matrix (bad JSON/types, invalid-regex skip counts, id regeneration, flag fallbacks, empty import)applyTtsTextCleanup,cleanTtsText,chapterGeneralSettingsRef)useTTSUtilities,useTTSController.{integration,mediaNav},useTTSProgressSync,WebViewReader.{eventHandlers,integration},ttsCleanupPresets7. Manual test checklist
u2014) → enable Clean TTS text in Reader → TTS Tab → add ruleu2014→ replace withXianxia→Shee-an-shah→ confirm pronunciation change([unclosed) → confirm it is skipped, no crash, playback unaffectedu2014text corruption" → rule appears in Find & Replace (deduped if already present) → Save → confirm appliedlnreader-tts-cleanupenvelope([unclosed) → imported with "N invalid rules skipped" summary; no crash; Cancel discards everything8. Files changed
Modified
src/utils/htmlParagraphExtractor.ts(+295) — cleanup engine, safety hardening, matchModesrc/hooks/persisted/useSettings.ts(+14)src/services/tts/novelTtsSettings.ts(+40) — optionalttsTextCleanup+resolveEffectiveTtsCleanupsrc/screens/reader/hooks/useTTSController.ts(+65/-15)src/screens/reader/hooks/useTTSUtilities.ts(+18)src/screens/reader/components/WebViewReader.tsx(+35) — effective-cleanup ref resolutionsrc/screens/settings/SettingsReaderScreen/tabs/AccessibilityTab.tsx(+38)src/screens/reader/components/ReaderBottomSheet/ReaderTTSTab.tsx(+60) — per-novel cleanup UI + reorder/validationsrc/screens/settings/SettingsReaderScreen/Modals/TtsTextCleanupModal.tsx(+141) — presets UI, import/export UI, Share integrationsrc/services/__tests__/NovelTtsSettings.test.ts(+70)New
src/screens/settings/SettingsReaderScreen/Modals/TtsTextCleanupModal.tsxsrc/screens/settings/SettingsReaderScreen/Modals/ttsCleanupPresets.ts— preset data +applyPresetToSettings,serializeCleanupSettings,parseCleanupSettingsImportsrc/screens/settings/SettingsReaderScreen/Modals/__tests__/ttsCleanupPresets.test.ts(21 tests)src/utils/__tests__/ttsTextCleanup.test.ts(32 tests)Commits (local, not pushed — 9 total)
2d35beff0feat(tts): add declarative text cleanup pipeline for TTS audio1a1ffb00dfix(tts): harden text cleanup regex safety and replacement semanticsaeec7abb6feat(tts): support per-novel text cleanup overridesb0b56f333feat(tts): add substring match mode and rule reordering to cleanup editorb47f527c0test(tts): assert cleanup wiring in controller integration testse4a78d623chore(tts): include stable sync callback in MMKV listener depsa638ebe7efix(tts): address text cleanup audit findings (modal robustness, docs, PRD)2d90673edfix(tts): resolve remaining exhaustive-deps warnings20b55b719feat(tts): add cleanup presets + JSON import/export to cleanup editor9. Out of scope (explicitly rejected)
getTextNodes+executeCustomUserReaderJSsynchronous hook (mechanism doesn't exist; wouldn't affect the initial queue; cannot run in background mode) — see §2.1eval/Function)10. User Guide — Visual Menu Map (ASCII + Mermaid)
10.1 How to access the menu
Both entry points converge on the same editor — navigation flow:
flowchart LR subgraph A["PATH A · GLOBAL SETTINGS (all novels)"] direction LR A1[More ...] --> A2[Settings] --> A3[Reader] A3 --> A4[Accessibility tab] A4 --> A5[TTS Text Cleanup section] end subgraph B["PATH B · QUICK ACCESS (per-chapter)"] direction LR B1[Open a chapter] --> B2[TTS bar / bottom sheet] B2 --> B3[TTS tab] B3 --> B4[Text Cleanup section] end A5 --> M[TtsTextCleanupModal] B4 --> MWhere the section sits on the Accessibility tab (global settings):
Where the section sits on the TTS tab (quick access, per-chapter):
Both paths open the same editor:
10.2 The rule editor (tap [+ Add rule] or the pencil [E])
10.3 The phonetic pair editor (tap [+ Add phonetic pair] or [E])
10.4 What happens to your text (pipeline order)
flowchart TD P["Every paragraph (ANY playback path)"] --> N["1. Unicode normalization<br/>(optional: NFD + strip combining marks)"] N --> R["2. Find & Replace rules<br/>(ordered top-to-bottom, literal or regex, empty replacement = strip)"] R --> D["3. Phonetic dictionary<br/>(whole-word, or substring for CJK)"] D --> T["Native TTS engine"]10.5 Where a save goes (global vs per-novel)
flowchart TD S["Save cleanup settings"] --> E{"Which entry point?"} E -->|"PATH A - Accessibility tab"| G["GLOBAL settings<br/>(applied to every novel)"] E -->|"PATH B - Reader TTS tab"| M{"Per-novel mode ON?"} M -->|"OFF"| G M -->|"ON"| PN["PER-NOVEL override<br/>(this novel only - others keep global)"] PN --> TAG["List row shows the 'per-novel' tag"]10.6 Limitations (read before building rules)
10.7 How to use it properly (quick start)
10.8 Quick-reference examples
10.9 Presets + JSON import/export (commit
20b55b719)Why presets are not "hardcoded site regexes": the presets are shipped
as UI template data and only enter the pipeline after a user taps Add,
landing in their own editable MMKV settings. The cleanup engine itself
remains site-agnostic — satisfying the issue's explicit request to keep the
repository lightweight (see §9).