Skip to content
Merged
82 changes: 49 additions & 33 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,41 +30,42 @@ pnpm run test:tts-wake-cycle

## Current Task

Production Readiness Action Plan Implementation (2026-01-03) - ✅ COMPLETED
TTS Text Cleanup Pipeline (2026-08-02) - ✅ COMPLETED

### Phase 1: Critical Security & Bug Fixes (P0) - ✅ COMPLETED

- **1.1**: Fixed cookie value truncation in WebviewScreen.tsx (handles values with `=`)
- **1.2**: Fixed Set-Cookie header parsing (handles comma + newline separation, equals in values)
- **1.3**: Replaced harsh `System.exit(0)` with graceful shutdown in DoHManagerModule.kt
- **1.4**: Added 5-second timeouts to DoH bootstrap client (connectTimeout, readTimeout, writeTimeout)
- **1.5**: Changed SharedPreferences from `apply()` to `commit()` (synchronous writes to prevent data loss)
- **1.6**: Removed obsolete `.pnpm-patches/cookies/` directory (kept patches/ directory)

### Phase 2: Security Hardening (P1) - ✅ COMPLETED

- **2.1**: ~~Added certificate pinning for DoH providers~~ → **REMOVED** (2026-01-11)
- Certificate pinning removed per OWASP 2025 guidance
- Reason: Third-party DoH providers rotate certs unpredictably, causing app outages
- Android's platform trust store + Certificate Transparency provides sufficient security
- See: OWASP Pinning Cheat Sheet - "don't pin if you don't control both sides"
- **2.2**: User confirmation dialog for app restart already implemented
- **2.3**: Cookie attribute filtering already implemented

### Phase 4: Performance Optimization (P2) - ✅ COMPLETED

- **4.1**: Increased TTS chapter list debounce from 500ms → 2000ms (4x reduction in DB queries)
- **4.2**: Code already optimized (single refresh pattern, no major duplication)
- **4.3**: Added React.memo with custom equality to ChapterItem (prevents re-renders on non-progress changes)

### Phase 5: Testing & Documentation (P2) - ✅ COMPLETED

- **5.1**: Cookie parsing tests comprehensive (special chars, URL encoding, multiple cookies, etc.)
- **5.2**: Updated AGENTS.md with completed task status
- **Tests**: All 1072 tests passing (zero regressions)
- **Feature**: Declarative, length-preserving text cleanup applied to every paragraph before it reaches the native TTS engine across ALL playback paths (initial queue, WebView tts-queue refills, fallback single-speak)
- **Capabilities**: Ordered find/replace + regex strip rules, phonetic pronunciation dictionary (whole-word/substring match), optional Unicode normalization, per-novel overrides
- **UI**: Global in Settings → Reader → Accessibility Tab; Quick access in Reader Bottom Sheet → TTS Tab ("Text Cleanup" section)
- **Commits**: 2d35beff0 (pipeline), 1a1ffb00d (regex safety), aeec7abb6 (per-novel overrides), b0b56f333 (substring mode + reorder), b47f527c0 (wiring tests), e4a78d623 (listener deps) — all 2026-08-02, branch `dev` (not yet pushed)
- **Tests**: 1235 passing (zero regressions)
- **Docs**: PRD at specs/tts-text-cleanup/PRD.md

### Previous Completed Tasks

- Production Readiness Action Plan Implementation (2026-01-03) - ✅ COMPLETED
- **Phase 1: Critical Security & Bug Fixes (P0)** - ✅ COMPLETED
- **1.1**: Fixed cookie value truncation in WebviewScreen.tsx (handles values with `=`)
- **1.2**: Fixed Set-Cookie header parsing (handles comma + newline separation, equals in values)
- **1.3**: Replaced harsh `System.exit(0)` with graceful shutdown in DoHManagerModule.kt
- **1.4**: Added 5-second timeouts to DoH bootstrap client (connectTimeout, readTimeout, writeTimeout)
- **1.5**: Changed SharedPreferences from `apply()` to `commit()` (synchronous writes to prevent data loss)
- **1.6**: Removed obsolete `.pnpm-patches/cookies/` directory (kept patches/ directory)
- **Phase 2: Security Hardening (P1)** - ✅ COMPLETED
- **2.1**: ~~Added certificate pinning for DoH providers~~ → **REMOVED** (2026-01-11)
- Certificate pinning removed per OWASP 2025 guidance
- Reason: Third-party DoH providers rotate certs unpredictably, causing app outages
- Android's platform trust store + Certificate Transparency provides sufficient security
- See: OWASP Pinning Cheat Sheet - "don't pin if you don't control both sides"
- **2.2**: User confirmation dialog for app restart already implemented
- **2.3**: Cookie attribute filtering already implemented
- **Phase 4: Performance Optimization (P2)** - ✅ COMPLETED
- **4.1**: Increased TTS chapter list debounce from 500ms → 2000ms (4x reduction in DB queries)
- **4.2**: Code already optimized (single refresh pattern, no major duplication)
- **4.3**: Added React.memo with custom equality to ChapterItem (prevents re-renders on non-progress changes)
- **Phase 5: Testing & Documentation (P2)** - ✅ COMPLETED
- **5.1**: Cookie parsing tests comprehensive (special chars, URL encoding, multiple cookies, etc.)
- **5.2**: Updated AGENTS.md with completed task status
- **Tests**: All 1072 tests passing (zero regressions)

- TTS Chapter List Progress Sync - Real-Time Fix (2026-01-03) - ✅ COMPLETED
- **Bug**: Chapter List showed stale progress during active TTS playback
- **Solution**: Added debounced `refreshChaptersFromContext()` call during paragraph progress saves (500ms debounce)
Expand Down Expand Up @@ -113,6 +114,18 @@ Production Readiness Action Plan Implementation (2026-01-03) - ✅ COMPLETED

## Recent Fixes

### TTS Text Cleanup Pipeline (2026-08-02) - ✅ COMPLETED

- **Feature**: Declarative, length-preserving text cleanup applied to every paragraph before it reaches the native TTS engine across ALL playback paths (initial queue, WebView tts-queue refills, fallback single-speak)
- **Capabilities**: Ordered find/replace + regex strip rules (literal or regex), phonetic pronunciation dictionary (whole-word/substring match), optional Unicode normalization (NFD + strip combining marks), per-novel overrides
- **Design Constraints**: No eval/Function in the RN layer; length-preserving (RN ↔ WebView paragraph index contract stays intact); no hardcoded site-specific regexes
- **Regex Safety**: 200-char length cap, catastrophic-backtracking shape detection, compile-time try/catch, literal replacement semantics (`$&` stays literal), sticky `y` flag dropped
- **Effective Settings**: `resolveEffectiveTtsCleanup()` → per-novel override when per-novel TTS enabled AND cleanup saved, else global; synced into `chapterGeneralSettingsRef` via `syncEffectiveTtsCleanup` (prop effect + MMKV listener + per-novel effect)
- **Files**: htmlParagraphExtractor.ts (pipeline, +350 lines), TtsTextCleanupModal.tsx (new, 685 lines), useSettings.ts, novelTtsSettings.ts, useTTSController.ts, useTTSUtilities.ts, WebViewReader.tsx, AccessibilityTab.tsx, ReaderTTSTab.tsx
- **Commits**: 2d35beff0 (pipeline), 1a1ffb00d (regex safety), aeec7abb6 (per-novel overrides), b0b56f333 (substring mode + reorder), b47f527c0 (wiring tests), e4a78d623 (listener deps)
- **Tests**: 1235 passing (zero regressions)
- **Docs**: PRD at specs/tts-text-cleanup/PRD.md

### TTS highlight offset page reload and button state desync fix (2026-05-26) - ✅ COMPLETED

- **Bug #1 - WebView reload on offset adjustment**: ✅ COMPLETED - Adjusting highlight offset does not reload reader
Expand Down Expand Up @@ -300,7 +313,9 @@ React Native Layer
├── TTSAudioManager.ts Native module wrapper
├── TTSState.ts State machine definition
├── ttsBridge.ts RN↔WebView bridge
└── ttsNotification.ts Media notification utils
├── ttsNotification.ts Media notification utils
├── htmlParagraphExtractor.ts Paragraph extraction + text cleanup pipeline
└── novelTtsSettings.ts Per-novel TTS settings + cleanup overrides

WebView Layer
└── core.js DOM parsing, highlighting, scroll
Expand All @@ -316,7 +331,8 @@ UI Components
├── TTSManualModeDialog.tsx Manual mode activation
├── TTSScrollSyncDialog.tsx Position mismatch
├── TTSChapterSelectionDialog.tsx Chapter picker
└── TTSExitDialog.tsx Exit confirmation
├── TTSExitDialog.tsx Exit confirmation
└── TtsTextCleanupModal.tsx Text cleanup rules & phonetic editor
```

## Path Aliases
Expand Down
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ This fork builds on the original LNReader with enhanced features focused on acce
- [TTS Feature Demo](#tts-feature-demo)
- [Key TTS Features Showcase](#key-tts-features-showcase)
- [Enhanced TTS Media Notification (Android)](#enhanced-tts-media-notification-android)
- [TTS Text Cleanup](#tts-text-cleanup)
- [Reader Experience](#reader-experience)
- [Network \& Security](#network--security)
- [UI \& Accessibility](#ui--accessibility)
Expand All @@ -72,6 +73,7 @@ This fork builds on the original LNReader with enhanced features focused on acce
- [Getting Started](#getting-started)
- [First-Time Setup](#first-time-setup)
- [Using TTS](#using-tts)
- [Using TTS Text Cleanup](#using-tts-text-cleanup)
- [Using Continuous Scrolling](#using-continuous-scrolling)
- [Backup \& Restore](#backup--restore)
- [Architecture](#architecture)
Expand Down Expand Up @@ -116,6 +118,7 @@ This fork includes extensive TTS enhancements for hands-free reading and accessi
| 🔧 **TTS Engine Picker** | Select system or custom TTS engines with quality badges and persistent selection |
| 🏷️ **Auto Chapter Title Prepend** | Auto-announces chapter title via TTS when not visibly present in content |
| 🖱️ **Advanced Button Gestures** | Tap to toggle playback, hold 0.5s + swipe to adjust highlight offset, hold 2s + drag to move |
| 🧹 **TTS Text Cleanup** | Strip watermarks/corrupted text & fix pronunciations before TTS reads — declarative rules, one-tap presets, per-novel overrides, JSON import/export |

</div>

Expand Down Expand Up @@ -189,6 +192,40 @@ Android devices can have multiple TTS engines installed. The default engine is o

---

#### TTS Text Cleanup

Sites like Novelight inject anti-scraper watermarks (spaced letters, unicode lookalikes, `u2014` corruption, "Do not rehost this novel" spam) that system TTS engines read aloud, and LN names/honorifics are commonly mispronounced. **Text Cleanup** fixes both with a declarative, **length-preserving** pipeline applied to every paragraph before it reaches the TTS engine — across all playback modes, including background playback.

<div align="center">

```mermaid
flowchart LR
%% Styles
classDef src fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#0d47a1,font-weight:bold
classDef clean fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#e65100,font-weight:bold
classDef out fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1b5e20,font-weight:bold

%% Nodes
PARA["📄 Every Paragraph<br/>(any playback path)"]:::src
NORM["🔤 1. Unicode Normalization<br/>(optional: NFD + strip combining marks)"]:::clean
RULES["🧹 2. Find &amp; Replace Rules<br/>(literal or regex, ordered)"]:::clean
PHON["🗣️ 3. Phonetic Dictionary<br/>(whole-word or substring)"]:::clean
TTS["🔈 Native TTS Engine"]:::out

%% Flow
PARA ==> NORM ==> RULES ==> PHON ==> TTS
```

</div>

- **Access**: Settings → Reader → Accessibility → **TTS Text Cleanup** (global), or Reader Bottom Sheet → TTS Tab → **Text Cleanup** (quick access, per-novel aware)
- **Presets (one-tap)**: Curated templates — Novelight spaced watermark, `u2014` corruption, "(Official version)" tags, "Do not rehost" spam, math-bold lookalikes, LN name pronunciations, and CJK substring pairs. Presets are UI data only; applying copies them into your editable rules
- **Import / Export**: Share or restore your rule set as a versioned JSON envelope (`lnreader-tts-cleanup` v1)
- **Per-novel overrides**: With per-novel TTS settings enabled, cleanup can be overridden per novel
- **Safety**: Regex length cap + ReDoS-shape detection + invalid-regex skip; length-preserving (paragraph count never changes, so highlight/scroll stay in sync)

---

### Reader Experience

Enhanced features for smoother, more immersive reading.
Expand Down Expand Up @@ -278,6 +315,9 @@ Robust backup system with multiple options and versioned schema.

### TTS Enhancements

- **TTS Text Cleanup**: Declarative rule pipeline strips anti-scraper watermarks, corrupted text, and fixes LN name pronunciations before TTS reads (Settings → Reader → Accessibility → TTS Text Cleanup; quick access in the Reader TTS tab)
- **Cleanup Presets**: One-tap curated templates (Novelight watermark, `u2014` corruption, LN/CJK pronunciations, and more) — UI data only, copied into your editable rules
- **Cleanup Import/Export**: Share or restore cleanup rule sets as versioned JSON from the editor
- **TTS Engine Picker**: Custom engine selection with native Android integration, quality badges, and persistent selection across sessions
- **Per-Novel TTS Settings**: Isolated voice/speed/pitch per novel — changes no longer overwrite global defaults
- **TTS Resume Reliability**: Fixed resume playback failure and wrong engine audio output after interruptions
Expand Down Expand Up @@ -325,6 +365,16 @@ View full changelog: [RELEASE_NOTES.md](RELEASE_NOTES.md)
- **Hold 0.5s (Amber Glow) + Swipe Up/Down**: Adjust paragraph highlight offset (resets per chapter).
- **Hold 2.0s (Teal Glow) + Drag**: Move the TTS button anywhere on the screen.

### Using TTS Text Cleanup

1. **Enable**: Settings → Reader → Accessibility → TTS Text Cleanup → toggle "Clean TTS text" ON (or open Reader → TTS Tab → Text Cleanup)
2. **One-tap presets**: Tap **Add** on a preset (e.g. "Fix `u2014` text corruption") to copy curated rules into your list — deduplicated, fully editable
3. **Custom rules**: Add find/replace rules (literal or regex) to strip watermarks or corrupted text
4. **Phonetic dictionary**: Add name pronunciations (e.g. `Xianxia` → "Shee-an-shah"); enable **substring** mode for unspaced CJK names
5. **Unicode normalization**: Optionally NFD-normalize + strip combining marks to fix lookalike characters
6. **Import / Export**: From the editor, share your rule set as JSON (Export) or restore one (Import — replaces the draft, Cancel discards)
7. **Verify**: Play a chapter in foreground and background — cleaned text is spoken and paragraph highlighting stays in sync

### Using Continuous Scrolling

1. Navigate to Settings → Reader
Expand Down Expand Up @@ -363,6 +413,7 @@ The TTS engine uses a **Hybrid 3-Layer Architecture** to ensure reliable playbac
- **Proactive Queue Refill**: Monitors queue size and refills (batch size ~20) before depletion to prevent audio gaps.
- **State Reconciliation**: On load, syncs progress from three sources: Database (permanent), MMKV (fast), and Native (current utterance).
- **Smart Wake-Up**: Detects app foregrounding and seamlessly syncs the visual reader position with the background audio position.
- **Declarative Text Cleanup**: Settings-driven find/replace + phonetic rules applied length-preserving at every audio entry point (initial queue, WebView DOM refills, fallback single-speak) — no arbitrary JS, no hardcoded site regexes.

<div align="center">

Expand Down
Loading
Loading