fix(tts): CMUDict-backed G2P + designer pronunciation overrides#1
Open
BernardMasika wants to merge 2 commits into
Open
fix(tts): CMUDict-backed G2P + designer pronunciation overrides#1BernardMasika wants to merge 2 commits into
BernardMasika wants to merge 2 commits into
Conversation
…t support
The built-in G2P covered ~120 hand-picked words; everything else fell to a
letter-by-letter spell-out, so most real sentences sounded broken. Root cause
was dictionary size, not the ARPABET-to-IPA conversion.
- EnglishG2P.LoadCmudict(path): loads a CMU Pronouncing Dictionary
(cmudict.dict format, ~125k words). Pure System.IO, stays Unity-free.
Lookup order: CommonWords -> CMUDict -> letter-spell fallback.
- EnglishG2P.UnknownWords: diagnostic set of words that fell through to the
letter-spell path (surfaces proper nouns needing hand-tuned entries).
- KokoroTtsRunner: optional 4th ctor arg cmudictPath, loaded during lazy
init. Existing 3-arg callers are unaffected.
- SautiVoiceProfile.g2pDictionaryPathRelative (default
VoiceAI/tts/cmudict.dict) + ResolveG2PDictionaryPath(); SautiSpeaker
passes it through.
- tools/setup-sauti.sh downloads cmudict.dict (3.5 MB, BSD-2-Clause,
pinned cmusphinx commit) with the essential model set.
CMUDict was chosen over espeak-ng deliberately: espeak-ng is GPL, which
conflicts with this package's Apache-2.0 license; CMUDict is BSD-2-Clause.
Measured on the shipped SynthesizeAsync ("The council will now deliberate
the verdict."): 6.53 s of mostly spelled-out audio -> 3.80 s pronounced.
Backward-compatible: absent the dictionary file, behaviour is unchanged.
CMUDict covers general English, but proper nouns and invented words (character names, places) will always be out-of-vocabulary and fall to the letter-spell path. This makes fixing them a data edit instead of a source edit. - EnglishG2P.AddOverride(word, arpabet[]) / ClearOverrides() / OverrideCount: a highest-priority pronunciation layer. Lookup order is now Overrides -> CommonWords -> CMUDict -> letter-spell. Case-insensitive; thread-safe (copy-on-write, lock-free reads on the synthesis path); pure C#. Invalid ARPABET tokens throw ArgumentException immediately so typos surface instead of silently degrading audio. Registering a word also removes it from the UnknownWords diagnostic. - SautiPronunciationOverrides ScriptableObject (Assets > Create > Sauti > Pronunciation Overrides): a word -> ARPABET list with an inline phoneme key in the tooltips. Apply() registers every entry; invalid entries are logged and skipped so one typo does not block the rest. - SautiSpeaker gets a pronunciationOverrides slot, applied automatically when its runner initialises (after CMUDict loads; overrides win anyway). - EditMode tests (EnglishG2POverrideTests) pin the lookup order, case-insensitivity, validation, diagnostic behaviour, and the ScriptableObject surface. Intended workflow: synthesise once -> read EnglishG2P.UnknownWords -> author overrides for what it lists. Example: baraza -> B AA0 R AA1 Z AA0. Verified in a consumer project (Unity 6000.0.70f1): 65/65 EditMode tests pass; A/B synthesis of "Justice is the foundation of the Baraza courtroom." goes from letter-spelled (4.50 s, UnknownWords=[baraza]) to pronounced (4.00 s, UnknownWords empty) via the ScriptableObject path. Backward-compatible: with no overrides registered, behaviour is unchanged.
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
TTS output for anything beyond the built-in word list sounded broken: words were spelled out letter-by-letter ("d-e-l-i-b-e-r-a-t-e") instead of pronounced. I hit this immediately when integrating Sauti into Baraza VR — almost every real sentence contains words outside the ~120-word table.
Root cause
Not the ARPABET→IPA conversion — that part is solid. The ceiling was purely dictionary size:
EnglishG2P.CommonWordshas ~120 entries, and everything else fell through to the letter-spell fallback.The fix (two commits)
Commit 1 — CMUDict support (the ceiling lift).
EnglishG2P.LoadCmudict(path)loads a CMU Pronouncing Dictionary (cmudict.dictformat, ~125k words). PureSystem.IO, so the file stays Unity-free and dotnet-lintable.KokoroTtsRunnergets an optional 4th ctor argcmudictPath(existing 3-arg callers unaffected),SautiVoiceProfilegetsg2pDictionaryPathRelative(defaultVoiceAI/tts/cmudict.dict), andsetup-sauti.shdownloads the dictionary (3.5 MB, pinned cmusphinx commit) with the essential model set. A newEnglishG2P.UnknownWordsdiagnostic set surfaces whatever still falls through (proper nouns).Why CMUDict and not espeak-ng: espeak-ng is GPL, which conflicts with this package's Apache-2.0 license. CMUDict is BSD-2-Clause.
Commit 2 — pronunciation overrides (the proper-noun path).
Proper nouns will always be out-of-vocabulary, so there's now a highest-priority override layer:
EnglishG2P.AddOverride(word, arpabet[])/ClearOverrides()— lookup order becomes Overrides → CommonWords → CMUDict → letter-spell. Case-insensitive, thread-safe (copy-on-write; lock-free reads on the synthesis path). Invalid ARPABET tokens throw immediately so typos can't silently degrade audio.SautiPronunciationOverridesScriptableObject (Assets → Create → Sauti → Pronunciation Overrides) — designers addword → ARPABETentries with a phoneme key right in the tooltips; aSautiSpeakerslot applies it automatically. Invalid entries are logged and skipped.EnglishG2POverrideTests) pin the lookup order, validation, and the ScriptableObject surface.Measured results (consumer project: Unity 6000.0.70f1, desktop CPU)
baraza → B AA0 R AA1 Z AA0via the ScriptableObject: letter-spelled 4.50 s +UnknownWords=[baraza]→ pronounced 4.00 s +UnknownWordsempty.Both commits are fully backward-compatible: no dictionary file and no overrides → behaviour is byte-identical to before.
FYI (separate issue, not addressed here)
The README's git-URL install (
?path=packaging/com.sauti.voice-ai) delivers a package with no code — thepackaging/folder only gets populated bytools/package-sauti.sh(which also needsrsync, absent on Windows). I installed by hand-staging +npm pack. Worth a look when you get a chance; happy to help fix that too.