feat(silero-native): precise word timestamps from dur_hat (#145) - #181
Merged
Conversation
Replace the char-proportional estimation with timestamps derived from the model's own duration predictor: tts_main's 4th output dur_hat gives per-symbol frame durations (12.5 ms @ 48 kHz); the engine converts them to the exact integer frame counts the graph renders (trunc(dur + 0.5), sos/eos clamps baked in) and pairs each symbol with the char it was emitted from (build_sequence provenance). timestamps.rs aligns the original chunk text's words to that symbol stream letter-by-letter (ё ≡ е; a leading '+' stress-marker run opens the word range — the model renders real audio frames for it), so word boundaries follow the model's own durations and punctuation pauses become gaps. The outward contract is unchanged (WordTimestamp, ms rounding, sorted non-overlapping, original_pos via char_mapping). - bundle-gated parity: per-symbol frames match the Python ONNX reference, frame sum reproduces the waveform length, first-word onsets within 50 ms of the reference, timestamps invariant across 48k/24k/8k output rates - parity fixtures regenerated with reference dur_hat + sequence - Piper and the Python ttsd engine keep their existing estimation Closes #145
…ymbol stream A word letter that never reached the model stream (latin chars, digits — the frontend drops them) used to break the alignment loop on the first mismatch, shifting every following word by one: a mixed word like "get_variablesслэш" cascaded ~70 words into zero-length timestamps and the highlight raced ahead. Skip the dropped letter instead of stopping; the stream cursor now only advances on matches. Also treat standalone digits/'_' inside words as skippable rather than a stop. Covered by unit tests, a playback-emulation regression case in the bundle-gated parity suite, and a full-text emulation run (0 violations).
Sync the delta spec into specs/silero-native-engine (Word Timestamps requirement rewritten for dur_hat-derived timestamps, six scenarios) and move the change to changes/archive/.
Archive commits no longer require draft approval; all other GitHub-bound texts still do.
An attached '+' run was folded into a word's range before its first letter was checked, so an unspoken word ahead of it (latin "usb" before the accentor's "+яблоко") stole the marker frames and orphaned them in a gap. Commit the fold only when the letter after the run matches the word; otherwise leave the cursor for the next word. Pinned by a regression test. Also from the pre-PR review: anchor the parity suite on both timeline ends (last word end vs the reference cumsum up to the last letter symbol), drop the per-chunk ids clone via BuiltSequence destructuring, and clamp ts_start to the chunk duration so end <= duration_sec holds by construction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace the char-proportional word-timestamp estimator in the
silero-native engine with exact timestamps derived from the model's own
duration predictor (
dur_hat, the 4th output oftts_main.onnx), fixinghighlight drift on long texts (#145). Piper and ttsd keep their existing
approximate estimators.
Changes
build_sequencenow returns per-symbol char provenance(
BuiltSequence { ids, chars }, sos/eos included).Engine::synthesizereadsdur_hat, converts it to the exact integerframe counts the exported graph renders (
trunc(dur + 0.5)) andreturns
Vec<SymbolDuration>per chunk; typed error if the output ismissing.
timestamps.rsrewritten: letter-level alignment of the originaltext's words to the symbol stream (ё ≡ е, non-letters skipped,
attached
+stress markers folded into the word's onset, letters thefrontend dropped are skipped without consuming the stream so one
unmapped word cannot misalign the chunk, zero-length fallback for
unspoken words).
WordTimestampcontract unchanged.sequence+dur_hat;new bundle-gated suite: frame counts match the reference and the
rendered waveform, word onsets and the last word's end within 50 ms
of the reference, sample-rate invariance (8/24/48 kHz), and a
playback-emulation regression test asserting highlighting never
jumps ahead.
specs/silero-native-enginesynced.Bugs found and fixed during the manual pass
+("правило + команда") cascaded following wordsinto zero-length timestamps → attached-only
+rule.the whole chunk by one word → skip-dropped-letter alignment.
"+яблоко") stole the marker frames → deferred
+fold.Validation
cargo test --manifest-path silero-native/Cargo.tomlgreen (68 unit +bundle-gated parity tier)
just lintgreensteps): 0 highlight jumps