fix(stt): ship GGML Whisper models — the shipped ONNX exports can never load#2
Open
BernardMasika wants to merge 1 commit into
Open
fix(stt): ship GGML Whisper models — the shipped ONNX exports can never load#2BernardMasika wants to merge 1 commit into
BernardMasika wants to merge 1 commit into
Conversation
Macoron/whisper.unity wraps whisper.cpp, whose native model format is
single-file GGML. The onnx-community multi-file Whisper exports shipped
since v1.2 are not loadable by that engine, so the STT stage was broken
end-to-end for every consumer following the install docs.
- tools/setup-sauti.sh: STT downloads swapped to ggerganov/whisper.cpp
GGML models, SHA-pinned — ggml-tiny.en.bin (75 MB, essential profile)
and ggml-small.en.bin (466 MB, --models all). Placed flat under
StreamingAssets/VoiceAI/stt/ (GGML is self-contained; no tokenizer/
config sidecars).
- Samples 02 / 05 / 06: model resolution is now a GGML file-preference
list (sttModelFilePreference) passed straight to WhisperManager.ModelPath,
replacing the subdir + ONNX anchor-file scheme.
- SautiSetupWizard: on-disk model check looks for stt/ggml-tiny.en.bin.
- ai-models/stt/manifest.json, ai-models/manifest.json, stt/README.md,
docs/reference/models.md, docs/experiments/*, developer-guide
architecture diagram, NOTICE: all updated to the GGML entries
(licenses unchanged — OpenAI Whisper is MIT; GGML conversion by
Georgi Gerganov, MIT).
Verified in a consumer project (Unity 6000.0.70f1, whisper.unity master):
ggml-tiny.en.bin loads via WhisperManager.InitModel in ~1.8 s and
correctly transcribes Kokoro-generated speech ("Justice is the foundation
of the Baraza courtroom.") in ~0.8 s desktop CPU. The previously shipped
ONNX files fail to load in the same setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
STT never worked as shipped:
WhisperManager.InitModelfails on every model the install docs deliver. This is bug B3 from the integration test report — the STT stage was dead end-to-end for any consumer.Root cause
A format mismatch between the models and the engine.
Macoron/whisper.unitywraps whisper.cpp, whose native model format is single-file GGML (ggml-*.binfromggerganov/whisper.cpp). What the setup script downloads is theonnx-communitymulti-file ONNX export (encoder + decoder + tokenizer + configs) — a format whisper.cpp has no loader for. The samples then pointWhisperManager.ModelPathatencoder_model_quantized.onnx, which can never load.The fix
Swap every STT surface to GGML:
tools/setup-sauti.shdownloadsggml-tiny.en.bin(75 MB, essential profile) /ggml-small.en.bin(466 MB,--models all) fromggerganov/whisper.cpp, SHA-256-pinned, placed flat underStreamingAssets/VoiceAI/stt/. GGML models are self-contained — tokenizer, config, and both graphs live inside the one.bin, so the five-file-per-variant layout goes away entirely.sttModelFilePreferencelist and hand it straight toWhisperManager.ModelPath(replacing the subdir + ONNX anchor-file scheme).SautiSetupWizardmodel check now looks forstt/ggml-tiny.en.bin.ai-models/stt/manifest.json,ai-models/manifest.json,ai-models/stt/README.md,docs/reference/models.md(+ bundle-size matrix),docs/experiments/02|05|06, the architecture asset-flow diagram,SHIP_READINESS.md, andNOTICEall updated. Licensing is unchanged in substance — OpenAI Whisper is MIT; the GGML conversions by Georgi Gerganov are MIT too.Size trade-off worth knowing: GGML Small is 466 MB vs 239 MB for the ONNX pair (flagship bundle grows ~1.6 → ~1.8 GiB), and GGML Tiny is 75 MB vs 43 MB. The difference buys an STT stage that actually loads;
.envariants keep the English-only contract.Verified (consumer project: Unity 6000.0.70f1, whisper.unity master, desktop CPU)
ggml-tiny.en.binloads viaWhisperManager.InitModel()in ~1.8 s; the shipped ONNX files fail to load in the same setup.GetTextAsync→ transcribed in ~0.8 s as " Ustice is the foundation of the Barraza courtroom." — Whisper Tiny on synthetic speech, so effectively perfect.Note on versioning
I deliberately did not bump
package.json— this PR and #1 would collide on it. The changelog entry sits under[Unreleased]; version it as you see fit when releasing (1.3.6 if this lands after #1).