Skip to content

fix(stt): ship GGML Whisper models — the shipped ONNX exports can never load#2

Open
BernardMasika wants to merge 1 commit into
mainfrom
fix/stt-ggml-models
Open

fix(stt): ship GGML Whisper models — the shipped ONNX exports can never load#2
BernardMasika wants to merge 1 commit into
mainfrom
fix/stt-ggml-models

Conversation

@BernardMasika

Copy link
Copy Markdown

What this fixes

STT never worked as shipped: WhisperManager.InitModel fails on every model the install docs deliver. This is bug B3 from the integration test report — the STT stage was dead end-to-end for any consumer.

Root cause

A format mismatch between the models and the engine. Macoron/whisper.unity wraps whisper.cpp, whose native model format is single-file GGML (ggml-*.bin from ggerganov/whisper.cpp). What the setup script downloads is the onnx-community multi-file ONNX export (encoder + decoder + tokenizer + configs) — a format whisper.cpp has no loader for. The samples then point WhisperManager.ModelPath at encoder_model_quantized.onnx, which can never load.

The fix

Swap every STT surface to GGML:

  • tools/setup-sauti.sh downloads ggml-tiny.en.bin (75 MB, essential profile) / ggml-small.en.bin (466 MB, --models all) from ggerganov/whisper.cpp, SHA-256-pinned, placed flat under StreamingAssets/VoiceAI/stt/. GGML models are self-contained — tokenizer, config, and both graphs live inside the one .bin, so the five-file-per-variant layout goes away entirely.
  • Samples 02 / 05 / 06 resolve a single GGML file via a sttModelFilePreference list and hand it straight to WhisperManager.ModelPath (replacing the subdir + ONNX anchor-file scheme).
  • SautiSetupWizard model check now looks for stt/ggml-tiny.en.bin.
  • Metadata/docs: ai-models/stt/manifest.json, ai-models/manifest.json, ai-models/stt/README.md, docs/reference/models.md (+ bundle-size matrix), docs/experiments/02|05|06, the architecture asset-flow diagram, SHIP_READINESS.md, and NOTICE all updated. Licensing is unchanged in substance — OpenAI Whisper is MIT; the GGML conversions by Georgi Gerganov are MIT too.

Size trade-off worth knowing: GGML Small is 466 MB vs 239 MB for the ONNX pair (flagship bundle grows ~1.6 → ~1.8 GiB), and GGML Tiny is 75 MB vs 43 MB. The difference buys an STT stage that actually loads; .en variants keep the English-only contract.

Verified (consumer project: Unity 6000.0.70f1, whisper.unity master, desktop CPU)

  • ggml-tiny.en.bin loads via WhisperManager.InitModel() in ~1.8 s; the shipped ONNX files fail to load in the same setup.
  • End-to-end round trip: fed Kokoro-TTS-generated audio of "Justice is the foundation of the Baraza courtroom." into GetTextAsync → transcribed in ~0.8 s as " Ustice is the foundation of the Barraza courtroom." — Whisper Tiny on synthetic speech, so effectively perfect.
  • Compile clean in the consumer project. Note: the EditMode test suite doesn't cover the changed surfaces (samples ship as source, the wizard check is editor-UI) — the live load + transcription above is the meaningful verification for this change.

Note on versioning

I deliberately did not bump package.json — this PR and #1 would collide on it. The changelog entry sits under [Unreleased]; version it as you see fit when releasing (1.3.6 if this lands after #1).

Macoron/whisper.unity wraps whisper.cpp, whose native model format is
single-file GGML. The onnx-community multi-file Whisper exports shipped
since v1.2 are not loadable by that engine, so the STT stage was broken
end-to-end for every consumer following the install docs.

- tools/setup-sauti.sh: STT downloads swapped to ggerganov/whisper.cpp
  GGML models, SHA-pinned — ggml-tiny.en.bin (75 MB, essential profile)
  and ggml-small.en.bin (466 MB, --models all). Placed flat under
  StreamingAssets/VoiceAI/stt/ (GGML is self-contained; no tokenizer/
  config sidecars).
- Samples 02 / 05 / 06: model resolution is now a GGML file-preference
  list (sttModelFilePreference) passed straight to WhisperManager.ModelPath,
  replacing the subdir + ONNX anchor-file scheme.
- SautiSetupWizard: on-disk model check looks for stt/ggml-tiny.en.bin.
- ai-models/stt/manifest.json, ai-models/manifest.json, stt/README.md,
  docs/reference/models.md, docs/experiments/*, developer-guide
  architecture diagram, NOTICE: all updated to the GGML entries
  (licenses unchanged — OpenAI Whisper is MIT; GGML conversion by
  Georgi Gerganov, MIT).

Verified in a consumer project (Unity 6000.0.70f1, whisper.unity master):
ggml-tiny.en.bin loads via WhisperManager.InitModel in ~1.8 s and
correctly transcribes Kokoro-generated speech ("Justice is the foundation
of the Baraza courtroom.") in ~0.8 s desktop CPU. The previously shipped
ONNX files fail to load in the same setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant