feat(voice): server-STT vocabulary bias + a device preference for server transcription - #697
Merged
Merged
Conversation
…ry, and let a device prefer it The third fix behind "the app struggles to understand me". Server-side transcription (Whisper) can be told which words to expect; the on-device recognizer that a phone picks by default cannot, and it is the one turning "komodo" into "Kardashian". - Engine: `POST /api/assistant/stt` accepts an optional `prompt` text field and forwards it to `/audio/transcriptions` as the backend's bias prompt, trimmed and bounded to 2 KiB. A backend that ignores it is no worse off; a client that sends none transcribes exactly as before. - Client: `sttVocabularyPrompt` builds that bias from the project slugs the client already knows plus a few fixed domain terms; the server-STT path sends it with the audio. - A device whose recognizer keeps mishearing the vocabulary can now choose server transcription: `setPreferServerStt` (Settings checkbox, shown only when the deployment offers server STT) makes the client select the server backend ahead of any on-device one. It takes effect on the next launch. Engine `clamp_stt_prompt` (+ tests: trim/empty, bound). Web `sttPref` module (+ tests: pref round-trip, prompt built from slugs, project line omitted when empty). Docs: the STT `prompt` field in ENGINE.md, the toggle in USER_GUIDE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy
thedancingdeveloper
force-pushed
the
feat/server-stt-vocabulary
branch
from
September 11, 2026 02:31
56947bf to
4ef80da
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Third and final fix behind "the app struggles to understand me". The phone picks its on-device recognizer, which mishears every domain word — and it's the transcriber that can't be taught. The server transcriber (Whisper) can be handed the vocabulary as a bias prompt, but the phone never uses it.
Changes
POST /api/assistant/sttaccepts an optionalprompttext field, forwarded to/audio/transcriptionsas the backend's bias prompt (trimmed, bounded to 2 KiB viaclamp_stt_prompt). A backend that ignores it is no worse off; noprompttranscribes exactly as before.sttVocabularyPrompt(slugs)builds the bias from the project slugs the client already knows plus a few fixed domain terms (shell,session,terminal, …); the server-STT path sends it alongside the audio.setPreferServerStt— a Settings checkbox shown only when the deployment offers server STT — makes the client select the server backend ahead of any on-device one, so a phone whose recognizer keeps mangling names can switch to the server + vocabulary. Effect on next launch.Verified
Engine
fmt+clippy -D warnings+assistant_speech::tests (prompt trim/empty, bound). Webtsc+ full vitest (113 files), incl. newsttPreftests (pref round-trip, prompt from slugs, empty-slug handling). Docs updated (ENGINE.md STT field; USER_GUIDE toggle).PR 3 of 3 (after #695 dictation vocabulary, #696 task + agent templates).
🤖 Generated with Claude Code
https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy