feat(assistant): read dictated turns against a project/session vocabulary - #695
Merged
Merged
Conversation
…lary The phone's on-device recognizer transcribes a spoken turn before the assistant sees it, and it mangles every name it does not know: 'check Kardashian on nude b' for 'check komodo on Node B', 'start a new show here' for 'a new shell here'. The model was then answering 'say exactly: ...', which a person cannot do — they said it right; the recognizer wrote it wrong. The prompt now states that turns are dictated and gives the model a <vocabulary> note each voice turn: the project slugs the core knows and the live session names. It reads a garbled sentence against that vocabulary and the conversation, acts on the clearly-likeliest reading or asks one short question that states it, and never demands exact words. The note is built only for turns carrying a recognized utterance (a typed turn was not misheard, and this spares it a core round trip); the project slugs are fetched once and cached for VOCABULARY_TTL; the names are wrapped as untrusted <vocabulary> data and defanged like every other core-derived string. The stub core gained a per-tool canned-answer hook so the vocabulary path can be tested end to end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On the phone, spoken requests reach the assistant already garbled — the on-device recognizer mangles domain words it has never heard. From a live transcript on dev:
The model was asking the user to say it exactly — impossible, since they did say it right and the recognizer wrote it wrong.
Changes
<vocabulary>note: a second system message each voice turn —projects: …(slugs the core knows) andsessions: …(live roster). Built only for turns carrying a recognized utterance (typed turns weren't misheard, and this spares them aproject.listround trip). Project slugs are fetched once and cached forVOCABULARY_TTL(5 min).<vocabulary>and defanged like every other core-derived string; the prompt names the new delimiter as untrusted, so the two source-scanning prompt tests still pass..answer(tool, text)for cannedtools/callresults, used by the new end-to-end vocabulary tests.Verified
cargo fmt,clippy -D warnings, and the fullassistant::+vogt_tools::suites (74 tests) pass. New tests: the note is offered every voice turn and cached across turns; a dead core costs no vocabulary and no error;project_slugs_inparses both list shapes and is bounded.This is PR 1 of 3 from debugging "the app struggles to understand me". PR 2 adds a task-bearing agent session (so "spawn a session and check X" runs Claude with the task); PR 3 adds server-STT vocabulary + a client preference.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy