fix(assistant): hands-free second turn — release audio before re-arming, settle delay, dead-mic watchdog - #690
Merged
Conversation
…ng, settle, watchdog On the phone the first hands-free turn worked end to end (capture → reply → spoken via Web Audio); the second never captured: the chip said Listening… for 30 s (max_turn) twice, with no partial, no speech-start, no stop. The diagnostics show turn 2's mic opening 31 ms after the app's own playback ended, with the AudioContext still holding the output stream — and the speech plugin cannot report a native start failure at all: with partialResults it resolves start() at once, and its onError only rejects that already-resolved call and stops listening without notifying JS. A recogniser that comes up dead is invisible. - audioPlayback.suspendAudio(): release the output stream when a reply ends naturally or is halted, before any mic re-opens behind it. - voiceTurn: `reopen_delay_ms` (400) settle time before re-arming after a reply (and after an empty turn / a watchdog restart); `mic_watchdog_ms` (8000): a freshly opened mic that reports nothing — no partial, no speech start, no stop — is closed and re-opened, bounded by `max_mic_restarts` (2), then max_turn / idle apply as before. New `recognizerStarted()` event (the plugin's listeningState: started) disarms the watchdog. A mute during the settle wait cancels the re-open; ending during it tears down at once (no mic in flight to protect). Optional `onRecognizerRestart` port. - Assistant.tsx: conversation capture emits rec.start / rec.state / rec.partial diagnostics and feeds started → recognizerStarted; watchdog restarts are logged. The finished TTS request is released on natural end. - Tests: settle wait, mute/end during the wait, watchdog restart + cap, disarm on liveness, suspend-once; existing cases pinned to an immediate re-open. Real-browser voice-turn, hands-free and barge-in tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On the phone the first hands-free turn now works end to end (capture → reply → spoken via Web Audio); the second turn never captured — "Listening…" for 30 s (
max_turn) with no partial, no speech-start, no stop. The diagnostics show turn 2's mic opening 31 ms after the app's own playback ended, with theAudioContextstill holding the output stream. And the speech plugin cannot report a native start failure: withpartialResultsit resolvesstart()at once, and itsonErroronly rejects that already-resolved call and stops listening without notifying JS — a recogniser that comes up dead is invisible.Fix (client-only):
suspendAudio()releases the output stream when a reply ends or is halted, before any mic re-opens behind it.voiceTurn:reopen_delay_ms(400) settle before re-arming;mic_watchdog_ms(8000) restarts a mic that reports nothing, bounded bymax_mic_restarts(2);recognizerStarted()(pluginlisteningState: started) disarms it. Mute during the wait cancels the re-open; end during it tears down immediately.rec.start/rec.state/rec.partialdiagnostics; watchdog restarts are logged.Tests: settle wait, mute/end during the wait, watchdog restart + cap, disarm on liveness, suspend-once; existing cases pinned to immediate re-open. Real-browser voice-turn, hands-free, and barge-in Playwright tests pass. Typecheck clean.
🤖 Generated with Claude Code
https://claude.ai/code/session_01YG9Nsxqc6CREb56Uisnmmy