Skip to content

Voice assistant: hands-free barge-in (WI-174 v2) - #682

Closed
thedancingdeveloper wants to merge 1 commit into
feat/voice-handsfreefrom
feat/voice-handsfree-v2
Closed

thedancingdeveloper wants to merge 1 commit into
feat/voice-handsfreefrom
feat/voice-handsfree-v2

Conversation

@thedancingdeveloper

Copy link
Copy Markdown
Contributor

Stacked on #680 (base feat/voice-handsfree). Adds opt-in barge-in to hands-free mode: with interrupt_response on, the speaker can talk over a playing reply — the device stops speaking and listens the moment they start.

What

  • voiceTurn.tsspeechDetected(): halts the reply and re-opens the mic, gated on interrupt_response and a reply actually playing (a stray onset outside playback is a no-op); the halted reply's own speechFinished is then ignored.
  • voiceVad.ts (new) — a pure OnsetDetector (leaky accumulator over frame RMS: a sustained ~500 ms onset fires once; a click or the echo-cancellation residual does not) + frameRms, unit-tested off frame energies. startBargeInDetection is a thin Web Audio shell (echo-cancelled getUserMediaAnalyserNode, 50 ms frames) that a WASM Silero VAD can replace behind the same seam later — not adopted here to avoid the onnxruntime bundle/model weight.
  • Assistant.tsx — runs the echo-cancelled capture only during speaking (a createEffect on voiceState), torn down the moment playback ends, so the second mic is never held longer than the reply it listens over.

Off by default — the WebView's AEC residual is imperfect and a false trigger cuts a reply short; half-duplex is the safe default. Config: vogt.assistant.voice.{interrupt_response,vad_threshold,vad_onset_ms}.

Tests

  • Full web unit suite green (931), typecheck clean.
  • New: 3 barge-in machine tests + 7 VAD tests; a real-browser Playwright test proves the onset halts the reply (cancel) and re-opens the mic mid-playback.

Notes

🤖 Generated with Claude Code

https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo

Lets the speaker talk over a playing reply: with interrupt_response on, the
device stops speaking and listens the moment you start.

- voiceTurn: `speechDetected()` — halts the reply and re-opens the mic, gated
  on interrupt_response and an actually-playing reply, so a stray onset outside
  playback cannot disturb the loop; the halted reply's own speechFinished is
  then ignored. Unit tests for off (default), on, and outside-playback.
- voiceVad: a pure `OnsetDetector` (leaky accumulator over frame RMS — a
  sustained ~500 ms onset fires once, a click or the AEC residual does not) +
  `frameRms`, unit-tested off frame energies; a thin `startBargeInDetection`
  Web Audio shell (echo-cancelled getUserMedia → AnalyserNode, 50 ms frames)
  that a WASM Silero VAD can replace behind the same seam later.
- Assistant.tsx: runs the echo-cancelled capture only during `speaking` (a
  createEffect on voiceState), torn down the moment playback ends — the second
  mic is never held longer than the reply it listens over. Off by default
  (imperfect WebView AEC can false-trigger); config vogt.assistant.voice.
  {interrupt_response,vad_threshold,vad_onset_ms}.

Tests: full web unit suite green (931); a real-browser Playwright round trip
proves the halt + reopen. Docs: ENGINE.md §6 + USER_GUIDE §2.5.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant