Skip to content

Voice assistant: tap-to-talk + hands-free conversation mode (WI-173, WI-174 v1) - #680

Merged
thedancingdeveloper merged 5 commits into
mainfrom
feat/voice-handsfree
Sep 10, 2026
Merged

Voice assistant: tap-to-talk + hands-free conversation mode (WI-173, WI-174 v1)#680
thedancingdeveloper merged 5 commits into
mainfrom
feat/voice-handsfree

Conversation

@thedancingdeveloper

@thedancingdeveloper thedancingdeveloper commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Delivers the phone voice uplift: the mic-tap race fix that makes tap-to-talk work (WI-173), the hands-free conversation loop (WI-174 v1), and the generic slice of the prod-TTS honesty fix (WI-172). Client-only, backend-agnostic — no engine or API change.

WI-173 — tap-to-talk (race fix + JS silence)

startListening's native path is async; a pointerup during startup ran removeAllListeners between the startup's addListener and start() and stopped a recognizer that wasn't up yet — the take was orphaned and nothing sent. Now a release during startup is deferred (nativeStarting) and the take runs on, ended by silence. JS owns silence detection (the dictation-mode end-of-speech is late/untunable), gated off while the button is held, with a grace after listeningState: stopped so the final result is what's sent.

WI-174 v1 — hands-free conversation

  • web/src/voiceTurn.ts — a pure, backend-agnostic state machine (idle → arming → listening → endpointing → sending → speaking → listening, plus paused_for_approval and an orthogonal muted). Every effect is a port; time is an injected clock. 14 unit tests cover the interleavings a device is worst at showing.
  • Assistant.tsx — a Conversation head toggle (disabled-with-reason when there's no live recognizer or no TTS; turning it on turns Spoken replies on), dedicated native/Web-Speech capture that feeds the machine (the tap path is untouched), applyReply split so both send and resolve route through the loop, half-duplex speak-then-reopen, a live status chip (role=status/aria-live), and mic-as-mute (tap or M).
  • v1 is half-duplex (mic closed while a reply plays; barge-in is v2). Server-STT is v1-excluded (no partials/endpoint of its own). Turn timing is vogt.assistant.voice.* localStorage in the OpenAI-Realtime vocabulary, so a Realtime-shaped backend adopts it unrenamed.

WI-172 — generic slice

Corrected the Assistant.tsx header comment + ENGINE.md: the Android WebView has no speechSynthesis, so the APK speaks through POST /api/assistant/tts, honest-empty when unconfigured. The advertise-no-backend lie is not in this repo (the engine defaults its speech base-URL lists empty on purpose); it's reintroduced only by a private estate overlay, which stays an operator action.

Tests

  • Full web unit suite green (920), typecheck clean.
  • New: 3 WI-173 tap tests, 14 voiceTurn machine tests, 3 wiring tests incl. a full hands-free round trip, and a real-browser Playwright hands-free round trip (gui.spec.ts).

Notes for the reviewer

  • The phone Playwright snapshot assistant-phone-structured needs regenerating on a stable box — the new head button legitimately changes it, and the dev box has documented render drift (height flaps run-to-run on main too), so no local regen was committed. gui.spec is not CI-gated.
  • Design note lives in docs/local/VOICE_HANDSFREE_DESIGN.md (gitignored working note); user/engine docs are in USER_GUIDE §2.5 and ENGINE.md §6.
  • Follow-ups: v2 (client VAD + barge-in) and WI-172's prod-TTS ops step (a DeployStack, out of this repo).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo

@thedancingdeveloper thedancingdeveloper mentioned this pull request Sep 10, 2026
Closed
thedancingdeveloper and others added 5 commits September 10, 2026 02:35
@thedancingdeveloper @claude
fix(assistant): race-free mic take + JS silence timer (tap-to-talk)
4155dd2 
WI-173: a tap on the mic raced the recognizer start, so a take never
auto-sent on silence. `startListening`'s native path is async — it awaits
the plugin import, permission and `start()` — and a `pointerup` during
those awaits ran `stopListening`, which called `removeAllListeners()`
between the startup's `addListener` and `start()` and stopped a recognizer
that was not up yet: the take was orphaned and nothing was sent.

- Defer a release that lands during native startup (`nativeStarting`)
  instead of tearing the recognizer down; the take runs on, ended by
  silence — tap-to-talk, as the design has always promised.
- Own silence detection in JS (the Android dictation-mode end-of-speech is
  late and untunable): a timer rearmed on every partial, gated off while
  the button is held so a mid-sentence pause never ends a hold.
- A grace window after `listeningState: stopped` so the final result
  (delivered as one more `partialResults` after `stopped`) is what gets
  sent, not the last interim guess.
- Durations read from `vogt.assistant.voice.{silence_duration_ms,
  final_result_grace_ms}` localStorage with generic defaults (1000/300),
  the same namespace the hands-free mode (WI-174) uses — no vendor or
  estate specifics.

WI-172 (repo slice): correct the file header comment — the Android WebView
has no `speechSynthesis`, so the APK speaks through `POST /api/assistant/tts`,
and only when a TTS backend is configured (the engine defaults its speech
base-URL lists to empty so `assistant_tts_enabled` reads false rather than
advertising a mouth that cannot speak).

Tests: tap-releases-before-start still sends on silence; a release mid-start
does not tear the recognizer down; the final result after `stopped` is sent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
@thedancingdeveloper @claude
feat(assistant): hands-free conversation state machine (WI-174 v1 core)
4c078b4 
The pure, backend-agnostic loop behind hands-free mode: speak → quiet →
send → speak the reply → re-open the mic, no touch between turns. Every
effect is a port and time is an injected clock, so the interleavings a
device is worst at showing — a release mid-arming, a final result after the
stop, an idle session, an approval that must never be answered by voice —
are unit-tested off a list of port calls rather than a microphone.

- States idle → arming → listening → endpointing → sending → speaking →
  listening, plus paused_for_approval (mic shut until the on-screen
  approve/deny — a misheard "yes" authorises nothing) and an orthogonal
  muted flag that keeps the session alive while capture stops.
- v1 half-duplex: mic closed while a reply plays, re-opened on speechFinished
  (no echo-safe capture on the WebView path, so barge-in/interrupt_response
  is designed-for but off).
- Turn detection is client-owned on every backend; config in the OpenAI
  Realtime vocabulary (silence_duration_ms, idle_timeout_ms, max_turn_ms,
  max_empty_turns, interrupt_response) read from vogt.assistant.voice.*
  localStorage with generic defaults, so a Realtime-shaped backend adopts
  them unrenamed. No vendor or estate specifics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
@thedancingdeveloper @claude
feat(assistant): wire hands-free conversation mode into the surface (…
76a006b 
…WI-174 v1)

Drives the pure loop from voiceTurn.ts on the real recognizers and synth:

- A Conversation head toggle, shown always so it is discoverable and disabled
  with its reason when the device has no live recognizer or no way to speak;
  turning it on turns spoken replies on (the loop needs a mouth) and primes
  the synth inside the user gesture.
- Dedicated conversation capture (native plugin + Web Speech) that forwards
  partials and the recognizer's own stop to the machine, kept separate from
  the push/tap-to-talk path so WI-173 is untouched. Server STT is v1-excluded
  (no partials/endpoint of its own) and the toggle says so.
- applyReply split into recordReply + speak: in a conversation both send and
  resolve route their reply through the machine, which speaks it and re-opens
  the mic when playback ends (speak/playServerTts gained an onDone callback);
  a failed turn re-opens rather than stranding the loop in "sending".
- A live status chip (role=status, aria-live): Listening… / Sending / Speaking
  / Paused for approval / Muted / Ended. In a conversation the mic renders as a
  mute toggle (tap, or M on desktop); leaving the surface ends the session.
- The existing foreground-service effect already holds while ttsOn, which a
  conversation implies — so a backgrounded conversation survives screen-off.

Tests: the control is disabled-with-reason when nothing can speak; turning it
on opens the recognizer and shows a live status; and a whole turn runs
hands-free — sends on silence, speaks the reply, and re-opens the mic with no
touch (the half a device demo cannot show).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
@thedancingdeveloper @claude
test(assistant): browser hands-free round trip + guard the induced-st…
86cccdb 
…op double send

- voiceTurn: fence closeMic so a synchronous onend → recognizerStopped from our
  own stop (Web Speech's stop() fires onend) cannot re-enter the turn end and
  send it twice; enter `sending` before closing so an induced stop reads as
  sending. Unit test locks it.
- gui.spec: a real-browser hands-free round trip on a controllable fake
  recognizer + synth — turn on, speak, go quiet, the turn sends on silence, the
  reply is spoken, and the mic re-opens for the next turn with no touch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
@thedancingdeveloper @claude
docs(assistant): document tap-to-talk and hands-free conversation
c389532 
- USER_GUIDE §2.5: a quick tap opens the mic and sends on silence; the
  hands-free conversation control — speak, go quiet, the turn sends, the reply
  is spoken, the mic re-opens with no touch; the status line, mute (tap or M),
  approval-still-on-screen, the end conditions, and the vogt.assistant.voice.*
  tuning keys.
- ENGINE.md §6 Voice: tap-to-talk (JS-owned silence + start-race handling);
  correct the TTS note (the Android WebView has no speechSynthesis, so the APK
  uses POST /api/assistant/tts, honest-empty when unconfigured); a hands-free
  bullet with the state machine, half-duplex v1, and the OpenAI-Realtime-named
  config keys.

The full design note lives in docs/local/VOICE_HANDSFREE_DESIGN.md (gitignored,
local working note).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BojRRh8yXBBqwemk6WkqVo
@thedancingdeveloper
thedancingdeveloper merged commit 6e94f8d into main Sep 10, 2026
15 checks passed
@thedancingdeveloper
thedancingdeveloper deleted the feat/voice-handsfree branch September 10, 2026 02:52
@thedancingdeveloper thedancingdeveloper mentioned this pull request Sep 10, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@thedancingdeveloper