diff --git a/docs/ENGINE.md b/docs/ENGINE.md index 7a39614a..382e8de3 100644 --- a/docs/ENGINE.md +++ b/docs/ENGINE.md @@ -1782,9 +1782,36 @@ become instructions. space or enter, since a control only pointers can work is one some people cannot use. `RECORD_AUDIO` is declared in the manifest; the plugin prompts at first use. -- **TTS** — Web Speech `speechSynthesis`, sentence-chunked, toggle persisted - in localStorage. The synth is primed on the toggle gesture because the - Android WebView requires a user gesture before the first utterance. + A quick tap (as opposed to a hold) opens a **tap-to-talk** take that ends + itself: JS owns the silence detection (the dictation-mode end-of-speech is + late and untunable), and a grace after the recognizer's own stop includes the + final result rather than the last interim guess. A release that lands before + the recognizer has finished starting is honoured as a tap, not a stop that + would orphan a recognizer that is not up yet. +- **TTS** — Web Speech `speechSynthesis` when the browser has it (the desktop + PWA), sentence-chunked, toggle persisted in localStorage. The synth is primed + on the toggle gesture because the Android WebView requires a user gesture + before the first utterance. The **Android WebView has no `speechSynthesis`**, + so the APK speaks through the server route `POST /api/assistant/tts` — and + only when a TTS backend is configured. The engine defaults its speech + base-URL lists to empty on purpose, so `assistant_tts_enabled` reads false + rather than advertising a mouth that cannot speak (`config.rs`). +- **Hands-free conversation** — a client-only loop (`web/src/voiceTurn.ts`, a + pure state machine) that keeps listening between turns: speak → silence → + send → speak the reply → re-open the mic, no touch between turns. States + `idle → arming → listening → endpointing → sending → speaking → listening`, + plus `paused_for_approval` (a pending write is announced, the mic closed + until the on-screen approve/deny — voice still never approves) and a `muted` + flag that keeps the session alive. v1 is half-duplex (mic closed while a + reply plays; barge-in is v2). Turn detection is client-owned on every + backend, tuned by localStorage in the OpenAI Realtime vocabulary so a + Realtime-shaped backend adopts it unrenamed: + `vogt.assistant.voice.silence_duration_ms` (1000), + `final_result_grace_ms` (300), `max_turn_ms` (30000), + `idle_timeout_ms` (60000), `max_empty_turns` (3), + `interrupt_response` (false in v1). Requires an event-driven recognizer + (native plugin or Web Speech) and a TTS path; the server-STT path is excluded + in v1 and the control is disabled with its reason. There is no setting that lets the assistant type without asking. The convenience it would buy a trusted single-user setup is outweighed by what it diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index a734d727..3d2a6682 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -457,7 +457,23 @@ and **Approve on screen** can be pressed. and an approval is a tap. Push-to-talk is *held*, not toggled — press to open the microphone, release to send — because a take auto-sends, and a toggle left on in a room with other people does not merely listen. The button is holdable -from the keyboard as well as by pointer. +from the keyboard as well as by pointer. A quick tap (rather than a hold) also +works: it opens the microphone and sends once you go quiet. + +**Hands-free conversation.** The head control beside Spoken replies starts a +hands-free conversation: speak, go quiet, and the turn sends on its own; the +reply is spoken and the microphone re-opens for the next turn, with no touch +between them. Turning it on turns Spoken replies on — the mode needs a voice to +answer with — and it is offered only where the device can both listen and speak +(the app, or a browser with voice input and speech synthesis), disabled with the +reason otherwise. A status line reads Listening… / Sending / Speaking / Paused +for approval / Muted / Ended; in a conversation the microphone button mutes and +unmutes with a tap (or **M** on the desktop), and a pending write still waits for +an on-screen approval — the announcement says so and never offers a spoken yes. +The conversation ends when you turn it off, leave the surface, mute-and-forget it +past the idle timeout, or hear nothing for a few turns. Turn timing is tunable +per device through `vogt.assistant.voice.*` browser settings (silence window, +idle timeout); the defaults suit a phone. An approved write is audited to **your** actor, using the core token paired with the token that pressed approve. There is no shared "assistant" actor to fall diff --git a/web/src/Assistant.tsx b/web/src/Assistant.tsx index 36bc036c..61d8c1b5 100644 --- a/web/src/Assistant.tsx +++ b/web/src/Assistant.tsx @@ -1,9 +1,17 @@ // Conversational assistant tab: transcript + composer, optional voice. // -// Voice support is progressive: TTS uses Web Speech `speechSynthesis` -// (available in browsers and the Android WebView); STT uses the -// @capacitor-community/speech-recognition native plugin and therefore only -// appears inside the Capacitor APK. Everything degrades to typed input. +// Voice support is progressive and chosen by capability, not by platform name: +// - TTS uses the browser's Web Speech `speechSynthesis` when it exists (the +// desktop PWA). The Android WebView has *no* `speechSynthesis`, so the APK +// speaks through the server route `POST /api/assistant/tts` instead — and +// only when a TTS backend is configured. With none, the reply is shown, not +// spoken: the engine defaults its speech base-URL lists to empty on purpose, +// so `assistant_tts_enabled` reads false rather than advertising a mouth +// that cannot speak (see `engine/server/src/config.rs`). +// - STT prefers the @capacitor-community/speech-recognition native plugin +// (APK only), then the browser's Web Speech recognizer, then the server +// pipeline (MediaRecorder → `POST /api/assistant/stt`) for a client with +// neither. Everything degrades to typed input. import { createEffect, createSignal, @@ -41,9 +49,33 @@ import { startVoiceService, stopVoiceService, } from "./voiceService"; +import { + readVoiceConfig, + VoiceConversation, + voiceStatusLabel, + type VoicePorts, + type VoiceState, +} from "./voiceTurn"; const TTS_PREF_KEY = "vogt.assistant.tts"; +/** Voice-turn tuning lives under one localStorage namespace, so any deployment + * can tune a take's silence window without a rebuild — and the hands-free + * conversation mode (WI-174) reads the same keys. Generic defaults, no vendor + * or estate specifics; a missing or malformed value falls back rather than + * throwing (private windows and locked-down browsers make reads fail). */ +const VOICE_CFG_PREFIX = "vogt.assistant.voice."; +function voiceMs(key: string, fallback: number): number { + try { + const raw = localStorage.getItem(VOICE_CFG_PREFIX + key); + if (raw === null) return fallback; + const n = Number(raw); + return Number.isFinite(n) && n >= 0 ? n : fallback; + } catch { + return fallback; + } +} + /** How long the server holds a pending action before it expires. The * card counts down against this so an approval you can no longer make stops * inviting you to make it. Kept in step with `PENDING_ACTION_TTL` in the @@ -188,8 +220,11 @@ interface AssistantProps { confirmAction?: (title: string, body?: string) => Promise; } -function speakSentences(text: string) { - if (!("speechSynthesis" in window) || !text.trim()) return; +function speakSentences(text: string, onDone?: () => void) { + if (!("speechSynthesis" in window) || !text.trim()) { + onDone?.(); + return; + } // Sentence-level chunks keep the synth responsive and interruptible. // // Split only where a terminator is followed by space or line end, so an @@ -197,12 +232,21 @@ function speakSentences(text: string) { // into "work." and "transition on WI-7", which a screen reader renders as // two sentences and a speaker reads with a pause in the middle of the one // word that says what is about to happen. - const sentences = text.split(/(?<=[.!?])\s+|\n+/).filter((part) => part.trim()); - for (const sentence of sentences) { - const trimmed = sentence.trim(); - if (!trimmed) continue; - window.speechSynthesis.speak(new SpeechSynthesisUtterance(trimmed)); + const sentences = text + .split(/(?<=[.!?])\s+|\n+/) + .map((part) => part.trim()) + .filter(Boolean); + if (!sentences.length) { + onDone?.(); + return; } + sentences.forEach((sentence, index) => { + const utterance = new SpeechSynthesisUtterance(sentence); + // The hands-free loop re-opens the mic when the reply finishes, so the + // last chunk carries the "done" — half-duplex has no capture until then. + if (index === sentences.length - 1 && onDone) utterance.onend = () => onDone(); + window.speechSynthesis.speak(utterance); + }); } function stopSpeaking() { @@ -268,6 +312,12 @@ export default function Assistant(props: AssistantProps) { const [reasonBusy, setReasonBusy] = createSignal(false); const [draft, setDraft] = createSignal(restored.text); const [ttsOn, setTtsOn] = createSignal(localStorage.getItem(TTS_PREF_KEY) === "1"); + // Hands-free conversation mode (WI-174). Session-only — it opens the mic, so + // it never auto-resumes on mount. `voiceState`/`voiceMuted` mirror the pure + // machine for the status chip. + const [conversationOn, setConversationOn] = createSignal(false); + const [voiceState, setVoiceState] = createSignal("idle"); + const [voiceMuted, setVoiceMuted] = createSignal(false); const [listening, setListening] = createSignal(false); const [sttAvailable, setSttAvailable] = createSignal(false); // Whether the server-side speech pipeline is configured. Read from @@ -330,6 +380,9 @@ export default function Assistant(props: AssistantProps) { let speechController: AbortController | null = null; let transcriptionController: AbortController | null = null; const [speechStatus, setSpeechStatus] = createSignal(""); + // The hands-free loop, created lazily when Conversation is turned on. Declared + // here because `applyReply` and `send` feed it their outcomes. + let conversation: VoiceConversation | null = null; const haltSpeech = () => { speechController?.abort(); @@ -351,17 +404,26 @@ export default function Assistant(props: AssistantProps) { * none. A server route that 404s (unconfigured) degrades this half silently * — a spoken reply that cannot be spoken is still shown in the transcript. */ - const speak = (text: string) => { + // `onDone`, when given, fires once the reply has finished playing (or at once + // when there is nothing to play, or no mouth to play it). The hands-free loop + // passes it to re-open the mic; a plain spoken reply passes nothing. + const speak = (text: string, onDone?: () => void) => { if ("speechSynthesis" in window) { - speakSentences(text); + speakSentences(text, onDone); return; } - if (serverTtsEnabled()) void playServerTts(text); - else setSpeechStatus("Spoken replies are unavailable. Read the reply below."); + if (serverTtsEnabled()) void playServerTts(text, onDone); + else { + setSpeechStatus("Spoken replies are unavailable. Read the reply below."); + onDone?.(); + } }; - const playServerTts = async (text: string) => { - if (!text.trim()) return; + const playServerTts = async (text: string, onDone?: () => void) => { + if (!text.trim()) { + onDone?.(); + return; + } haltSpeech(); const controller = new AbortController(); speechController = controller; @@ -373,12 +435,16 @@ export default function Assistant(props: AssistantProps) { const audio = new Audio(url); const release = () => URL.revokeObjectURL(url); audio.addEventListener("ended", release, { once: true }); + // The reply is over: re-open the mic. Fired on the natural end only, so a + // reply cut short by the next turn (abort) does not race a new take open. + if (onDone) audio.addEventListener("ended", onDone, { once: true }); controller.signal.addEventListener("abort", release, { once: true }); currentAudio = audio; await audio.play(); } catch (e) { if (controller.signal.aborted) return; setSpeechStatus("Spoken replies are unavailable. Read the reply below."); + onDone?.(); if (e instanceof ApiError && e.status === 404) { setServerTtsEnabled(false); } else { @@ -562,6 +628,7 @@ export default function Assistant(props: AssistantProps) { profile: profile(), }); haltSpeech(); + clearSilence(); transcriptionController?.abort(); inFlight()?.abort(); // Leaving the surface ends the conversation, so release the held service @@ -569,12 +636,19 @@ export default function Assistant(props: AssistantProps) { stopVoiceService(); voiceEndedCleanup?.(); pushSpeakerCleanup?.(); + // Leaving ends a hands-free conversation: stop the machine and close its + // recognizer, so nothing keeps listening after the surface is gone. + conversation?.end("user"); + conversation = null; + void closeConversationMic(); // Abandoned, not sent: leaving the surface mid-sentence must not put // half an utterance into the conversation on the way out. if (listening()) void abandonTake(); }); - const applyReply = (reply: AssistantReply) => { + /** Put a reply into the transcript and set its pending action — the visible + * record, with no speech. Speaking is decided by the caller. */ + const recordReply = (reply: AssistantReply) => { if (reply.reply !== null && reply.reply !== undefined) { setTranscript((cur) => [ ...cur, @@ -587,12 +661,27 @@ export default function Assistant(props: AssistantProps) { actions: reply.actions, }, ]); - if (ttsOn()) speak(reply.reply); } setPendingAction(reply.pending_action ?? null); - if (reply.pending_action && ttsOn()) { - speak(announce(reply.pending_action)); + }; + + const applyReply = (reply: AssistantReply) => { + recordReply(reply); + // In a hands-free conversation the machine owns speech: it speaks the reply + // (or the pending-action announcement) and re-opens the mic when playback + // ends. Feeding it here routes both `send` and `resolve` through the loop. + if (conversation?.isActive()) { + const spoken = reply.pending_action + ? announce(reply.pending_action) + : (reply.reply ?? ""); + conversation.replied({ + text: spoken.trim() ? spoken : null, + hasPendingAction: Boolean(reply.pending_action), + }); + return; } + if (reply.reply !== null && reply.reply !== undefined && ttsOn()) speak(reply.reply); + if (reply.pending_action && ttsOn()) speak(announce(reply.pending_action)); }; const send = async (text: string, utterance?: string) => { @@ -630,6 +719,10 @@ export default function Assistant(props: AssistantProps) { ); props.onError(`assistant: ${String(e)}`); } + // A hands-free turn that did not land must not leave the loop stuck in + // "sending": re-open the mic (the mode stays; the failed bubble carries + // the Retry). A no-op outside a conversation. + conversation?.sendFailed(); } finally { setBusy(false); setInFlight(null); @@ -735,6 +828,47 @@ export default function Assistant(props: AssistantProps) { let serverChunks: Blob[] = []; let abandonServer = false; + // Tap-to-talk, and the startup race it fixes (WI-173). A press opens the + // take; a release that lands *after* the native recognizer is up ends it + // (push-to-talk), and a release that lands *before* `start()` resolved is not + // a stop at all — the recognizer is not listening yet, so tearing it down + // there is what orphaned it and stripped the very listeners that same startup + // had just added. Such a release is deferred and the take runs on, ended by + // silence instead: the tap-to-talk the design has always promised. + let held = false; // the mic button is physically down right now + let nativeStarting = false; // committed to a native take, `start()` not yet resolved + // The JS-owned silence detector. The Android recognizer's own end-of-speech + // is late and untunable in dictation mode, so a tap's turn ends here: a timer + // rearmed on every partial result, firing once the transcript stops changing. + let silenceTimer: ReturnType | null = null; + const clearSilence = () => { + if (silenceTimer !== null) { + clearTimeout(silenceTimer); + silenceTimer = null; + } + }; + // Rearm the silence timer for a tap. Skipped while the button is held: a hold + // *is* the take's length, so a mid-sentence pause must not end it. + const armSilence = () => { + if (held) return; + clearSilence(); + silenceTimer = setTimeout(() => { + silenceTimer = null; + void stopListening(); + }, voiceMs("silence_duration_ms", 1000)); + }; + // The recognizer stopped on its own (usually the speaker went quiet). End the + // take, but after a short grace, because the plugin fires `listeningState: + // stopped` *before* the final, best result — delivered as one last + // `partialResults`. Ending immediately would send the last interim guess. + const endTakeAfterFinal = () => { + clearSilence(); + silenceTimer = setTimeout(() => { + silenceTimer = null; + void stopListening(); + }, voiceMs("final_result_grace_ms", 300)); + }; + const closeRecognizer = async () => { setListening(false); if (sttBackend === "web") { @@ -774,7 +908,12 @@ export default function Assistant(props: AssistantProps) { /** End the take and send what was said. */ const stopListening = async () => { if (!takeOpen) return; + // A release during native startup is not a stop: the recognizer is not up + // yet, so tearing it down here is what orphaned it. Leave the take open and + // let silence (or the recognizer's own stop) end it — tap-to-talk. + if (nativeStarting) return; takeOpen = false; + clearSilence(); await closeRecognizer(); // The server pipeline sends from the recorder's `onstop` once the audio has // been transcribed, not from the draft — there is nothing in the composer @@ -802,6 +941,7 @@ export default function Assistant(props: AssistantProps) { /** End the take and send nothing — for leaving the surface mid-sentence. */ const abandonTake = async () => { takeOpen = false; + clearSilence(); // Tell the server recorder's `onstop` to drop the audio rather than post it. abandonServer = true; await closeRecognizer(); @@ -943,29 +1083,242 @@ export default function Assistant(props: AssistantProps) { return; } haltSpeech(); + // Commit to the take, then wire and start with no teardown in between: a + // release that lands mid-startup is deferred by `stopListening` (which + // sees `nativeStarting`) rather than run against a half-built recognizer, + // and `removeAllListeners` only ever runs *before* our listeners — so it + // cannot strip the pair this same startup just added. takeOpen = true; + nativeStarting = true; setListening(true); await SpeechRecognition.removeAllListeners(); await SpeechRecognition.addListener("partialResults", (data) => { const best = data.matches?.[0]; - if (best) setDraft(best); + if (best) { + setDraft(best); + // Each new partial resets the silence clock: a tap ends its turn a + // beat after the transcript stops changing. + armSilence(); + } }); await SpeechRecognition.addListener("listeningState", (data) => { - // The other end of the take: the recognizer gave up before the - // button was released, usually because the speaker went quiet. Same - // path, so what was said is sent once and only once. - if (data.status === "stopped") void stopListening(); + // The other end of the take: the recognizer gave up before the button + // was released, usually because the speaker went quiet. Ended through + // the one `stopListening`, after a grace for the final result, so what + // was said is sent once and only once. + if (data.status === "stopped") endTakeAfterFinal(); }); await SpeechRecognition.start({ partialResults: true, popup: false, }); } catch (e) { + nativeStarting = false; + takeOpen = false; setListening(false); props.onError(`speech recognition: ${String(e)}`); + return; } + // Startup is done: a release that arrives from here on ends the take at once + // (push-to-talk), and any release that already landed was deferred and now + // leaves the take running until silence — the tap. + nativeStarting = false; }; + // -- hands-free conversation mode (WI-174) -------------------------------- + // + // The pure loop in `voiceTurn.ts` decides *when*; these functions are its + // ports — capture that forwards the recognizer's events to the machine + // (rather than the tap path's send-on-stop), and playback that tells the + // machine when a reply has finished so it can re-open the mic. Kept separate + // from the tap path above so push/tap-to-talk is untouched. + let convWebRecognition: WebSpeechRecognition | null = null; + + const ttsCapable = () => "speechSynthesis" in window || serverTtsEnabled(); + /** Whether hands-free can run here: an event-driven recognizer (v1 excludes + * the server-STT path, which has no partials/endpoint of its own) and a way + * to speak. */ + const conversationSupported = () => + sttAvailable() && + (sttBackend === "native" || sttBackend === "web") && + ttsCapable(); + const conversationDisabledReason = () => { + if (!sttAvailable() || (sttBackend !== "native" && sttBackend !== "web")) { + return "Hands-free needs a live voice recognizer (the app, or a browser with voice input)."; + } + if (!ttsCapable()) return "Hands-free needs spoken replies, which aren't available here."; + return ""; + }; + + const openConversationMicNative = async () => { + try { + const { SpeechRecognition } = await import( + "@capacitor-community/speech-recognition" + ); + const perm = await SpeechRecognition.requestPermissions(); + if (perm.speechRecognition !== "granted") { + conversation?.end("no_backend"); + return; + } + await SpeechRecognition.removeAllListeners(); + await SpeechRecognition.addListener("partialResults", (data) => { + const best = data.matches?.[0]; + if (best) conversation?.partial(best); + }); + await SpeechRecognition.addListener("listeningState", (data) => { + if (data.status === "stopped") conversation?.recognizerStopped(); + }); + await SpeechRecognition.start({ partialResults: true, popup: false }); + conversation?.micReady(); + } catch (e) { + props.onError(`speech recognition: ${String(e)}`); + conversation?.end("no_backend"); + } + }; + + const openConversationMicWeb = () => { + const Ctor = webSpeechCtor(); + if (!Ctor) { + conversation?.end("no_backend"); + return; + } + try { + const recognition = new Ctor(); + recognition.continuous = true; + recognition.interimResults = true; + recognition.lang = navigator.language || "en-US"; + recognition.onresult = (event) => { + let heard = ""; + for (let i = 0; i < event.results.length; i += 1) { + heard += event.results[i]?.[0]?.transcript ?? ""; + } + if (heard.trim()) conversation?.partial(heard.trim()); + }; + recognition.onend = () => conversation?.recognizerStopped(); + recognition.onerror = (event) => { + // `no-speech`/`aborted` are ordinary turn ends, handled by the machine's + // own silence timer; anything else means the recognizer is gone. + if (event.error !== "no-speech" && event.error !== "aborted") { + conversation?.end("no_backend"); + } + }; + convWebRecognition = recognition; + recognition.start(); + conversation?.micReady(); + } catch (e) { + props.onError(`speech recognition: ${String(e)}`); + conversation?.end("no_backend"); + } + }; + + const closeConversationMic = async () => { + if (convWebRecognition) { + const recognition = convWebRecognition; + convWebRecognition = null; + try { + recognition.stop(); + } catch { + /* already stopped */ + } + } + if (sttBackend === "native") { + try { + const { SpeechRecognition } = await import( + "@capacitor-community/speech-recognition" + ); + await SpeechRecognition.stop(); + await SpeechRecognition.removeAllListeners(); + } catch { + /* plugin gone mid-flight — nothing to stop */ + } + } + }; + + /** Tear the conversation down. `userInitiated` when the toggle/leave did it, + * as opposed to the machine ending itself (idle, empty turns, backend gone).*/ + const endConversation = (userInitiated: boolean, reason?: string) => { + const machine = conversation; + conversation = null; + setConversationOn(false); + setVoiceState("idle"); + setVoiceMuted(false); + void closeConversationMic(); + haltSpeech(); + if (userInitiated) machine?.end("user"); + setSpeechStatus( + reason === "idle" + ? "Conversation ended — no speech for a while." + : reason === "empty_turns" + ? "Conversation ended — heard nothing." + : reason === "no_backend" + ? "Conversation ended — voice is unavailable here." + : "", + ); + }; + + const conversationPorts: VoicePorts = { + openMic: () => { + haltSpeech(); + if (sttBackend === "web") openConversationMicWeb(); + else void openConversationMicNative(); + }, + closeMic: () => void closeConversationMic(), + sendTurn: (text) => { + // The same repair pass the tap path uses: a work-item ref the recognizer + // mangled is the subject of the sentence. + const { text: repairedText, repairs } = repairUtterance(text, slugs()); + setRepaired(repairs.length ? describeRepairs(repairs) : ""); + void send(repairedText, text); + }, + speak: (text) => speak(text, () => conversation?.speechFinished()), + stopSpeaking: () => haltSpeech(), + onChange: (state, muted) => { + setVoiceState(state); + setVoiceMuted(muted); + }, + onEnded: (reason) => endConversation(false, reason), + }; + + const toggleConversation = () => { + if (conversationOn()) { + endConversation(true); + return; + } + if (!conversationSupported()) return; + setSpeechStatus(""); + // Hands-free needs a mouth: turning it on turns spoken replies on. + if (!ttsOn()) { + setTtsOn(true); + localStorage.setItem(TTS_PREF_KEY, "1"); + } + // Prime the synth inside the user gesture — the Android WebView requires it. + window.speechSynthesis?.speak(new SpeechSynthesisUtterance("")); + conversation = new VoiceConversation(conversationPorts, readVoiceConfig()); + setConversationOn(true); + conversation.begin(); + }; + + // Desktop shortcut: `M` mutes/unmutes an active conversation, unless the + // caret is in a text field (where `m` is just a letter). + createEffect(() => { + if (!conversationOn()) return; + const onKey = (e: KeyboardEvent) => { + if ((e.key === "m" || e.key === "M") && !e.repeat) { + const target = e.target as HTMLElement | null; + const typing = + target && + (target.tagName === "TEXTAREA" || + target.tagName === "INPUT" || + target.isContentEditable); + if (typing) return; + e.preventDefault(); + conversation?.toggleMute(); + } + }; + window.addEventListener("keydown", onKey); + onCleanup(() => window.removeEventListener("keydown", onKey)); + }); + return (
@@ -1027,6 +1380,40 @@ export default function Assistant(props: AssistantProps) { + {/* + Hands-free conversation: speak, go quiet, the turn sends, the reply is + spoken, the mic re-opens — no touch between turns. Shown always, so the + feature is discoverable, but disabled with its reason when the device + has no live recognizer or no way to speak. + */} + + + } + > + {/* Conversation is on: the mic is a mute toggle, live-labelled. */} + {/* Stop while a turn is in flight, Send otherwise. Stop aborts the diff --git a/web/src/__tests__/assistant.test.tsx b/web/src/__tests__/assistant.test.tsx index 4f5a1752..3256ffc6 100644 --- a/web/src/__tests__/assistant.test.tsx +++ b/web/src/__tests__/assistant.test.tsx @@ -68,6 +68,10 @@ async function mountAssistant(engine: Record = {}) { describe("the assistant's microphone", () => { beforeEach(() => { for (const fn of Object.values(recognition)) fn.mockClear(); + // A clean silence window for every take, so one test's tuning cannot leak + // into the next. The values are read fresh at take time. + localStorage.removeItem("vogt.assistant.voice.silence_duration_ms"); + localStorage.removeItem("vogt.assistant.voice.final_result_grace_ms"); }); it("opens only while the button is held", async () => { @@ -175,6 +179,68 @@ describe("the assistant's microphone", () => { assistantRequests().filter((url) => url.includes("message")), ).toHaveLength(0); }); + + it("keeps the take open when the release lands before the recognizer is up (tap-to-talk), and sends on silence", async () => { + // The WI-173 race, from the outside: pressing and releasing in the same + // tick leaves the release ahead of the plugin's `start()`. The old code + // treated it as a stop and tore down a recognizer that was not up yet, so + // the take was orphaned and nothing was ever sent. It must instead run on, + // ended by silence — the tap the design has always promised. + localStorage.setItem("vogt.assistant.voice.silence_duration_ms", "0"); + const { mic } = await mountAssistant(); + fireEvent.pointerDown(mic, { pointerId: 1 }); + fireEvent.pointerUp(mic, { pointerId: 1 }); + await settle(); + // The recognizer came up despite the early release, and is still listening. + expect(recognition.start).toHaveBeenCalledTimes(1); + expect(mic.dataset.listening).toBe("yes"); + // The words arrive, then quiet — the silence timer (0ms here) sends once. + listenerFor("partialResults")?.({ matches: ["open the backlog"] }); + await settle(); + expect( + assistantRequests().filter((url) => url.includes("message")), + ).toHaveLength(1); + }); + + it("does not tear down a recognizer that is still starting when the button is released", async () => { + // The other interleaving: the release lands after the take is committed but + // before `start()` resolved. Hold `start()` open to sit in exactly that + // window. Removing the listeners or stopping here is what stripped the pair + // the same startup had just added. + let resolveStart: () => void = () => {}; + recognition.start.mockImplementationOnce( + () => new Promise((resolve) => (resolveStart = () => resolve())), + ); + const { mic } = await mountAssistant(); + fireEvent.pointerDown(mic, { pointerId: 1 }); + await settle(); // suspended at the held start() + fireEvent.pointerUp(mic, { pointerId: 1 }); + await settle(); + // The release was deferred: only the pre-wire clear ran, and nothing was + // stopped. The recognizer is still listening. + expect(recognition.removeAllListeners).toHaveBeenCalledTimes(1); + expect(recognition.stop).not.toHaveBeenCalled(); + expect(mic.dataset.listening).toBe("yes"); + resolveStart(); + await settle(); + }); + + it("sends the final result that lands just after the recognizer stops, not the last interim guess", async () => { + // The plugin emits `listeningState: stopped` before the final, best result + // — delivered as one more `partialResults`. A grace window after `stopped` + // is what lets the final transcript be the one that gets sent. + localStorage.setItem("vogt.assistant.voice.final_result_grace_ms", "0"); + const { mic } = await mountAssistant(); + fireEvent.pointerDown(mic, { pointerId: 1 }); + await settle(); + listenerFor("partialResults")?.({ matches: ["what is on"] }); + listenerFor("listeningState")?.({ status: "stopped" }); + listenerFor("partialResults")?.({ matches: ["what is on top"] }); + await settle(); + const bodies = assistantMessageBodies(); + expect(bodies).toHaveLength(1); + expect(bodies[0]?.text).toBe("what is on top"); + }); }); @@ -376,6 +442,103 @@ describe("what the recognizer heard, repaired before it is sent", () => { }); }); +// -- Hands-free conversation (WI-174) -------------------------- +// +// The loop itself is proven in `voiceTurn.test.ts` off a fake clock; these +// prove the *wiring* — that the surface offers it only when it can run, that +// turning it on opens the recognizer, and that a spoken reply re-opens the mic +// for the next turn without a touch. The half a device demo cannot show: that +// the mic came back on its own. + +/** A speech-synth stub whose last utterance fires `onend`, so the hands-free + * loop's "reply finished → re-open the mic" actually advances under test. */ +function captureConversationSpeech(): { spoken: () => string[] } { + const spoken: string[] = []; + class Utterance { + text: string; + onend: (() => void) | null = null; + constructor(text: string) { + this.text = text; + } + } + vi.stubGlobal("SpeechSynthesisUtterance", Utterance); + vi.stubGlobal("speechSynthesis", { + speak: (u: { text: string; onend?: (() => void) | null }) => { + spoken.push(u.text); + if (u.onend) queueMicrotask(() => u.onend?.()); + }, + cancel: () => {}, + }); + return { spoken: () => spoken.filter((t) => t.trim()) }; +} + +describe("hands-free conversation", () => { + beforeEach(() => { + for (const fn of Object.values(recognition)) fn.mockClear(); + localStorage.removeItem("vogt.assistant.voice.silence_duration_ms"); + }); + + it("is disabled, with a reason, when there is nothing to speak with", async () => { + // No on-device synthesis, no server TTS: hands-free has no mouth, so the + // control is present (discoverable) but disabled and says why. + vi.unstubAllGlobals(); // drop any speechSynthesis a prior test stubbed + const { container } = await mountAssistant(); + const convo = container.querySelector( + '[data-testid="assistant-conversation"]', + ) as HTMLButtonElement; + expect(convo).toBeTruthy(); + expect(convo.disabled).toBe(true); + expect(convo.getAttribute("title")).toContain("spoken replies"); + }); + + it("opens the recognizer when turned on, and shows a live status", async () => { + captureConversationSpeech(); + const { container } = await mountAssistant(); + const convo = container.querySelector( + '[data-testid="assistant-conversation"]', + ) as HTMLButtonElement; + expect(convo.disabled).toBe(false); + fireEvent.click(convo); + await settle(); + expect(convo.getAttribute("aria-pressed")).toBe("true"); + expect(recognition.start).toHaveBeenCalledTimes(1); + expect(container.querySelector('[data-testid="voice-status"]')).toBeTruthy(); + }); + + it("runs a whole turn hands-free: sends on silence, speaks the reply, re-opens the mic", async () => { + const speech = captureConversationSpeech(); + localStorage.setItem("vogt.assistant.voice.silence_duration_ms", "0"); + const { container } = await mountAssistant({ + "POST /api/assistant/message": { + body: { + reply: "On top is the forge adapter.", + pending_action: null, + tool_trace: [], + }, + }, + }); + fireEvent.click( + container.querySelector('[data-testid="assistant-conversation"]')!, + ); + await settle(); + expect(recognition.start).toHaveBeenCalledTimes(1); // mic open for turn one + + // Speak, then go quiet: the silence timer (0ms) ends the turn and sends. + listenerFor("partialResults")?.({ matches: ["what is on top"] }); + await settle(); + expect( + assistantRequests().filter((url) => url.includes("message")), + ).toHaveLength(1); + + // The reply is spoken through the loop, and its end re-opens the mic — the + // whole point of hands-free, and the part no microphone can show you. + await settle(); + expect(speech.spoken()).toContain("On top is the forge adapter."); + await settle(); + expect(recognition.start).toHaveBeenCalledTimes(2); // turn two, no touch + }); +}); + // -- Provider profiles ---------------------------------------- describe("choosing which backend answers", () => { diff --git a/web/src/__tests__/voiceTurn.test.ts b/web/src/__tests__/voiceTurn.test.ts new file mode 100644 index 00000000..bf1f45dc --- /dev/null +++ b/web/src/__tests__/voiceTurn.test.ts @@ -0,0 +1,258 @@ +// The hands-free loop, proven without a recogniser, a synth, or a DOM. +// +// `voiceTurn.ts` is a pure state machine on purpose: every effect is a port, +// and time is an injected clock. So these tests drive the exact interleavings a +// device is worst at showing — a release mid-arming, a final result after the +// stop, an idle session, an approval that must never be spoken past — and read +// them off a list of port calls rather than off a microphone. + +import { describe, expect, it } from "vitest"; + +import { + VOICE_CONFIG_DEFAULTS, + VoiceConversation, + readVoiceConfig, + voiceStatusLabel, + type Scheduler, + type VoiceConfig, + type VoicePorts, + type VoiceState, +} from "../voiceTurn"; + +/** A manual clock: timers fire only when the test advances time, earliest + * first, and a timer that schedules another is honoured in the same advance. */ +function makeClock() { + let now = 0; + let nextId = 1; + let timers: { id: number; fn: () => void; at: number }[] = []; + const scheduler: Scheduler = { + set(fn, ms) { + const id = nextId++; + timers.push({ id, fn, at: now + ms }); + return id; + }, + clear(id) { + timers = timers.filter((t) => t.id !== id); + }, + }; + return { + scheduler, + advance(ms: number) { + const target = now + ms; + for (;;) { + const due = timers + .filter((t) => t.at <= target) + .sort((a, b) => a.at - b.at)[0]; + if (!due) break; + now = due.at; + timers = timers.filter((t) => t.id !== due.id); + due.fn(); + } + now = target; + }, + }; +} + +function makePorts() { + const events: string[] = []; + const states: VoiceState[] = []; + const ports: VoicePorts = { + openMic: () => events.push("openMic"), + closeMic: () => events.push("closeMic"), + sendTurn: (t) => events.push(`sendTurn:${t}`), + speak: (t) => events.push(`speak:${t}`), + stopSpeaking: () => events.push("stopSpeaking"), + onChange: (s) => states.push(s), + onEnded: (r) => events.push(`ended:${r}`), + }; + const count = (prefix: string) => + events.filter((e) => e === prefix || e.startsWith(`${prefix}:`)).length; + return { ports, events, states, count }; +} + +function cfg(over: Partial = {}): VoiceConfig { + return { ...VOICE_CONFIG_DEFAULTS, ...over }; +} + +/** Begin and reach `listening` — the common preamble. */ +function live(over: Partial = {}) { + const clock = makeClock(); + const p = makePorts(); + const vc = new VoiceConversation(p.ports, cfg(over), clock.scheduler); + vc.begin(); + vc.micReady(); + return { clock, vc, ...p }; +} + +describe("the hands-free conversation loop", () => { + it("sends a turn once, when the speaker goes quiet", () => { + const { clock, vc, events, count } = live({ silence_duration_ms: 1000 }); + vc.partial("what is on top"); + expect(vc.getState()).toBe("listening"); + clock.advance(1000); + expect(events).toContain("sendTurn:what is on top"); + expect(count("sendTurn")).toBe(1); + expect(events).toContain("closeMic"); // half-duplex: mic shut while sending + expect(vc.getState()).toBe("sending"); + }); + + it("speaks the reply, then re-opens the mic — the loop's whole point", () => { + const { clock, vc, events, count } = live({ silence_duration_ms: 1000 }); + vc.partial("what is on top"); + clock.advance(1000); + vc.replied({ text: "The forge adapter is on top.", hasPendingAction: false }); + expect(vc.getState()).toBe("speaking"); + expect(events).toContain("speak:The forge adapter is on top."); + const opensBefore = count("openMic"); + vc.speechFinished(); + expect(vc.getState()).toBe("arming"); + expect(count("openMic")).toBe(opensBefore + 1); + vc.micReady(); + expect(vc.getState()).toBe("listening"); + }); + + it("does not tear down a recogniser that is still arming when the user ends", () => { + const clock = makeClock(); + const { ports, events } = makePorts(); + const vc = new VoiceConversation(ports, cfg(), clock.scheduler); + vc.begin(); // arming; openMic issued + vc.end("user"); // lands before micReady + expect(vc.getState()).toBe("arming"); // deferred, not ended + expect(events).not.toContain("closeMic"); + vc.micReady(); // recogniser is up now — apply the end + expect(vc.getState()).toBe("ended"); + expect(events).toContain("closeMic"); + expect(events).toContain("ended:user"); + }); + + it("sends the final result that lands in the grace window after the recogniser stops", () => { + const { clock, vc, events } = live({ final_result_grace_ms: 300, silence_duration_ms: 1000 }); + vc.partial("what is on"); + vc.recognizerStopped(); // endpointing; grace running + vc.partial("what is on top"); // the late, best result + expect(vc.getState()).toBe("endpointing"); + clock.advance(300); + expect(events).toContain("sendTurn:what is on top"); + }); + + it("ends the session after too many empty turns in a row", () => { + const { clock, vc, events } = live({ max_empty_turns: 3, final_result_grace_ms: 100, max_turn_ms: 100_000 }); + for (let turn = 0; turn < 3; turn += 1) { + vc.recognizerStopped(); // heard nothing + clock.advance(100); + if (turn < 2) vc.micReady(); // the loop re-armed the mic + } + expect(vc.getState()).toBe("ended"); + expect(events).toContain("ended:empty_turns"); + }); + + it("ends the session when it has been idle too long", () => { + const { clock, vc, events } = live({ idle_timeout_ms: 5000, max_turn_ms: 100_000 }); + clock.advance(5000); // no speech at all + expect(vc.getState()).toBe("ended"); + expect(events).toContain("ended:idle"); + }); + + it("pauses for an on-screen approval and never re-opens the mic to answer it", () => { + const { clock, vc, count } = live({ silence_duration_ms: 1000 }); + vc.partial("move WI-7 to done"); + clock.advance(1000); // sending + const opensBeforeAction = count("openMic"); + vc.replied({ + text: "I'd like to make a Vogt change: work.transition on WI-7. Approve on screen.", + hasPendingAction: true, + }); + expect(vc.getState()).toBe("paused_for_approval"); + // Announced, but the mic did not re-open — a misheard "yes" authorises nothing. + expect(count("openMic")).toBe(opensBeforeAction); + // The user approves on screen; the action's follow-up reply resumes the loop. + vc.replied({ text: "Done.", hasPendingAction: false }); + expect(vc.getState()).toBe("speaking"); + vc.speechFinished(); + expect(vc.getState()).toBe("arming"); + }); + + it("keeps the session alive when muted, and captures nothing until unmuted", () => { + const { clock, vc, events, count } = live({ silence_duration_ms: 1000 }); + vc.toggleMute(); + expect(vc.isMuted()).toBe(true); + expect(vc.isActive()).toBe(true); + expect(events).toContain("closeMic"); + vc.partial("ignored while muted"); + clock.advance(1000); + expect(count("sendTurn")).toBe(0); + vc.toggleMute(); + expect(vc.isMuted()).toBe(false); + expect(vc.getState()).toBe("arming"); // re-opened for a fresh turn + }); + + it("does not double-send when our own closeMic synchronously stops the recognizer", () => { + // Web Speech `stop()` fires `onend`, which the host forwards as + // `recognizerStopped`. A synchronous one lands mid-endpoint, before the + // turn has left `listening` — the exact shape that used to schedule a + // second endpoint and send the turn twice. + const clock = makeClock(); + const events: string[] = []; + let vc!: VoiceConversation; + const ports: VoicePorts = { + openMic: () => events.push("openMic"), + closeMic: () => { + events.push("closeMic"); + vc.recognizerStopped(); // the induced, synchronous stop + }, + sendTurn: (t) => events.push(`sendTurn:${t}`), + speak: () => {}, + stopSpeaking: () => {}, + onChange: () => {}, + onEnded: () => {}, + }; + vc = new VoiceConversation( + ports, + cfg({ silence_duration_ms: 0, final_result_grace_ms: 0 }), + clock.scheduler, + ); + vc.begin(); + vc.micReady(); + vc.partial("what is on top"); + clock.advance(0); // silence → endpoint → sending → closeMic (induces stop) + clock.advance(1000); // let any stray grace timer fire + expect(events.filter((e) => e.startsWith("sendTurn:")).length).toBe(1); + }); + + it("stops a playing reply and closes the mic when the session ends", () => { + const { vc, events } = live(); + vc.end("user"); + expect(events).toContain("stopSpeaking"); + expect(events).toContain("closeMic"); + expect(events).toContain("ended:user"); + expect(vc.getState()).toBe("ended"); + }); +}); + +describe("voice config", () => { + it("is the generic defaults when nothing is stored", () => { + expect(readVoiceConfig(() => null)).toEqual(VOICE_CONFIG_DEFAULTS); + }); + + it("takes localStorage overrides, and ignores malformed values", () => { + const store: Record = { + "vogt.assistant.voice.silence_duration_ms": "800", + "vogt.assistant.voice.interrupt_response": "true", + "vogt.assistant.voice.idle_timeout_ms": "not-a-number", + }; + const c = readVoiceConfig((k) => store[k] ?? null); + expect(c.silence_duration_ms).toBe(800); + expect(c.interrupt_response).toBe(true); + expect(c.idle_timeout_ms).toBe(VOICE_CONFIG_DEFAULTS.idle_timeout_ms); + }); +}); + +describe("the status chip label", () => { + it("reads Muted over the capture phases, and names the rest", () => { + expect(voiceStatusLabel("listening", true)).toBe("Muted"); + expect(voiceStatusLabel("listening", false)).toBe("Listening…"); + expect(voiceStatusLabel("sending", false)).toBe("Sending"); + expect(voiceStatusLabel("speaking", true)).toBe("Speaking"); // mute does not hide speaking + expect(voiceStatusLabel("paused_for_approval", false)).toBe("Paused for approval"); + }); +}); diff --git a/web/src/styles.css b/web/src/styles.css index f82c40b4..5be165d8 100644 --- a/web/src/styles.css +++ b/web/src/styles.css @@ -11113,6 +11113,65 @@ a.vogt-projects-tag:hover { opacity: 0.5; cursor: not-allowed; } +/* Hands-free conversation (WI-174): the head toggle, the live status chip, and + the mic's muted state. */ +.assistant-convo[aria-pressed="true"] { + color: var(--accent); + border-color: var(--accent); +} +.assistant-convo:disabled { + opacity: 0.4; + cursor: not-allowed; +} +.assistant-voice-status { + display: inline-flex; + align-items: center; + gap: 6px; + font-size: 12px; + color: var(--fg-muted, var(--fg)); + opacity: 0.9; + margin: 2px 0; +} +.assistant-voice-dot { + width: 8px; + height: 8px; + border-radius: 50%; + background: var(--activity-running); + flex: none; +} +.assistant-voice-status[data-state="listening"] .assistant-voice-dot, +.assistant-voice-status[data-state="arming"] .assistant-voice-dot, +.assistant-voice-status[data-state="endpointing"] .assistant-voice-dot { + animation: assistant-voice-pulse 1.4s ease-in-out infinite; +} +.assistant-voice-status[data-state="speaking"] .assistant-voice-dot { + background: var(--accent); +} +.assistant-voice-status[data-state="sending"] .assistant-voice-dot, +.assistant-voice-status[data-state="paused_for_approval"] .assistant-voice-dot { + background: var(--activity-waiting); +} +.assistant-voice-status[data-muted="yes"] .assistant-voice-dot { + background: var(--activity-errored); + animation: none; +} +@keyframes assistant-voice-pulse { + 0%, + 100% { + opacity: 1; + } + 50% { + opacity: 0.35; + } +} +@media (prefers-reduced-motion: reduce) { + .assistant-voice-status .assistant-voice-dot { + animation: none !important; + } +} +.assistant-mic[data-muted="yes"] { + opacity: 0.55; +} .assistant-stop { border-color: var(--activity-waiting); color: var(--activity-waiting); diff --git a/web/src/voiceTurn.ts b/web/src/voiceTurn.ts new file mode 100644 index 00000000..fecaeab8 --- /dev/null +++ b/web/src/voiceTurn.ts @@ -0,0 +1,466 @@ +// Hands-free conversation, as a pure state machine (WI-174 v1). +// +// The Assistant surface already has spoken replies and a hold/tap microphone; +// what it lacks is anything that keeps *listening between turns*. This module is +// that loop, and only that loop: speak → go quiet → the turn sends → the reply +// is spoken → the mic re-opens → no touch between turns. +// +// It is deliberately backend-agnostic and side-effect-free. Every effect — open +// the mic, close it, send a turn, speak a reply — is a *port* the host supplies +// (`Assistant.tsx` on Android's native recognizer, Web Speech on the desktop, +// or server STT). The machine only decides *when*. That is what makes the loop +// testable without a recognizer, a synth, or a DOM, and what lets a different +// backend — a duplex realtime session, say — drop in behind the same ports +// later without touching the transitions here. +// +// There is no open standard for hands-free turn-taking; the shape below (a +// VAD/endpointing loop, an idle timeout, a mute that keeps the session alive, a +// listening/thinking/speaking state machine) is the de facto one shared by +// OpenAI Realtime, Gemini Live, Grok Voice Agent, LiveKit, Pipecat and Open +// WebUI Call mode. The config keys and their names mirror the OpenAI Realtime +// vocabulary so a Realtime-shaped backend can adopt them unrenamed. +// +// v1 is half-duplex: the mic is closed while a reply plays and re-opens when it +// finishes (the Android WebView has no echo-safe capture during playback, so +// barge-in — `interrupt_response` — is designed-for but off; see +// `docs/local/VOICE_HANDSFREE_DESIGN.md`). + +/** The phases of a hands-free session. `muted` is orthogonal to these — a + * session stays in its phase while muted, it just stops capturing. */ +export type VoiceState = + | "idle" // not in a conversation + | "arming" // opening the mic for a turn; not capturing yet + | "listening" // capturing a turn + | "endpointing" // the turn ended (silence or the recognizer stopped); settling before send + | "sending" // the turn is with the engine, awaiting a reply + | "speaking" // a reply is playing; the mic is closed (v1 half-duplex) + | "paused_for_approval" // a reply carried a pending action; mic closed until it is resolved on screen + | "ended"; // the conversation is over; `reason` says why + +/** Why a conversation ended, for the status chip and for the caller. */ +export type EndReason = + | "user" // turned off, or the surface was left + | "idle" // no speech for `idle_timeout_ms` + | "empty_turns" // `max_empty_turns` in a row heard nothing + | "no_backend" // STT or TTS went away mid-session + | "error"; // an unrecoverable fault + +/** Turn-detection and lifetime tuning, in the OpenAI Realtime vocabulary. + * Read from `vogt.assistant.voice.*` localStorage so any deployment can tune a + * session without a rebuild; the same `silence_duration_ms` / + * `final_result_grace_ms` the single-turn tap (WI-173) already reads. */ +export interface VoiceConfig { + /** Quiet after the last transcript change that ends a turn. Field numbers: + * OpenAI 500, Gemini ~800, LiveKit 550, voicemode 1000, Open WebUI 2000. On + * the Android recogniser (whose dictation-mode end-of-speech is late) JS + * owns this; 1000 is the honest default there, ~800 with a real VAD. */ + silence_duration_ms: number; + /** Grace after the recogniser's own stop, because the plugin emits `stopped` + * before the final (best) result — one more partial. */ + final_result_grace_ms: number; + /** A hard ceiling on a single turn, so a stuck-open recogniser still ends. */ + max_turn_ms: number; + /** No speech for this long ends the whole session. Every stack has one. */ + idle_timeout_ms: number; + /** This many turns in a row that heard nothing ends the session. */ + max_empty_turns: number; + /** Barge-in: interrupt a playing reply when the speaker starts. Off in v1 + * (no echo-safe capture on the WebView path); the machine is built so it can + * turn on in v2 without a transition change. */ + interrupt_response: boolean; +} + +export const VOICE_CONFIG_DEFAULTS: VoiceConfig = { + silence_duration_ms: 1000, + final_result_grace_ms: 300, + max_turn_ms: 30_000, + idle_timeout_ms: 60_000, + max_empty_turns: 3, + interrupt_response: false, +}; + +const VOICE_CFG_PREFIX = "vogt.assistant.voice."; + +/** Read the config, applying localStorage overrides over the generic defaults. + * A missing or malformed value keeps the default rather than throwing — the + * same defensive read the tap path uses, because a locked-down browser makes + * even `getItem` throw. Injectable for tests. */ +export function readVoiceConfig( + read: (key: string) => string | null = (key) => { + try { + return localStorage.getItem(key); + } catch { + return null; + } + }, +): VoiceConfig { + const num = (key: keyof VoiceConfig, fallback: number): number => { + const raw = read(VOICE_CFG_PREFIX + key); + if (raw === null) return fallback; + const n = Number(raw); + return Number.isFinite(n) && n >= 0 ? n : fallback; + }; + const bool = (key: keyof VoiceConfig, fallback: boolean): boolean => { + const raw = read(VOICE_CFG_PREFIX + key); + if (raw === null) return fallback; + return raw === "1" || raw === "true"; + }; + return { + silence_duration_ms: num("silence_duration_ms", VOICE_CONFIG_DEFAULTS.silence_duration_ms), + final_result_grace_ms: num("final_result_grace_ms", VOICE_CONFIG_DEFAULTS.final_result_grace_ms), + max_turn_ms: num("max_turn_ms", VOICE_CONFIG_DEFAULTS.max_turn_ms), + idle_timeout_ms: num("idle_timeout_ms", VOICE_CONFIG_DEFAULTS.idle_timeout_ms), + max_empty_turns: num("max_empty_turns", VOICE_CONFIG_DEFAULTS.max_empty_turns), + interrupt_response: bool("interrupt_response", VOICE_CONFIG_DEFAULTS.interrupt_response), + }; +} + +/** A reply the engine returned for a sent turn, reduced to the two facts the + * loop turns on: whether there is something to speak, and whether it carried a + * pending action that must be approved on screen before the loop goes on. */ +export interface VoiceReply { + /** The spoken text, or null/empty when the reply was an action only. */ + text: string | null; + /** A `pending_action` was attached: the loop pauses, mic closed, until the + * on-screen approve/deny — voice never approves anything (ENGINE.md §6). */ + hasPendingAction: boolean; +} + +/** The effects the machine asks the host to perform. Every one is a command, + * never a query: the machine holds no reference to a recogniser or a synth. */ +export interface VoicePorts { + /** Begin capturing a turn. The host confirms the recogniser is live by + * calling `micReady()`; a capture failure is reported via `end("error")`. */ + openMic(): void; + /** Stop capturing. Idempotent — called on every turn end and on mute. */ + closeMic(): void; + /** Send the captured turn's text to the engine. The host reports the outcome + * through `replied()` or `sendFailed()`. */ + sendTurn(text: string): void; + /** Speak a reply. The host calls `speechFinished()` when playback ends (or at + * once when there is nothing to play). */ + speak(text: string): void; + /** Stop any reply currently playing (session end, barge-in in v2). */ + stopSpeaking(): void; + /** State changed — for the status chip. Called with the new phase and the + * live mute flag on every transition. */ + onChange(state: VoiceState, muted: boolean): void; + /** The conversation ended; `reason` is why. The host turns the toggle off. */ + onEnded(reason: EndReason): void; +} + +/** A tiny timer seam so tests drive time deterministically. Real one below. */ +export interface Scheduler { + set(fn: () => void, ms: number): number; + clear(id: number): void; +} + +export const realScheduler: Scheduler = { + set: (fn, ms) => setTimeout(fn, ms) as unknown as number, + clear: (id) => clearTimeout(id), +}; + +/** + * The hands-free loop. Constructed with its ports; driven by the host calling + * the event methods (`begin`, `micReady`, `partial`, `recognizerStopped`, + * `replied`, `speechFinished`, `sendFailed`, `toggleMute`, `end`). It performs + * no I/O of its own beyond the injected scheduler. + */ +export class VoiceConversation { + private state: VoiceState = "idle"; + private muted = false; + private text = ""; // the current turn's best transcript so far + private emptyTurns = 0; + private silenceTimer: number | null = null; + private maxTurnTimer: number | null = null; + private idleTimer: number | null = null; + // A release/stop that arrived while the mic was still arming: applied on + // `micReady`, never against a recogniser that is not up yet (the WI-173 race, + // at the loop's altitude). + private endRequestedWhileArming: EndReason | null = null; + // True while we are closing the mic ourselves. Web Speech's `stop()` fires + // `onend` (which the host forwards as `recognizerStopped`); a synchronous one + // would otherwise re-enter the turn end we are already in — a double send. + private closingMic = false; + + constructor( + private readonly ports: VoicePorts, + private readonly cfg: VoiceConfig = readVoiceConfig(), + private readonly clock: Scheduler = realScheduler, + ) {} + + getState(): VoiceState { + return this.state; + } + isMuted(): boolean { + return this.muted; + } + /** In a live conversation (not idle, not ended). */ + isActive(): boolean { + return this.state !== "idle" && this.state !== "ended"; + } + + // -- events the host feeds in ------------------------------------------ + + /** The user turned Conversation on. */ + begin(): void { + if (this.isActive()) return; + this.muted = false; + this.emptyTurns = 0; + this.endRequestedWhileArming = null; + this.arm(); + } + + /** The recogniser is live and capturing (host confirms `openMic` succeeded). */ + micReady(): void { + if (this.state !== "arming") return; + if (this.endRequestedWhileArming) { + const reason = this.endRequestedWhileArming; + this.endRequestedWhileArming = null; + // Out of `arming` first, so `end` runs its real teardown rather than + // re-deferring: the recogniser is up now, so closing it is correct. + this.state = "listening"; + this.end(reason); + return; + } + this.text = ""; + this.enter("listening"); + // The turn's hard ceiling, and the session idle clock, both start now. + this.armMaxTurn(); + this.armIdle(); + } + + /** A partial transcript arrived. Captures the best text and rearms the turn's + * silence clock; any speech also resets the session idle clock. */ + partial(text: string): void { + if (this.muted) return; + if (this.state !== "listening" && this.state !== "endpointing" && this.state !== "arming") { + return; + } + // A partial can beat `micReady` on a fast device; treat it as listening. + if (this.state === "arming") { + this.text = ""; + this.enter("listening"); + this.armMaxTurn(); + } + if (text.trim()) this.text = text.trim(); + // In the grace window after the recogniser's own stop, a late partial is + // the final (best) result: capture it and let the grace timer fire. Do not + // rearm the long silence window — the turn has already ended. + if (this.state === "endpointing") return; + this.armSilence(); + this.armIdle(); + } + + /** The recogniser stopped on its own — the fallback turn end. Waits out the + * grace so the final result (one more partial after `stopped`) is included. */ + recognizerStopped(): void { + // Not a real end if we are the ones closing the mic (turn already ending). + if (this.closingMic) return; + if (this.muted) return; + if (this.state !== "listening" && this.state !== "endpointing") return; + this.enter("endpointing"); + this.clearSilence(); + this.silenceTimer = this.clock.set(() => this.endpoint(), this.cfg.final_result_grace_ms); + } + + /** The engine answered a sent turn (or a resolved approval). */ + replied(reply: VoiceReply): void { + if (this.state !== "sending" && this.state !== "paused_for_approval") return; + if (reply.hasPendingAction) { + // Announce (the host speaks it) and wait for the on-screen decision. No + // mic: a misheard "yes" must authorise nothing. + this.enter("paused_for_approval"); + if (reply.text && reply.text.trim()) this.ports.speak(reply.text); + return; + } + if (reply.text && reply.text.trim()) { + this.enter("speaking"); + this.ports.speak(reply.text); + return; + } + // Nothing to say — straight back to listening. + this.resumeListening(); + } + + /** A reply finished playing (or there was nothing to play). Half-duplex: the + * mic re-opens now. */ + speechFinished(): void { + if (this.state !== "speaking") return; + this.resumeListening(); + } + + /** The turn's send failed. The mode stays (the surface shows the failed + * bubble + Retry); the loop re-opens the mic for another turn. */ + sendFailed(): void { + if (this.state !== "sending") return; + this.resumeListening(); + } + + /** Toggle mute. Muting keeps the session alive but stops capture; unmuting + * re-opens the mic if the session is in a capture phase. */ + toggleMute(): void { + if (!this.isActive()) return; + this.muted = !this.muted; + if (this.muted) { + this.clearSilence(); + this.clearMaxTurn(); + this.closeMic(); + // The idle clock keeps running while muted: a muted session left forever + // still ends. + this.emit(); + return; + } + // Unmuted: if we were in a capture phase, start a fresh turn. + if (this.state === "listening" || this.state === "endpointing") { + this.arm(); + } else { + this.emit(); + } + } + + /** End the conversation. Safe from any state; idempotent once ended. */ + end(reason: EndReason = "user"): void { + if (this.state === "ended" || this.state === "idle") { + // Nothing live, but still surface the reason for a caller that asked. + if (this.state !== "ended") { + this.state = "ended"; + this.ports.onEnded(reason); + } + return; + } + if (this.state === "arming") { + // The mic is still coming up; remember the end and apply it at micReady, + // rather than closing a recogniser that is not up yet. + this.endRequestedWhileArming = reason; + return; + } + this.clearAll(); + this.closeMic(); + this.ports.stopSpeaking(); + this.state = "ended"; + this.ports.onChange(this.state, this.muted); + this.ports.onEnded(reason); + } + + // -- internal transitions ---------------------------------------------- + + /** Open the mic for a turn (or wait, if muted). */ + private arm(): void { + this.enter("arming"); + this.armIdle(); + if (this.muted) return; // a muted session sits armed until unmuted + this.ports.openMic(); + } + + /** The turn ended: send what was heard, or count an empty turn. */ + private endpoint(): void { + this.clearSilence(); + this.clearMaxTurn(); + const heard = this.text.trim(); + if (heard) { + this.emptyTurns = 0; + // Enter `sending` first, then close: an induced stop reads as sending, not + // as another turn to end. + this.enter("sending"); + this.closeMic(); + this.ports.sendTurn(heard); + return; + } + // Heard nothing this turn. + this.emptyTurns += 1; + if (this.emptyTurns >= this.cfg.max_empty_turns) { + this.end("empty_turns"); + return; + } + // Try again: a fresh turn on the same open session. + this.closeMic(); + this.arm(); + } + + /** Reopen the mic after a reply (or a no-op reply / failed send). */ + private resumeListening(): void { + this.text = ""; + this.arm(); + } + + private enter(state: VoiceState): void { + this.state = state; + this.emit(); + } + + private emit(): void { + this.ports.onChange(this.state, this.muted); + } + + /** Close the mic, fenced so a synchronous `onend` → `recognizerStopped` from + * our own stop cannot re-enter the turn end. */ + private closeMic(): void { + this.closingMic = true; + try { + this.ports.closeMic(); + } finally { + this.closingMic = false; + } + } + + private armSilence(): void { + this.clearSilence(); + this.silenceTimer = this.clock.set(() => this.endpoint(), this.cfg.silence_duration_ms); + } + private clearSilence(): void { + if (this.silenceTimer !== null) { + this.clock.clear(this.silenceTimer); + this.silenceTimer = null; + } + } + private armMaxTurn(): void { + this.clearMaxTurn(); + this.maxTurnTimer = this.clock.set(() => this.endpoint(), this.cfg.max_turn_ms); + } + private clearMaxTurn(): void { + if (this.maxTurnTimer !== null) { + this.clock.clear(this.maxTurnTimer); + this.maxTurnTimer = null; + } + } + private armIdle(): void { + this.clearIdle(); + this.idleTimer = this.clock.set(() => this.end("idle"), this.cfg.idle_timeout_ms); + } + private clearIdle(): void { + if (this.idleTimer !== null) { + this.clock.clear(this.idleTimer); + this.idleTimer = null; + } + } + private clearAll(): void { + this.clearSilence(); + this.clearMaxTurn(); + this.clearIdle(); + } +} + +/** A short, human label for the status chip, given the phase and mute flag. */ +export function voiceStatusLabel(state: VoiceState, muted: boolean): string { + if (muted && (state === "listening" || state === "arming" || state === "endpointing")) { + return "Muted"; + } + switch (state) { + case "arming": + case "listening": + return "Listening…"; + case "endpointing": + return "Listening…"; + case "sending": + return "Sending"; + case "speaking": + return "Speaking"; + case "paused_for_approval": + return "Paused for approval"; + case "ended": + return "Ended"; + default: + return ""; + } +} diff --git a/web/tests/browser/gui.spec.ts b/web/tests/browser/gui.spec.ts index 647de608..bf285732 100644 --- a/web/tests/browser/gui.spec.ts +++ b/web/tests/browser/gui.spec.ts @@ -5409,6 +5409,113 @@ test("A spoken reply whose TTS route is unconfigured shows the fallback notice w await expect(page.getByTestId("speech-status")).toHaveText(/Spoken replies are unavailable/); }); +/** + * Steer the browser onto a *controllable* on-device recognizer and synthesis, + * so a hands-free conversation can be driven turn by turn. `__vogtVoice.say` + * emits a partial into the live recognizer; the fake synth captures the spoken + * text and fires the utterance's `onend`, which is what re-opens the mic. A + * short silence window ends a turn promptly without a wall-clock wait. + */ +async function primeConversationEnv(page: Page): Promise { + await page.addInitScript(() => { + localStorage.setItem("vogt.assistant.tts", "1"); + localStorage.setItem("vogt.assistant.voice.silence_duration_ms", "80"); + const state: { starts: number; spoken: string[]; current: unknown } = { + starts: 0, + spoken: [], + current: null, + }; + class FakeRecognition { + continuous = false; + interimResults = false; + lang = ""; + onresult: ((e: unknown) => void) | null = null; + onend: (() => void) | null = null; + onerror: ((e: unknown) => void) | null = null; + start() { + state.starts += 1; + state.current = this; + } + stop() { + this.onend?.(); + } + abort() { + this.onend?.(); + } + } + const w = window as unknown as { + webkitSpeechRecognition?: unknown; + SpeechRecognition?: unknown; + SpeechSynthesisUtterance?: unknown; + __vogtVoice?: unknown; + }; + w.webkitSpeechRecognition = FakeRecognition; + delete w.SpeechRecognition; + class FakeUtterance { + text: string; + onend: (() => void) | null = null; + constructor(text: string) { + this.text = text; + } + } + w.SpeechSynthesisUtterance = FakeUtterance; + Object.defineProperty(window, "speechSynthesis", { + configurable: true, + value: { + speak: (u: { text?: string; onend?: (() => void) | null }) => { + if (u?.text) state.spoken.push(u.text); + if (u?.onend) setTimeout(() => u.onend?.(), 0); + }, + cancel: () => {}, + }, + }); + w.__vogtVoice = { + state, + say(text: string) { + const r = state.current as { onresult?: (e: unknown) => void } | null; + r?.onresult?.({ results: { length: 1, 0: { length: 1, 0: { transcript: text } } } }); + }, + }; + }); +} + +test("Assistant hands-free conversation: sends on silence, speaks the reply, and re-opens the mic with no touch", async ({ page }) => { + await primeConversationEnv(page); + await installFixtures(page, { assistant_enabled: true }); + const sent: string[] = []; + await page.route("**/api/assistant/message", async (route) => { + sent.push(JSON.parse(route.request().postData() ?? "{}").text as string); + await route.fulfill({ + json: { reply: "On top is the forge adapter.", pending_action: null, tool_trace: [] }, + }); + }); + + await openVoiceAssistant(page); + + // Turn hands-free on: the mic opens itself, and the status chip goes live. + await page.getByTestId("assistant-conversation").click(); + await expect(page.getByTestId("assistant-conversation")).toHaveAttribute("aria-pressed", "true"); + await expect(page.getByTestId("voice-status")).toBeVisible(); + await expect + .poll(() => page.evaluate(() => (window as unknown as { __vogtVoice: { state: { starts: number } } }).__vogtVoice.state.starts)) + .toBeGreaterThanOrEqual(1); + + // Speak, then go quiet: the silence window ends the turn and sends it — no + // button pressed. The transcript that crossed the wire is what was heard. + await page.evaluate(() => (window as unknown as { __vogtVoice: { say(t: string): void } }).__vogtVoice.say("what is on top")); + await expect.poll(() => sent).toContain("what is on top"); + await expect(page.getByText("On top is the forge adapter.")).toBeVisible(); + + // The reply is spoken and, when it finishes, the mic re-opens for the next + // turn on its own — the half a device demo cannot show you. + await expect + .poll(() => page.evaluate(() => (window as unknown as { __vogtVoice: { state: { spoken: string[] } } }).__vogtVoice.state.spoken)) + .toContain("On top is the forge adapter."); + await expect + .poll(() => page.evaluate(() => (window as unknown as { __vogtVoice: { state: { starts: number } } }).__vogtVoice.state.starts)) + .toBeGreaterThanOrEqual(2); +}); + for (const mouseTracking of [false, true]) { test(`terminal swipe owns ${mouseTracking ? "reports wheels to a normal-buffer application" : "moves saved rows once"}`, async ({ page, context }) => { test.skip(test.info().project.name !== "phone", "Touch input needs the phone context");