feat(desktop): hands-free voice — listens, sends what you say, reads the answer aloud, stops when you speak over it

[brcampidelli]

What

A Voice button beside Dictate on the Code screen. On, it keeps the microphone open, sends what you say as turns, reads each answer aloud, and stops reading the moment you speak over it. Item 8 of the list audited on 2026-09-16 ("Voice") — the continuous mode and the barge-in. The wake word is not in this PR (it needs openWakeWord over a continuous audio stream; separate work).

How

Listening — lib/voice/microphone.ts captures 16 kHz mono in 64 ms frames (ScriptProcessorNode; raw samples rather than MediaRecorder, because an utterance is cut out of a continuous capture and needs the audio from before the decision that speech started). lib/voice/segmenter.ts turns loudness readings into utterances: three loud frames in a row to start, 700 ms of quiet to end, 320 ms of pre-roll kept so the first syllable is not clipped, anything with less than 350 ms of speech discarded, a 30 s cap so a monologue still reaches the agent. Each utterance is encoded as WAV (lib/voice/wav.ts — a slice of a WebM stream is not a file) and sent through the same /api/transcribe dictation uses, then out as a turn (force, so a spoken request never raises the batch-proposal card — a card is a thing to click on).

Reading aloud — lib/voice/speaker.ts wraps speechSynthesis: the window's own voices, no key, no download, every language the app has, cancellable mid-word. lib/voice/speech-text.ts reduces the answer's Markdown to sentences: fenced code becomes "(code, shown on screen)", links keep their label, URLs their host, emphasis and headings are dropped. A window without speechSynthesis says so and keeps listening.

Barge-in — the segmenter's floor rule. The floor is tracked from the readings and never from a loud run (a person talking must not raise it under themselves). When the agent starts to read, the first 800 ms of what the mic hears calibrate the floor to the agent's own voice through the speakers, with no event possible; from then on speech 4× louder than that floor cancels the reading and becomes the next message. Two numbers, stated in the module so they can be measured against a real room rather than believed.

The screen — the status line says what the machine is doing: listening, hearing you, transcribing (with how long it took: the local model on a slow machine takes seconds and a line that hides that reads as a hang), reading aloud. The mode is never remembered across launches. No mic is "no microphone available", not a pretence; where nothing can transcribe the button is disabled with dictation's own reason.

Backend — a recording (.wav, .webm, …) is now stored as kind="audio" and no longer pushed through the document converter, which the voice mode would otherwise have paid for on every utterance, several a minute, to get an answer the transcriber never reads.

Honest about what was not tested

I could not put a real microphone and a real room in front of this. The segmenter and the state machine are tested with loudness levels and fakes (below); the two barge-in numbers (800 ms, 4×) and the assumption that a person near the mic is louder than the agent through the speakers are hypotheses until someone speaks at it. Echo cancellation is requested from the WebView; what it cancels differs by platform and is not relied on. The status line's transcription time is there so the first live session yields numbers.

Tests

lib/voice/segmenter.test.ts (10): start/end/hangover, pre-roll kept, short utterance discarded, a single loud frame is not speech, the 30 s cut, flush; the floor follows the room but not a talking person, climbs to the agent's voice during calibration and only louder speech interrupts. wav.test.ts (1): header fields and 16-bit samples, clipping clamps. speech-text.test.ts (3). VoiceMode.test.tsx (9), with a fake mic and a fake voice: on → listening; an utterance → WAV named speech.wav → transcribed → sent, with the time shown; an answer that lands while on is read without its Markdown and the mode returns to listening; speech over the reading cancels it and goes out as the next message; an answer from before the mode came on is never read; off releases the mic and silences the voice; no voice / no mic / nothing can transcribe. test_attachments.py: a recording is stored as audio and a converter that raises proves it is not consulted. Sabotage: cancel() removed from the barge-in branch → the barge-in test fails; the audio kind reverted → the attachments test fails; both restored.

Desktop: tsc clean, 159 files / 1133 tests green. Backend full gate on a clean copy in WSL: ruff and mypy clean; 6546 passed with one failure in test_the_adapter_returns_the_tool_call_it_was_given[ollama_chat-stream-two_parallel_in_order], a file this branch does not touch, which passed 3/3 when re-run alone on the same tree — a timing flake under full-suite load, noted here rather than hidden.

🤖 Generated with Claude Code