Fill voice-mode silence with localized fillers and work sounds

[Piggidragon]

Why

In voice mode, once the single "Let me think about that" phrase has played, nothing is heard while the agent reasons or runs tools. Long silences feel sterile and make it hard to tell whether anything is still happening.

What

Spoken fillers, tied to what is happening

Nothing plays on a timer alone. Wordless murmurs are not used: Chatterbox renders Hmm. and lone words like Okay. twice. The agent already announces each tool call itself (voice-first instructions), so fillers do not repeat that.

• Acknowledgement 0.3 s after a request: Mal sehen. for a question (transcript ends in ?), Alles klar. / Mach ich. otherwise. One- and two-word replies (Danke) get none, and at most one plays every 15 s. It replaces the live "let me think" notice for that turn.
• Slow tool after 6 s, naming what is still running where the call shows it: Die Tests laufen noch., installs, builds, commands, page loads, searches. slowFiller() in web/src/tool-kind.ts does the classification.
• Tool result: Das hat nicht geklappt. 0.8 s after a failed call (at most twice per turn). Okay, mal sehen. 1.2 s after a tool stretch of 4 s or more, if the agent is quiet.
• Still on it only after 15 s of silence, then after 30 s and 45 s.
• A reply that arrives while a filler plays waits for it to finish.
• The phrase tables in web/src/voice-phrases.ts cover en, de, es, fr, it, pt (BR), ru, zh, ja, ko, hi and ar. The thinking and compaction notices use the same tables. First-person phrases are worded gender-neutrally where the language marks gender (hi, ru, ar), since the voice may be either.
• Language: the saved input language is used. With auto-detect, the browser's language is used instead. A language without a table gets wordless murmurs only, never guessed wording, and its status notices stay in English, as before.
• Rendering happens once, on the portal. server/src/voice-clips.ts renders missing clips in the background when voice is set up (/voice/connect), its settings are saved or the portal starts. It pauses while live speech or transcription requests run (5 s quiet window) and keeps a managed runtime's lease while a batch runs. Clips are stored as PCM under DATA_DIR/voice-clips/<version>/<sha256(text)>.pcm. version is a fingerprint of the speech settings and the voice preset or Aria reference, so a voice or language change renders a new set. The four most recently used sets are kept. Auto-detect languages that clients asked for are remembered and kept rendered.
• Voice mode only downloads clips. GET /api/voice/clips?languages=… returns the phrase list with a ready flag and the status notices in the voice language. GET /api/voice/clips/:version/:hash serves a clip as immutable. The browser loads the ready clips and checks back while the rest render. It never renders speech itself.
• The phrase tables live in server/src/voice-phrases.ts. The speech handler's upstream request moved into upstreamSpeech(), which the handler and the renderer share.
• Barge-in, compaction and End fade out a filler that is playing. VOICE_STATUS_SPEECH=false and the sequential pipeline turn all spoken fillers off.

Work sounds (follow the existing interface-sounds toggle)

• web/src/work-sounds.ts synthesizes quiet sounds with WebAudio, so there are no downloads. There are irregular typing runs while a command or edit runs, and a page rustle when a read, search or browser call starts. Both play only during Thinking, never over speech.
• Two soft voiceCue tones mark a tool the listener waited on (≥ 1.5 s) finishing. The failure tone is lower.

Refactors

• web/src/tool-kind.ts: tool classification is shared with VoiceToolActivity.

Tests

• tests/hands-free.test.mts: growing murmur gaps; a pre-rendered thinking phrase replaces live synthesis; barge-in aborts a filler; statusSpeech: false silences fillers; tool announcements (quick tools skipped, once per kind per turn, quiet window after speech); German compaction notices.
• tests/voice-clips.test.mts: clips render once and survive a restart, rendering waits for live speech, a voice change mid-batch, disabled status speech, path safety, pruning, remembered languages.
• tests/voice.test.mts: clip list in the requested language, immutable serving from disk, version changes with the voice.
• tests/voice-fillers.test.mts: silence trimming; every language table is complete; language resolution; tool classification.
• web tsc -b + vite build pass. The hands-free, fillers, speech-pipeline, voice, pcm-stream and voice-panels suites pass.

Not yet checked by ear. How natural the murmurs sound depends on the TTS backend, and the non-English phrases are worth a check by native speakers. Phrases, gains and timings are constants and easy to tune.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}