feat(voice): add Phase 1 voice interaction for Lucia

[Ticed]

Summary

Adds the Phase 1 release candidate for single-host macOS voice interaction in Lucia.

This introduces provider-backed streaming speech, hands-free voice-mode controls, local speech-helper installation, and supporting tests and documentation.

This is opened as a draft for maintainer review. Automated speech regression is now included in CI; only the normal PR checks and a final interactive macOS smoke pass remain
before merge.

Changes

• Add CoreAudio-backed macOS capture and playback support.
• Add a namespace-composable speech-provider contract through speech9p and speechshim9p.
• Add Kokoro TTS, Parakeet streaming STT, Whisper fallback, and wake/VAD helper integration.
• Add half-duplex echo protection, microphone lifecycle control, cancellation, chimes, and streaming partial/final records.
• Add the resident voicemode state machine and unified voice-mode entry through:
  • the Voice resource
  • the compose voice control
  • Ctrl+Space
  • Esc-V
  • Option/Alt+V
  • /voice mode on|off
• Preserve keyboard compose text and make Esc return cleanly to keyboard mode.
• Add utterance grace, append, deduplication, spoken cancellation, approvals, denial, refinement, confidence confirmation, and a capped queued follow-up.
• Add an LLM-free speech test mode and configurable local OpenAI-compatible LLM support.
• Add helper installation, regression tooling, architecture documentation, Phase 1 exit criteria, and Phase 2 scope.

Testing

Completed during implementation:

• tools/speech-regress.sh — 14 suites passed on macOS
• Speech regression suite added to Linux CI
• tools/verify-dis-paths.sh
• Bash and Python syntax checks
• macOS SDL build
• New and expanded speech, provider, voice-mode, Lucia, installer, and host-side tests

Before final merge:

• Confirm the GitHub Actions and full repository test results.
• Complete one interactive macOS smoke pass covering microphone transcription, audible playback and cancellation, voice controls, and an LLM-backed turn.

Phase boundary

Phase 1 covers usable single-host macOS voice interaction.

Two-host/Jetson deployment, remote-audio productization, model distribution, richer queue management, and other cross-host work remain Phase 2 or later.

Checklist

• Code follows the existing style of the files modified
• No .dis build artifacts were committed inside appl/ or tests/
• Documentation was updated
• No secrets, API keys, or credentials are included

[pdfinn]

Thanks for this — it's a serious piece of work. The branch is current with master, CI is green, the tests are genuinely behavioral rather than smoke, the docs are unusually candid about security posture, and the core design (the /n/speech provider seam, the half-duplex echo scheme, the voicemode state machine, compose-text preservation) is sound. The review below found four correctness blockers, one supply-chain blocker, several posture decisions that need to be made explicitly on the maintainer side, and a set of smaller fixes. None of it is fatal to the approach; with the blockers fixed this is a solid Phase 1.

Blockers

1. Typed input silently vanishes in voice mode (merged-tree bug). The voice-mode mutual-exclusion path in appl/cmd/lucibridge.b does writemsg("assistant", "voice mode active — …"). Master's hardened luciuisrv (d1d16f9) validates roles against human|veltro|system|tool and rejects assistant with Rmsg.Error, which lucibridge only logs to stderr. Net merged behavior: with voice mode on, a typed non-slash message is consumed from the input queue, discarded, and the explanatory bubble never renders — the user's text disappears with no feedback. The merge is textually clean; the bug only exists in the combined tree. Use veltro or system, and ideally preserve the typed text rather than dropping it.

Related: the PR's lucibridge.b pre-image is not master — it's master plus the unmerged #505 (fix(security): report failed tool mutations). Merging as-is silently carries that change. Please rebase that file onto master proper (every other modified file is already clean against master tip).

2. boot.sh can abort boot for everyone. The speech probe builds /n/local^$speechprefix^/speech.ctl.sh with no $#speechprefix guard. Where os sh yields nothing (Windows Nt emu, devcmd not bound, restricted namespaces), speechprefix is the empty list and the ^ concatenation raises sh: null list in concatenation, aborting boot.sh before lucibridge, voicemode, the Tasks board, and the plumber start. This hits users who never enable voice. Both ftest probes need an if {! ~ $#speechprefix 0} guard.

3. Supply chain: the installer must pin what it executes. tools/install-speech-helpers.sh clones mudler/parakeet.cpp at default-branch HEAD (--depth 1 --recurse-submodules), compiles it, and executes the resulting binary — whoever controls that repo's HEAD gets code execution on every installer run. The four model downloads are similarly unpinned: Kokoro model/voices from GitHub release assets (replaceable in place), Whisper and Parakeet GGUF via Hugging Face resolve/main (mutable branch refs), two of them from individuals' conversion accounts rather than the model authors. There is no checksum or signature verification anywhere in the file — the only integrity gate is a size floor on the whisper model. Before merge: pin the parakeet.cpp clone to a commit SHA, pin the HF URLs to revision commits, and add a SHA256 manifest for the model files. (The rest of the installer is well-behaved: no sudo, no pipe-to-shell, atomic downloads, everything under ~/.local/share/infernode-speech.)

4. The CoreAudio driver is architecturally wrong for the emu kproc model — and it's dead code. emu/MacOSX/audio.c blocks in pthread_cond_wait while holding the direction QLock (audio_file_read holds inlock across ringread). Inferno's kill path (oshostintr → pthread_kill(SIGUSR1)) interrupts syscalls but does nothing to a condvar wait, so on the TCC mic-denied posture — where the input AudioQueue starts but never delivers buffers, exactly the case tests/host/audio_macos_test.sh describes — the reading proc blocks forever, is unkillable from inside Inferno, and audio_file_close deadlocks behind the held lock. Other backends wrap host blocking in osenter()/osleave() (see emu/FreeBSD/audio.c) for precisely this reason.

Meanwhile emu/MacOSX/emu still lists audio audio-sdl3, so the new driver never builds — yet the CoreAudio/AudioToolbox link flags and the driver-specific test assertions (cannot start CoreAudio …: -66680) ship anyway, and the host test silently exercises the wrong driver. If the CONF is ever flipped, the new driver also regresses drain-on-close (INFR-185: AudioQueueStop(outq, true) + ringfree drops up to 256KB of buffered playback) and the play_buffer_ms/rec_buffer_ms ctl verbs voice setup writes (INFR-194 — audioparse returns 0 on unknown verbs, so the write fails with Ebadarg). Either flip the CONF in this PR and reconcile explicitly with audio-sdl3.c, or drop the driver, the mkfile link flags, and the CoreAudio test assertions until the phase that wires it.

5. Flush/Clunk permanently wedges the wake/listen streams. In both speech9p.b and speechshim9p.b, Flush cancels the async tag and Clunk drops the fid state, but neither clears listenbusy/wakebusy, cancels the spawned asyncwake/asynclisten proc, nor resets the shared provider fd. The parked proc stays blocked in readmountedfd (a wake read blocks indefinitely by design), and since the busy flag only clears in asyncdone, every subsequent open+read returns "error: wake busy" forever. It self-heals only along voicemode's exact exit choreography (cancel + mic off killing the helper); any other client — speechtest, a manual cat, a crash — leaves the stream dead. Flush/Clunk of a parked streaming read needs to reset the provider fd and clear the busy flag (or key the busy state on fid and clear it at Clunk). The bug exists identically in both servers because the async scaffolding is hand-copied between them — see the de-duplication note below.

Posture decisions to make explicit (maintainer side)

These are flagged for discussion, not unilateral fixes — but they need conscious sign-off rather than riding in implicitly:

• lib/voice/speech-{terminal,engine,capture} ship in release artifacts (lib/ is in the release copy loop) and speech-terminal runs /lib/voice/listen, i.e. listen -A 'tcp!*!…' { export /dev } — anonymous export of the entire /dev tree on all interfaces. They're dormant, operator-invoked, and docs/SPEECH-REMOTE-AUDIO.md honestly labels them development-only, but this is the same shape of posture that got tests/agent-harness/ ring-fenced. We need to decide: accept as documented dev tooling, or keep the trio out of the release copy loop until the Phase 2 auth work.
• Boot executes a host-writable file verbatim. boot.sh now runs sh on ~/.local/share/infernode-speech/speech.ctl.sh inside the boot namespace at every boot — a quiet persistence/injection hook for anything with host-user file write. A restricted key value parser instead of sh would close it; at minimum the trust decision deserves a comment.
• The mic is armable by any /mnt/ui-rw holder. input-mode and conversation/voiceinput are 8r666, so anything holding the UI service rw can flip the desktop into voice mode (muting typed input — see blocker 1) or inject "spoken" turns through the path that deliberately bypasses the keyboard pause. Incremental over the pre-existing /mnt/ui exposure, but mic activation without user gesture is a new capability.
• The logon tightening should probably be its own PR. Requiring secstoreautounlock=1 in addition to factotum keys before skipping login ("Never let an API key bypass the login screen") is a correct fix, but it changes boot behavior for every user regardless of voice, and bundling it here makes it invisible in release notes.
• Every GUI boot now starts the full speech surface unconditionally — two resident 9P servers plus the voicemode daemon, with "no speech helpers found" warnings on machines that never opted in. The degradation design is otherwise right (mic only opens on a voice session, helpers soft-fail); the question is whether default-on is intended for Phase 1.

Worth fixing, not blocking

• /n/speech/cancel is a no-op for every engine except the provider path. dosay for cmd/api/local/module routes through playpcm, which does one blocking write of the whole buffer and never checks cancelreq — barge-in only works with the kokoro/shim backend. Fine for Phase 1's shipped config, but the cancel file should reject or document non-provider engines rather than accept a write that does nothing. Also: cancelreq is only reset on a sayq write, so after one cancel the cancel file reads "cancel pending" indefinitely.
• The Alt+V chord lives in the portable emu/port/draw-sdl3.c and SDL_KMOD_ALT includes RALT — on European layouts AltGr+V becomes untypeable on Linux/Android, and the app receives raw ESC,v instead. Gate it to macOS desktop (__APPLE__ && !TARGET_OS_IOS). Also unguarded: key autorepeat (held Alt+V toggles repeatedly) and a stale chord flag if focus is lost while Alt is held.
• Esc chording collisions in lucifer.b kbdproc. The ESC state has no timeout and the chord is intercepted before routing to workspace apps, so Emacs-style M-v in any hosted editor flips the desktop into voice mode; in voice mode, bare Esc is consumed globally, and terminal arrow keys (ESC [ A) half-escape — the ESC exits voice mode and literal [A lands in the compose box.
• lib/voice/speech-engine was never run end-to-end: it invokes bare speechshim9p, which can't resolve on Inferno sh's (/dis .) path — the binary lives at /dis/veltro/speechshim9p.dis (boot.sh gets this right). The script always hits its own "did not mount" guard. The accompanying voice_scripts_test.b can't catch it because it only greps the script text.
• CI cost. The speech-regress.sh step re-runs Limbo suites that the existing runner.dis full-suite step auto-discovers minutes later in the same job — every PR pays twice, inside a 30-minute timeout that the worst-case suite budget can approach. Suggest dropping the Limbo suites from the CI step (keep the three host tests) or path-filtering it. Separately, speech-regress.sh and speech_helpers_test.sh call timeout(1), which stock macOS doesn't ship — broken for the local workflow on the platform the feature targets.
• Dead interface surface. Speech->Partial is added to module/speech.m with a doc comment but nothing constructs or consumes it. The SpeechEngine module contract is batch-shaped (synthesize returning PCM, recognize(audio)) and its only implementation, speechprovider.b, has to no-op both semantics to fit its streaming-file reality — and nothing in-tree wires it. Suggest deferring both until a genuine in-process engine exists.
• De-duplicate the styx scaffolding. The async-helper machinery, serveloop, Flush/Clunk handling, and the INFR-28 raw-Rmsg.Read streaming reply are copy-pasted between speech9p.b and speechshim9p.b ("same pattern as speech9p"). Blocker 5 exists twice because of this; the copies will drift.
• Mount convention. New defaults /n/speechshim and /n/parakeet (and voicemode's /n/speech) — house rule is /mnt/* for app-authored mounts; the pre-existing /n/speech is the precedent you followed, so this is understandable, but let's settle it now before Phase 2 multiplies the mounts.
• tools/parakeet_stream.cpp: mel_buf is only reset when the model emits end-of-utterance, and append_mel_frames recopies the whole buffer per block — under continuous non-speech audio memory grows without bound and per-block cost grows linearly. Trim already-fed frames (keep pre_encode_cache_size of history). Also guard sess.text().substr(finalized_chars) with min(finalized_chars, text.size()) — a hypothesis retraction currently terminates the helper.
• Small cleanups: readuserinput() in lucibridge is dead after the pretoolapproval rewire yet was modified — delete it; the voice-toggle logic is triplicated across luciconv, lucictx, and lucibridge; luciconv.voiceactive() does a synchronous 9P open/read per keystroke instead of caching the input-mode event it already receives; Qconvcontrol/draft fields in luciuisrv skip the safeeventtext/safeattrtext sanitization pattern d1d16f9 just established (contained, but the next audit pass will flag it).

What's good

Worth saying explicitly: the async-read design (parking blocking helper reads, completing via the helper channel so ctl/cancel stay live during a blocking wake read) is the right architecture and the INFR-28 offset-bypass streaming reply is handled correctly; the half-duplex echo suppression and the grace/append/dedup/confirm state machine are internally consistent; compose-text preservation is done right (server-side draft file, compose buffer untouched); the new luciuisrv files follow house patterns; the voicemode/shim/speech9p test suites run against fake helpers so they don't silently skip in CI; the installer's hygiene (atomic writes, no sudo, env-overridable URLs) is good modulo pinning; and the docs' honesty about the remote-audio security posture is appreciated.

Suggested path to merge

1. Fix blockers 1, 2, and 5; rebase lucibridge.b onto master proper.
2. Pin the installer (commit SHAs, HF revisions, SHA256 manifest).
3. Drop the CoreAudio driver + link flags + CoreAudio test assertions from Phase 1 (or flip the CONF and reconcile with audio-sdl3.c — but that's a bigger conversation).
4. Split the logon change into its own PR.
5. We'll settle the posture questions (release inclusion of lib/voice/speech-*, the boot ctl-script execution, the 8r666 mic surface, default-on boot) in this thread before undraft.

🤖 Generated with Claude Code

[Ticed]

Superseded. This branch grew into one 64-file draft that is now conflicting with master, and it is not reviewable in that shape.

I have split it. Open so far, each a single logical change:

• Build and test-tree integrity: guard mk nuke, stale bytecode, and dis paths #550 build and test tree guards
• Close announced sockets on hangup #551 announced socket hangup
• Recover text tool calls and handle a non-SSE response body #552 llmclient SSE and tool JSON parsing
• Resolve API hostnames before dialing #553 llmclient hostname resolution
• Select the host audio device by name, and read back audioctl caps #554 host audio device selection

The speech service itself is the rest of it, and it introduces a new file interface, so it follows a namespace sketch rather than arriving as a PR. Closing this one so it stops showing up as pending review.