From c9182b46aee42698f2c1e2a858004d06d27e04e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?mah-chey=27=20=7C=20/=CB=88m=C9=91=CB=90=2Et=CA=83e=C9=AA/?= =?UTF-8?q?=20=7C=20/=CB=88mat=CD=A1=C9=95=C9=9Bj/?= <155201063+Szowesgad@users.noreply.github.com> Date: Wed, 4 Feb 2026 23:44:50 +0100 Subject: [PATCH 01/77] Refactor and enhance VAD integration for robust speech handling (#14) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * rebase: on [LIST] commit of the following: [ok-commit] f593a8cc777a3213dba2599a0e06a6b7c30fd5e8 Author: Szowesgad Date: Tue Jan 27 17:11:17 2026 +0100 Improve E2E env handling and validation scripts Updated .env.example to clarify its role as the E2E ground truth. Added .env.e2e to .gitignore. Enhanced Makefile to auto-generate and use a clean E2E env for E2E and roundtrip tests. Refactored scripts/validate-envs.sh to support new CLI options for env example validation and E2E env emission, improving automation and reliability of environment setup for tests.[ok-commit] commit 1120944786ad2e49e5d761f19c7d0492f19463a8 Author: Szowesgad Date: Tue Jan 27 17:09:35 2026 +0100 Add initial prompt support for Whisper STT engine Introduces the WHISPER_INITIAL_PROMPT environment variable to guide Whisper transcription with domain vocabulary and formatting hints. Updates decoding parameters, engine logic, and documentation to support and describe this feature. Also adds a new hotkeys contract documentation and minor improvements to .gitignore and UI status messages. commit a77f58c11eefe3beb93f90e54b4c1b97c7c81cbc Author: Szowesgad Date: Tue Jan 27 16:43:48 2026 +0100 Update VAD defaults and environment registry Changed VAD threshold and silence defaults to be less conservative across documentation, scripts, and examples. Refactored environment variable registry: removed deprecated entries, added new variables for models, embedding, LLM, TTS, quality, and system configuration. Updated code to use CODESCRIBE_VAD_ENABLED for auto_silence logic and removed CODESCRIBE_LOG_STDOUT support. Improved env var validation script to handle more patterns and ignore build system vars.[ok-commit] commit 90b4174c9f00d0d0801a81ab07e476e5e0bdf889 Author: Szowesgad Date: Tue Jan 27 15:41:48 2026 +0100 fix(moshi): Cache unavailability to prevent log spam - Add moshi_unavailable flag to RecordingController - Early validate MoshiConfig before trying to create engine - Cache failure state so subsequent hotkey presses don't spam logs - Silent return (Ok) when Moshi unavailable - not an error, just missing models Before: "Moshi model not found" on EVERY Ctrl+Option press After: Single warning, then silent skip Co-Authored-By: Claude Opus 4.5 commit 23a780acb43f336e7faf79abd4b5773d686e2f5e Author: Szowesgad Date: Tue Jan 27 14:19:42 2026 +0100 chore: Clean up legacy VAD env var references - Replace CODESCRIBE_VAD_MAX_SILENCE_SEC → CODESCRIBE_VAD_SILENCE_SEC - Replace AUTO_SILENCE → CODESCRIBE_VAD_ENABLED - Fix duplicate CODESCRIBE_VAD_ENABLED in .env.debug.example - Change eprintln spam to tracing::debug in embedded.rs Files updated: - examples/README.md - scripts/download-silero.sh - docs/OVERLAY_STREAMING.md - docs/ADR/2026-01-26-env.md - docs/ADR/2026-01-26-OVERLAY_STREAMING.md - .env.debug.example - core/stt/whisper/embedded.rs Co-Authored-By: Claude Opus 4.5 commit f248e50a0be43a3df5515c57a1e3cf4d40c078d3 Author: Szowesgad Date: Tue Jan 27 14:10:29 2026 +0100 docs(env): Add complete LLM_USE_STREAMING and TRANSCRIPT_SEND_MODE docs Added detailed documentation with defaults and options: - LLM_USE_STREAMING: default 1, options 1/true (SSE) vs 0/false (buffered) - TRANSCRIPT_SEND_MODE: streaming vs buffered modes Co-Authored-By: Claude Opus 4.5 commit 95c713d8fea4cd3b30aa35cb25fbfdea8fe5069a Author: Szowesgad Date: Tue Jan 27 13:55:34 2026 +0100 fix(vad): Clean VAD config, fix embedding detection, HOLD mode user control BREAKING CHANGES: - Renamed env vars (old ones deprecated): - AUTO_SILENCE → CODESCRIBE_VAD_ENABLED - CODESCRIBE_VAD_MAX_SILENCE_SEC → CODESCRIBE_VAD_SILENCE_SEC - CODESCRIBE_VAD_SENSITIVITY → removed (use THRESHOLD directly) Core fixes: - Fix cfg!(embed_*) not detecting embedded models in workspace builds Now checks weights.len() > 0 instead of compile-time cfg!() macro - VAD defaults less aggressive: threshold 0.35 (was 0.5), silence 2.5s (was 1.2s) - Whisper params stricter: compression 2.0, no_speech 0.5, logprob -0.5 VAD behavior change: - HOLD mode: NO VAD auto-stop (user controls by releasing key) - TOGGLE mode: VAD active (auto-stop after silence) - If user wants to hold in silence for 45 minutes - that's their choice New files: - docs/ENV_REGISTRY.toml: Central registry of all env vars - scripts/validate-envs.sh: Pre-commit validator for env vars Updated docs: - docs/env.md: New VAD var names and defaults - .env.example, .env.debug.example: Clean config templates Co-Authored-By: Claude Opus 4.5 commit e65aedb40a1c9604fb6375c7fa9ba5adae0fa806 Author: Szowesgad Date: Tue Jan 27 12:12:45 2026 +0100 Add comprehensive ADR and user documentation Introduced multiple ADR markdown files covering licensing, installation, team setup, architecture, backlog, feasibility analysis, and Whisper Live streaming. Also added a user guide and related documentation to the docs/ADR directory. Removed obsolete local backend section from .env.debug.example and renamed runner.sh script to tools/runner.sh. commit 76c75749773272ec8563890c21dcd32764307204 Author: Szowesgad Date: Tue Jan 27 09:37:45 2026 +0100 Improve overlay window input and live transcription handling Introduces a custom NSWindow subclass for overlays to allow borderless windows to receive input. Updates live transcription to not auto-stop on silence and adds a stop_without_saving method to audio recorders, preventing unnecessary WAV file creation. Also improves thread safety in overlay state handling and adds an environment variable to control logging to stdout.[ok-commit] commit b53638c9618f8b13f152f4871b87a79ed8171959 Author: Szowesgad Date: Tue Jan 27 07:37:26 2026 +0100 Improve logging and overlay handling, add tracing Adds the tracing-appender dependency and initializes structured logging to both file and stdout in the CLI. Replaces eprintln! with tracing macros for better log management. Fixes overlay blur view initialization in bootstrap and voice chat overlays. Updates assistive loop logic and delta callback routing in RecordingController for more robust session handling.[ok-commit] commit 58037e6901bfd6e230c7cb1ab1291defa1d00d7f Author: Szowesgad Date: Tue Jan 27 05:33:42 2026 +0100 Improve overlay logging and update model embedding logic Added more informative logging to overlay and voice chat show/hide functions for better traceability. Updated hotkey event handling in codescribe daemon to apply settings immediately, reducing race conditions. Revised build.rs to embed Whisper model by default in release builds, with opt-out via CODESCRIBE_NO_EMBED, and clarified logic and warnings for model embedding. commit 66bdad1389403b14ca7a22ccc98d3f889d9bdf0b Author: Szowesgad Date: Tue Jan 27 04:10:28 2026 +0100 Add round-trip pipeline demo and tests Introduces interactive and test-based round-trip (TTS→STT→Embeddings) validation for pipeline integrity. Adds Makefile targets, scripts for Silero VAD model download, HuggingFace cache support for models, and updates documentation for quick start and settings. Improves hotkey handling for conversation mode and refines TTS/STT/embedding integration. commit 68769f248fd1f0ba6e1330a9da44ab94e9692cb5 Author: Szowesgad Date: Tue Jan 27 04:12:33 2026 +0100 Add onboarding bootstrap overlay for macOS Introduces a bootstrap onboarding overlay for first-time users on macOS, including UI, handlers, and integration with the tray menu. The overlay guides users through initial setup steps and is triggered on first launch or via the tray menu. Packaging scripts and Makefile updated for improved model bundling and release process, with enhanced environment variable support for signing and notarization.[ok-commit] commit e44ec4417c906c5713630af79fbfd7e0e9dae42c Author: Szowesgad Date: Mon Jan 26 23:11:45 2026 +0100 Implement assistive voice chat streaming and UI updates Adds support for streaming user transcription into chat bubbles during assistive sessions, including new helpers for appending and finalizing user message text. Refactors overlay routing, session state, and chat bubble styling for improved clarity and Quantum theme consistency. Introduces an assistive VAD loop for hands-off recording and updates UI color palette and font handling for chat bubbles.[ok-commit] commit 17c7a193a3bd725319a0822c8445afd9b548ff42 Author: Szowesgad Date: Mon Jan 26 19:40:40 2026 +0100 Add VAD sensitivity and ONNX support, improve cache handling Introduces CODESCRIBE_VAD_SENSITIVITY and CODESCRIBE_VAD_SILENCE_SEC as preferred VAD controls, mapping sensitivity to threshold and allowing simple silence override. Adds support for ONNX embedder models, improves HuggingFace cache path resolution, and refines environment variable handling for model embedding and cache directories.[ok-commit] commit 9c053a96b31ad6ff7c32a871a069f42847c3a6a8 Author: Szowesgad Date: Mon Jan 26 18:56:46 2026 +0100 Refactor model embedding and HuggingFace cache detection Model embedding is now opt-in for release builds, requiring explicit environment variables to embed models, E5, or TTS. The HuggingFace cache search now checks all possible cache locations instead of just one, improving robustness in both build.rs and hf_cache.rs. Error handling and warnings for missing models have been streamlined to avoid hard failures and provide clearer instructions.[ok-commit] commit effdffe786d75980e7e4253b1690b7b8134ff0be Author: Szowesgad Date: Mon Jan 26 17:39:07 2026 +0100 Support both weights.safetensors and model.safetensors Updated model loading logic to accept either weights.safetensors or model.safetensors files for Whisper models. Refactored cache lookup and model path resolution to handle both file names, improving compatibility with different model distributions.[ok-commit] commit def6c72616eb10137caee451afdc7aec61a8eb87 Author: Szowesgad Date: Mon Jan 26 17:23:52 2026 +0100 Add case-insensitive fallback for HF cache lookup Update `find_hf_snapshot` and `find_snapshot` to handle case-insensitive repository matches when locating Hugging Face cache directories. Also, adjust `download-csm.sh` to use multiple --include flags for clarity and compatibility.[ok-commit] commit c9ba7bfd16edcf890eec12d22f741690a4c7bfa1 Author: Szowesgad Date: Mon Jan 26 17:12:17 2026 +0100 Trim whitespace from CODESCRIBE_EMBED_MODEL env var Updated the build script to trim whitespace from the CODESCRIBE_EMBED_MODEL environment variable before checking if it is empty. This ensures that values with only whitespace are treated as empty.[ok-commit] commit 2658fc0c979a2fd4833fb1085771a01efaf97866 Author: Szowesgad Date: Mon Jan 26 16:47:36 2026 +0100 Switch model loading to HuggingFace cache Refactored model loading logic to use HuggingFace cache via the `hf download` CLI, replacing hardcoded local paths and improving flexibility. Added `core/hf_cache.rs` for cache utilities, updated build scripts and runtime model resolution to support repo overrides, and revised download scripts to use the HuggingFace cache directly. Updated documentation and warnings to reference `hf download` commands.[ok-commit] commit 43e6d78ad83181981dd7b79e61885b052b3b77ca Author: Szowesgad Date: Mon Jan 26 16:04:12 2026 +0100 Switch moshi dependency to crates.io version Updated core/Cargo.toml to use moshi 0.6.4 from crates.io instead of the GitHub repository. Updated Cargo.lock to reflect this change and adjusted related dependencies. [ok-commit] commit 0c7c10c7b702aa6708f273da29673f4767c98a6b Author: Szowesgad Date: Mon Jan 26 15:08:44 2026 +0100 Add word-level emission and backspace support to streaming Implements word-level emission in buffered streaming mode with a configurable max words per tick (CODESCRIBE_EMIT_WORDS_MAX). Adds support for backspace (\u{0008}) deltas in both overlay and chat UI, and ensures deltas are applied with proper redaction. Updates environment examples and documentation to reflect new settings.[ok-commit] commit d10a9a8742a03da8b3f9f81c20845fb1cb17fd0f Author: Szowesgad Date: Mon Jan 26 11:27:41 2026 +0100 Switch embedder from fastembed to offline E5 model Replaces the fastembed dependency with a local/embedded E5 model using Candle and tokenizers for offline text embeddings. Adds support for embedding the E5 model directly into the binary in release builds, with new build.rs logic and a generated embedded.rs module. Updates Makefile and scripts to support E5 model download, and adjusts documentation and config to reflect the new embedding approach. Removes fastembed and related dependencies from Cargo files and codebase.[ok-commit] commit 7d5b9f8229825bba5de26debf0a344302e0dfefc Author: Szowesgad Date: Mon Jan 26 08:27:59 2026 +0100 Add streaming tag parser for demuxed LLM output Introduces a new demux module with a StreamingTagParser for parsing flat, non-nested tags such as and in streamed LLM output. Updates core/lib.rs to include the new module and switches the moshi dependency in Cargo.toml to use the Kyutai Labs GitHub repository. [ok-commit] # Conflicts: # core/Cargo.toml commit 2ee2b107b315b1efe211adf9301fdd2b82a8b70d Author: Szowesgad Date: Mon Jan 26 09:23:54 2026 +0100 Switch moshi to git dependency and update configs Replaced the local path dependency for moshi with a git-based dependency for better portability. Updated .gitignore patterns for logs and codescribe memory files. Improved comments and logic in config loader to set history and debug logging as default-on but still overridable by environment variables. Translated comments in helpers.rs from Polish to English. commit b9e7e98f5fefc61eccf177d0fab271c0dd48534d Author: Szowesgad Date: Mon Jan 26 08:06:50 2026 +0100 Set formatting-specific LLM env vars in e2e test Added LLM_FORMATTING_ENDPOINT, LLM_FORMATTING_MODEL, and LLM_FORMATTING_API_KEY environment variables in the e2e_retry_responses test to ensure both generic and formatting-specific configurations are covered. commit c546d5da3c438d99c66c1878b431a304b397ec73 Author: Szowesgad Date: Mon Jan 26 07:55:10 2026 +0100 Refactor formatting and improve code consistency This commit applies consistent formatting across multiple modules, including improved indentation, line breaks, and grouping of imports and re-exports. It also updates the 'ort' dependency feature in core/Cargo.toml and makes minor logic and style improvements in tests and core modules for better readability and maintainability. commit cecfbcd6dcbdd528d051a74421ba77b5d6fae479 Author: Szowesgad Date: Mon Jan 26 06:32:56 2026 +0100 Fix clippy warnings for strict -D warnings compliance - Collapse nested if-let statements (clippy::collapsible_if) - Remove useless .into() on anyhow::Error (clippy::useless_conversion) - Use let-chains for cleaner conditional matching Co-Authored-By: Claude Opus 4.5 commit 051c99e34d9e01c343a3b7366aff9b0212572bc3 Author: Szowesgad Date: Mon Jan 26 05:50:28 2026 +0100 Add session generation counter to RecordingController Introduces a conversation_generation AtomicU64 to RecordingController to track session generations and prevent cross-session race conditions. The audio loop and playback UI updates now check the session generation to ensure only the current session can update the UI, improving reliability when multiple sessions are started in quick succession. commit 96c5a86578b93e25b1ae572dd9c308bd76277b75 Author: Szowesgad Date: Mon Jan 26 05:35:24 2026 +0100 Refine conversation mode UI state and audio handling Standardizes conversation state status messages, adds explicit UI state updates, and improves audio playback to be non-blocking for full-duplex operation. Also ensures proper cleanup and state reset on errors and when stopping conversation mode, and updates drawer entry mode detection for conversation logs. commit cc54f393714273bf3557c2a982556af2c2a7f974 Author: Szowesgad Date: Mon Jan 26 05:26:26 2026 +0100 Add Moshi full-duplex conversation mode Implements a new full-duplex conversation mode using the Moshi engine, allowing simultaneous microphone input and AI response playback. Adds state management, hotkey handling, and UI updates for conversation mode, including a resampler for arbitrary input rates. Updates core conversation context to track total speaking duration and exposes new state enums and helpers throughout the app. commit db848a926ba1f5920fcf4dd8a2150f2ea6308279 Author: Szowesgad Date: Mon Jan 26 04:54:51 2026 +0100 Integrate Moshi LM and Mimi codec into conversation engine This update loads Moshi LM and Mimi codec models in ConversationEngine, implements audio encode/decode, and runs full inference for user audio input. Configuration paths and parameters are updated for new model formats. Adds E2E tests for initialization, audio processing, and state reset to validate the full conversational pipeline. commit 74f77d8419c6e4da3d81a05e9c1998b06427702e Author: Szowesgad Date: Mon Jan 26 02:36:56 2026 +0100 Unify model and data paths to ~/.codescribe directory Standardized all references from .CodeScribe or project-relative paths to ~/.codescribe for models and lexicon files across code, scripts, and documentation. This change ensures consistent model/data location for easier setup and usage. commit 55e89b26881047fe9447e0e47809770ea47744b7 Author: Szowesgad Date: Mon Jan 26 01:58:41 2026 +0100 Standardize model paths to ~/.codescribe/models Updated scripts and Rust code to store all model files under ~/.codescribe/models for consistency. Added download-silero.sh to automate Silero VAD model retrieval. Adjusted default_model_path in silero_ort.rs and updated related scripts to use the new location. commit ac69553521d4bea4faa2bc46bc06168c7ad04842 Author: Szowesgad Date: Mon Jan 26 01:44:03 2026 +0100 Remove deprecated SILENCE_* config and update VAD tests Eliminates legacy SILENCE_DB and SILENCE_HANG_SEC configuration from code, environment, and documentation. Updates VAD-related tests to use the new speech probability threshold and hang_sec parameters, and adds a comprehensive end-to-end VAD flow test suite. # Conflicts: # tests/e2e_vad_auto_stop.rs # tests/e2e_vad_flow.rs commit 991ace5d6868ae5a26b9a3a7bbbe78bb34f9d5c1 Author: Szowesgad Date: Mon Jan 26 01:31:52 2026 +0100 Migrate silence detection to Silero VAD config vars Replaces deprecated SILENCE_DB and SILENCE_HANG_SEC with CODESCRIBE_VAD_THRESHOLD and CODESCRIBE_VAD_MAX_SILENCE_SEC throughout the codebase and documentation. Updates default values, environment variable handling, and comments to reflect the new Silero VAD-based configuration. Improves VAD initialization logic for thread safety and reliability. Updates documentation to clarify the deprecation and new configuration approach.  Conflicts:  .env.debug.example  core/config/default_env.txt  core/config/loader.rs commit 5b504656511f025c8cc15c631214396158ec5faa Author: Szowesgad Date: Mon Jan 26 01:22:41 2026 +0100 Refactor VAD config and initialization for Silero Updated VAD environment variables and documentation to use Silero-specific parameters, replacing legacy options. Added clamping for VAD config values to ensure valid ranges. Improved VAD worker initialization to confirm model loading with a timeout and propagate errors. Updated code to initialize VAD with the provided config instead of defaults. commit a797865d5fbd2206d44da667f0e6d5b65563d958 Author: Szowesgad Date: Mon Jan 26 01:10:33 2026 +0100 Centralize VAD config and update usage across modules Moved VAD configuration to a centralized VadConfig in core/vad/config.rs. Updated streaming_recorder.rs to use the new VadConfig, removed the local VADConfig struct, and adjusted related logic and tests. Updated default_env.txt to reflect new VAD environment variable names and structure. Improved error handling for VAD initialization in recorder.rs. Removed unused import in silero_ort.rs. commit 9a3a8111e3cf413e90e06f3b9bf2b5786bea519a Author: Szowesgad Date: Mon Jan 26 01:05:01 2026 +0100 Refactor Silero VAD to use non-blocking worker and update deps Refactors the Silero VAD implementation to use a non-blocking, fire-and-forget worker thread with atomic probability updates, improving audio callback safety and performance. Updates ndarray-npy to 0.10 with default features disabled to remove unnecessary zip dependency, and removes zip and zopfli from dependencies. Adds max_utterance_sec to VadConfig. Cleans up unused silence_db config propagation in controller and improves resampling logic. commit 029033b5b796688d27a9fec04dc41ef81b4a062f Author: Szowesgad Date: Mon Jan 26 00:49:27 2026 +0100 Add Moshi conversation and E5 embedder modules Introduces a new conversation module with Moshi full-duplex voice AI, including configuration, context, engine, and turn-taking management. Adds an embedder module using fastembed for E5 text embeddings, with singleton/global access. Replaces WebRTC VAD with Silero VAD (ORT backend), updates dependencies (adds moshi, ort, fastembed, removes webrtc-vad), and refactors audio recorder to use VAD-based speech detection. Removes deprecated RMS-based silence detection and updates related code and tests. commit fecce504176b90dd21496c436a0c1236eabeaad2 Author: Szowesgad Date: Mon Jan 26 00:19:52 2026 +0100 Add TTS and VAD modules with embedded model support Introduces a new text-to-speech (TTS) module using the CSM-1B model with support for embedded model data, audio playback, and WAV export. Adds a neural voice activity detection (VAD) module using WebRTC VAD, replacing the previous RMS-based detection. Updates the streaming recorder to use the new VAD, and exposes public APIs for TTS and VAD. Updates build.rs to support embedding TTS models, and documents the new voice stack. Adds new dependencies: webrtc-vad. # Conflicts: # core/Cargo.toml # core/audio/streaming_recorder.rs # core/lib.rs # core/tts/audio_output.rs # core/tts/engine.rs # core/tts/mod.rs # core/tts/singleton.rs # core/vad/config.rs # core/vad/mod.rs # scripts/download-csm.sh commit 9bec9bc5bea154460191808c2e421501718f4343 Author: Szowesgad Date: Sun Jan 25 22:32:28 2026 +0100 Remove legacy backend config and update cloud STT handling Eliminates legacy backend discovery, deprecated config fields, and AI provider selection. Cloud STT and LLM endpoints are now explicit and canonical in .env, with migration logic for legacy keys. Quality daemon state now tracks availability, and tray/menu UI reflects this. Removes unused Node client and related scripts. Refactors AI formatting to use a unified interface and removes legacy formatting code from CLI.[ok-commit] commit e8ba3ca476384b24715b5711c49696850184525f Author: Szowesgad Date: Tue Jan 20 15:51:43 2026 +0100 Restructure project layout and update module paths Moved source files from 'src' and 'codescribe-core' to a new modular structure under 'app' and 'core' directories. Updated Cargo.toml and module imports to reflect the new paths, introduced namespacing for core modules, and added new mod.rs files for better organization. This refactor improves codebase maintainability and clarity.[ok-commit] Refactor voice chat overlay to support chat log and sending Reworked the voice chat overlay to maintain a chat log with user, assistant, and error messages. Added input field, send button, and support for streaming assistant responses. Introduced send callback mechanism and improved UI state management for sending, draft text, and error handling. Updated related controller logic to use new chat overlay API. Add persistent hands-off chat overlay with auto-send toggle Introduces a full-featured chat overlay for hands-off mode, including persistent chat history, user/assistant roles, and an input field with an auto-send toggle. The overlay UI is improved with reversed message flow (newest first), selectable text, and a new tray menu action to show the overlay. Voice Activity Detection (VAD) timeout is set to 5 seconds for hands-off mode, and short silences are ignored. The commit also adds a 'Copy Last to Clipboard' tray action, updates documentation, and includes tests for overlay persistence and auto-send logic. Refactor voice chat UI for split view and separate drafts Introduces a split view layout with distinct left (manual input) and right (voice streaming) panels. Separates manual and voice draft buffers, updates state management, and adjusts UI logic and tests to support the new structure. Also adds placeholder for attachment handling and updates default overlay dimensions for improved chat history visibility. Add voice draft management and UI enhancements Introduces voice draft saving, listing, reading, and deletion in history.rs, with corresponding API exports. Updates the voice chat overlay UI to include a right-side panel for drafts and settings, a collapse button, tab bar, and file attachment support. Also adds context menu for copying the last assistant response and refines layout logic. Removes obsolete files layout_patch.rs and test.txt. Add transcription overlay for non-assistive modes Introduces a new macOS transcription overlay (transcription_overlay.rs) for non-assistive recording modes, featuring a minimal floating window with live transcription, status display, and auto-hide. Refactors controller logic to route transcription deltas and overlay display based on session mode, and updates public API in lib.rs. Also updates dependencies, version to 0.7.3, and README formatting. Refactor UI code to use new ui_helpers module Introduced a new ui_helpers.rs module to encapsulate common Objective-C UI patterns and reduce msg_send! boilerplate. Updated transcription_overlay.rs, ui.rs, and voice_chat_ui.rs to use these helpers for window, view, color, and control management, improving code readability and maintainability. Add chat bubbles and drafts panel to voice chat UI Replaces the single text view with a bubble-based chat rendering using NSStackView, including support for streaming updates and per-message copy buttons. Adds a drafts panel in the right sidecar with a scrollable list of draft files, Edit/Copy actions, and animated drawer collapse/expand. Refactors UI state and helper functions to support these features. Always show transcription overlay and add transfer button Transcription overlay is now always shown for live preview, regardless of session mode, and the chat overlay is only opened intentionally by the user. Added a "Do chatu" button to the transcription overlay to allow transferring the current transcription to the voice chat overlay. Improved overlay state management and window delegate handling in voice chat UI to ensure proper cleanup and prevent use-after-free. Extract controller helpers into new module Splits various utility functions such as session state management, transcription delta routing, and voice chat callbacks into a new `helpers.rs` module. Cleans up `controller.rs` by delegating logic to `helpers` and `types`. Adds new unit tests in `tests.rs` to validate controller behavior. Enable always-on history and audio logging, simplify history handling - Default `history_enabled` and `dump_audio_logs` to `true` to ensure data preservation. - Removed toggles for history saving, audio dumping, and related tray menu options. - Simplified history submenu by unifying options into "Open history folder." - Refactored transcription configuration and helpers to enforce mandatory raw transcript saving. - Removed redundant handlers and menu construction logic for history and audio toggle options. - Addressed formatting consistency in `transcription_overlay.rs` for better readability and maintainability. Remove unused imports, redundant attributes, and improve safety in Objective-C calls - Cleaned up unnecessary `#[allow(unused_imports)]` and `#![allow(dead_code)]` attributes across multiple modules for better code hygiene. - Refactored Objective-C bindings to explicitly mark unsafe functions with comments, clarifying requirements and ensuring proper usage. - Renamed constants to SCREAMING_SNAKE_CASE for consistency with Rust coding standards. - Updated transcription logic to include configurable silence detection thresholds (`silence_db` and `silence_hang_sec`) through environment variables. - Enhanced Voice Activity Detection (VAD) session handling by resetting flags and adding custom callbacks in recorder configuration. - Improved Objective-C interop safety by wrapping `msg_send!` calls in `unsafe` blocks where applicable. - Simplified UI helpers and internals with clear separation of Objective-C safety boundaries in `ui_helpers.rs`. - Removed legacy and unused functions, including obsolete chat log rendering helpers. feat: “Streaming improvements with env vars” [ok-commit] [ok-commit] feat(cli): add streaming transcribe + live mode [ok-commit] chore: add codescribe repo runner [ok-commit] chore: support codex runner [ok-commit] chore: use repo mcp config chore: bump codescribe to version 0.7.5 - Updated `Cargo.toml` and `Cargo.lock` to reflect version change from `0.7.4-dev` to `0.7.5`. - Prepares for release with stable versioning for both `codescribe` and `codescribe-core`. - No functional changes introduced, focusing solely on version progression.[ok-commit] chore: remove unused dependencies from project - Cleaned up `Cargo.toml` and `Cargo.lock` by removing unused dependencies like `cpal`, `symphonia`, `candle-core`, `tokenizers`, `indicatif`, and others. [ok-commit] - Reduced dependency footprint for better maintenance and smaller build artifacts. - Ensured no functional changes to the project while simplifying dependency configuration. chore: cleanup and refactor project structure - Updated `.gitignore` to exclude CodeScribe runtime state files (`.codescribe/state/`). - Replaced the default model in `runner.sh` with `claude-opus-4-5-20251101` for consistency with current AI configurations. - Improved markdown formatting in `BACKLOG.md` and related documentation for better readability and alignment. - Expanded chip compatibility requirements in the guide to include Apple Silicon M5 (`README.md`). [ok-commit] - Removed deprecated Python backend management code (`backend.rs`) to streamline the codebase and reduce technical debt. Increase silence hang threshold and improve transcript post-processing Raised the default silence hang duration from 0.8s to 1.5s in environment, config, and audio recorder. Refactored transcript handling to always apply post-processing (lexicon, cleanup, semantic gate) before output, ensuring all output paths receive clean text. Updated embedder model references from BGEM3 to ParaphraseMLMiniLM. Removed unused files: examples/inspect_keys.rs, src/launchd.rs, and src/libraxis.rs. Added a background quality daemon process to main.rs for continuous self-improvement. [ok-commit] Adjust transcription overlay height with tailing Add in-app draft editing to voice overlay Refactor decision mode logic and add tests Replaces decision mode branching with unconditional state reset after recording finishes, ensuring paste works in all modes. Adds hot-reload support for custom lexicon rules, refactors daemon argument construction for testability, and improves UI event handling with additional safety. Includes comprehensive regression and contract tests for controller and lexicon logic. [ok-commit] Add commit recording button to transcription overlay (macOS) Introduces a 'Zakończ' (commit) button to the transcription overlay UI on macOS, allowing users to stop the current recording and enter decision mode immediately. Adds controller registration and commit request logic, and updates overlay state management to handle the new button's visibility and actions.[ok-commit] Enable test mode for audio-free controller tests Controller logic now skips backend health checks and audio device access when running in test mode, allowing tests to run without audio hardware. Removed #[ignore] attributes from controller tests, and added test-mode guards to voice draft finalization to prevent deadlocks. Added comprehensive tests to quality_loop.rs and introduced a detailed architecture vision document for the quality loop. [ok-commit] Bump version to 0.7.6 Update codescribe and codescribe-core package versions from 0.7.5 to 0.7.6 in Cargo.toml and Cargo.lock for a new release. [ok-commit] Remove completed stream chunk output inventory and task docs Deleted documentation files related to streaming chunk output inventory and implementation tasks for CodeScribe CLI transcription. These files contained requirements, context, and recommendations for improving streaming output in the CLI already implemented. [ok-commit] Refactor transcription to use single-pass engine API Replaces streaming chunked transcription with a single-pass engine call in quality report and IPC server, simplifying the flow and ensuring consistent chunking logic. Adds detailed logging and diagnostics to streaming_recorder, improves overlap deduplication with fuzzy matching in whisper engine, and updates .gitignore for new memory files. Also adds resume logic to quality report to skip already-processed pairs.[ok-commit] Refactor endpoint routing and streaming logic for LLM APIs Replaces domain-based endpoint detection with path-based routing using a new EndpointFormat enum. Streaming is now controlled by the LLM_USE_STREAMING environment variable, defaulting to enabled. Refactors Ollama and Responses API handling for clarity and reliability, and simplifies the API key check logic. Removes unused semgrep_pro_20260119.log file. [ok-commit] Add buffered streaming transcription with VAD Introduces a buffered transcription worker using VAD-based utterance segmentation and a typing-speed emitter for smoother streaming output. Adds VAD configuration, segmenter, and supporting tests. Updates StreamPostProcessor with process_utterance for VAD segments, and refactors chunk processing logic to support both buffered and unbuffered streaming modes. [ok-commit] Refactor voice chat UI and drawer, remove draft API Reworks the voice chat overlay UI to use a tabbed interface with Agent and Drawer tabs, removes the voice draft API and related code, and introduces a new card-based drawer for managing transcriptions. Updates the UI helpers with segmented control and card view helpers, and refactors state management and event handlers to support the new design. The commit also cleans up unused code and streamlines the overlay's public API.[ok-commit] Add clipboard copy and search field helpers to UI Introduces a `copy_to_clipboard` function for copying text to the system clipboard and a `create_search_field` helper for creating NSSearchField UI elements. Also updates E2E tests to set LLM_USE_STREAMING=0 for compatibility with mock server responses.[ok-commit] Add exhaustive env example, docs, and env tests Introduces .env.debug.example with all supported environment variables and safe defaults, a comprehensive Polish documentation (docs/env.md) detailing env usage, precedence, and requirements, and new Rust tests for validating required env vars and precedence logic in codescribe-core.[ok-commit] Clamp overlay window position to visible frame Introduces clamp_overlay_position to ensure overlay windows stay within the visible screen area, updating both transcription and voice chat overlays to use it. Also adds tests for the clamping logic and ensures overlays are brought to the front when shown. Documentation is updated to clarify which environment variables require restart or support hot reloading.[ok-commit] Refactor voice chat UI API and handlers Replaces send_agent_message_impl with send_draft_message_impl for improved draft handling. Adds new API functions: clear_voice_chat_text, filter_drawer, is_auto_send_enabled, and send_voice_chat_draft. Updates handler and API usage to use new function names and logic, and removes update_drawer_filter. Improves clarity and separation of concerns in voice chat overlay message sending and drawer filtering. [ok-commit] Add commit/discard for chat drafts and improve test config Introduces commit/discard action bar for draft user messages in the voice chat overlay, including new handler methods and UI updates. Refactors Makefile to load environment variables for tests and adds support for local LLM endpoints via TEST_USE_LOCAL_LLM. Updates .env.example and docs/env.md for simplified and clarified configuration. Reduces transcription overlay auto-hide delay to 5 seconds and prevents auto-hide when hovered. Updates tests and environment validation to support new behaviors. Bumps version to 0.7.7. [ok-commit] Add stream overlap ratio config and update defaults Introduces the CODESCRIBE_STREAM_OVERLAP_RATIO environment variable for finer control of audio chunk overlap in streaming mode. Updates default environment files and documentation to reflect new and revised configuration options, sets buffered streaming as default, and changes the default Ollama model. Also adds logic to enter decision mode after non-assistive recordings and improves .gitignore. [ok-commit] * Add CODESCRIBE_LOOP_USE_CLOUD_STT env override and docs Introduces the CODESCRIBE_LOOP_USE_CLOUD_STT environment variable to force cloud STT usage in the quality loop without changing app-wide defaults. Updates documentation and environment registry to reflect the new variable, and adds an end-to-end test to validate .env.example against the registry. Also corrects the default for CODESCRIBE_BUFFERED_STREAM in docs. [ok-commit] * Remove legacy VAD auto-stop and related config/flags Eliminates the CODESCRIBE_VAD_ENABLED flag and all code paths, callbacks, and documentation related to VAD-based auto-stop. VAD now only segments utterances (flushes) instead of stopping recordings, and all references to 'auto-stop' are updated to 'utterance boundary' or 'utterance flush'. Removes the e2e_vad_auto_stop.rs test, updates Makefile, config, and documentation to reflect the new segmentation-only behavior.[ok-commit] * Add Silero VAD embedding and refactor streaming VAD logic This commit introduces embedded Silero VAD model support for release builds, refactors the streaming and buffered transcription pipeline to use a unified VAD gating mechanism, and separates lexicon postprocessing for buffered mode. It updates build.rs to embed the VAD model, adds core/vad/embedded.rs, and modifies the VAD loader to prefer embedded bytes. Tests and configuration are updated accordingly.[ok-commit] * Refactor VAD gate with supervisor mode and config options Introduces a new VAD gate architecture with Supervisor mode, supporting pre-roll and speech padding for improved speech boundary detection. Adds `CODESCRIBE_VAD_GATE_MODE` and `CODESCRIBE_VAD_ITER` environment variables for gate [ok-commit]mode selection, updates documentation and environment examples, and refactors the streaming recorder to use the new gate configuration. Legacy VAD tuning variables are now ignored by the live gate, and documentation reflects these changes. * Improve Silero VAD v5 integration and add tests Refactors Silero VAD wrapper to support v5 model API, including unified state tensor, context window, and updated input/output handling. Adds detailed tests for VAD index synchronization and real audio segmentation in streaming_recorder.rs. Enhances logging and clarifies [ok-commit] Supervisor gate mode behavior. * Improve VAD segmentation test for real audio Enhanced the VAD supervisor test to process multiple WAV files, including hardcoded edge cases and recent files. The test now reports segmentation quality, compares speech-only duration to total audio, and checks for hallucination markers in old transcripts. Improved output formatting and added assertions for speech detection. [ok-commit] * Update copyright notice in documentation Replaced the footer in all documentation files from 'Created by M&K (c)2026 VetCoders' to 'Copyright © 2024–2026 VetCoders' for consistency and accuracy. [ok-commit] * Add DoS protection to demux parser and update Whisper params Introduces a maximum buffer size in the demux parser to prevent unbounded memory growth and potential DoS attacks. Updates Whisper decoding parameters to use a 5-gram repetition block for improved quality and expands documentation for the WHISPER_INITIAL_PROMPT environment variable with usage guidance and examples. * Refactor voice chat UX, assistive context, and VAD pipeline (#15) * Improve dedup overlap handling and add sink convenience Enhances dedup overlap logic in `strip_suffix_overlap` to use char boundaries, preventing panics with multi-byte UTF-8 (e.g., diacritics, emoji). Adds tests for these cases. In `chunker.rs`, gate-level pre_roll and speech_pad are now derived from VAD config and environment variables. Adds a `from_callback` convenience constructor to `sinks.rs` for easier external use, with corresponding test.[ok-commit] Refactor pipeline deduplication and delta sink handling “Virtual Commit: 5, 6, 7, 8”: Moves chunk and suffix deduplication logic into core/pipeline/dedup.rs and updates the streaming pipeline to use these unified helpers. Introduces DeltaSink trait and concrete sinks (CallbackSink, CollectorSink) in core/pipeline/sinks.rs, replacing direct Fn(&str) callbacks throughout the codebase. Updates all relevant controller and CLI code to use the new sink adapters. Adds integration and unit tests for deduplication and sink behavior. [ok-commit] Improve streaming VAD config and correction handling Adds a safety net to cap raw_buffer growth in chunker during silence, ensures VAD config respects environment overrides, and always applies postprocessing in streaming_recorder. In streaming, corrections are now guarded to prevent rewriting most of the text, and [ok-commit] last_suffix handling is improved to avoid state corruption when postprocessing filters out hallucinations. Includes new tests for suffix and correction guard logic. * Document CODESCRIBE_VAD_SPEECH_PAD_SEC env variable Added documentation for the new CODESCRIBE_VAD_SPEECH_PAD_SEC environment variable in ENV_REGISTRY.toml and env.md, describing its purpose as speech padding after speech offset and its default behavior. Also updated test module in chunker.rs to use serial_test for relevant tests.[ok-commit] * [ok-commit] feat(ui): bubble overlap fixes + multiline input (ranch Fala 1) Cherry-picked from origin/feat/ux-poc-assistive-context: - e9d63c2 fix(ui): stop overlay bubble overlap and enable scrolling - cc03eee fix(overlay): measure bubble height with NSString boundingRect - de934a3 fix(overlay): render full bubble text - acf3c4c fix(overlay): prevent Copy button overlap and bubble clipping - 2861914 fix(overlay): avoid narrow bubbles for long responses - ef9237c fix(overlay): widen streaming bubbles early - a08d944 fix(overlay): widen streaming bubbles sooner - 71cc1ad feat(overlay): multiline agent input Co-Authored-By: Klaudiusz * [ok-commit] feat(ui): overlay stability fixes (ranch Fala 2) Cherry-picked from origin/feat/ux-poc-assistive-context: - aace207 fix(overlay): prevent header controls overlap - 7ccdec5 fix(overlay): keep tabs from overlapping header icons - 0828ea3 fix(ui): keep agent chat content visible - 72ac85f fix(ui): make overlay interactive and drawer usable - bd6eab1 fix(ui): force chat overlay visible for responses - 32364ec fix(ui): stabilize voice chat overlay updates - 19c0c33 fix(overlay): avoid deadlock when toggling favorites - 6bc72e3 fix(ui): avoid deadlock when closing voice overlay Also added: set_tooltip helper function. Co-Authored-By: Klaudiusz * [ok-commit] fix(ui): re-show chat overlay on assistive output (ranch b05ad18) Ensure voice chat overlay is shown when setting user/assistant text in the controller. Applies to all assistive and formatting paths. Co-Authored-By: Klaudiusz * [ok-commit] fix(overlay): enable Cmd+C/Cmd+V in agent app (ranch Fala 3 partial) Cherry-picked from origin/feat/ux-poc-assistive-context: - 3c4e5f1 fix(overlay): enable Cmd+C/Cmd+V in agent app Skipped 3bd43dc, f848580, 0a79ce2 (depend on selection.rs from excluded 27b8b1b). Co-Authored-By: Klaudiusz * [ok-commit] feat(ui): context management + agent input (ranch Fala 4 clean) Cherry-picked from origin/feat/ux-poc-assistive-context: - 597952b fix(voice-chat): render overlay contents - 6693cb9 fix(ux): keep assistive overlay on Agent tab (excl. selection.rs) - d2eae30 fix(overlay): make agent input field editable Co-Authored-By: Klaudiusz * feat(ui): context management + copy/paste support (ranch Fala 4b) Cherry-picked UI portions from origin/feat/ux-poc-assistive-context: - 34cffcc fix(voice-chat): show context + reduce dead overlay (UI parts) - 0fd49ba feat(voice-chat): add copy/paste last response (UI parts) Added: set_voice_chat_target_app API, last_target_app state field, copy/paste handler infrastructure. Skipped controller hunks from 34cffcc, 0fd49ba, 7d58432, 1f13d8c (heavy controller restructuring conflicts with DeltaSink/CallbackSink).[ok-commit] Co-Authored-By: Klaudiusz * Add assistive context capture for selection in macOS Implements best-effort capture of selected text and frontmost app context for assistive mode on macOS, using Accessibility APIs and optional Cmd+C fallback. Updates RecordingController to store and use this context, modifies clipboard handling to support Cmd+C simulation, and adds new helpers in app/ui for retrieving selected text and its length. This enables richer context-aware AI interactions without polluting the clipboard.[ok-commit] Co-Authored-By: Klaudiusz * Improve voice chat UX, safety, and accessibility Adds accessibility labels to UI elements, improves send button logic and labeling, and caps chat message history for performance. Introduces path safety checks for file operations in the voice chat overlay to prevent access outside the transcriptions directory. Refactors locking in message send/commit flows to avoid deadlocks. Updates VAD worker for clean shutdown and improves error handling in audio and model initialization. Adds new environment variables for assistive context and logging,and bumps version to 0.7.10. [ok-commit] * Update test and formatting commands in Makefile Consolidate test commands to use --include-ignored for running all tests, including those previously marked as ignored. Restrict Prettier formatting and checking to tracked files using git ls-files to avoid processing untracked files.[ok-commit] --------- Co-authored-by: Klaudiusz --------- Co-authored-by: Klaudiusz --- .env.debug.example | 21 +- .env.example | 29 +- .gitignore | 4 + Cargo.lock | 232 +--- Cargo.toml | 9 +- Makefile | 80 +- README.md | 2 +- app/controller/helpers.rs | 15 +- app/controller/mod.rs | 279 ++-- app/lib.rs | 20 +- app/os/clipboard.rs | 18 + app/os/hotkeys.rs | 36 +- app/os/mod.rs | 2 + app/os/selection.rs | 252 ++++ app/ui/bootstrap/handlers.rs | 56 + app/ui/bootstrap/mod.rs | 421 ++++++ app/ui/mod.rs | 212 +++ app/ui/overlay/mod.rs | 16 +- app/ui/shared/helpers.rs | 439 +++++- app/ui/tray/handlers.rs | 2 + app/ui/tray/menu.rs | 6 + app/ui/tray/types.rs | 4 + app/ui/voice_chat/api.rs | 282 +++- app/ui/voice_chat/handlers.rs | 16 +- app/ui/voice_chat/mod.rs | 144 +- app/ui/voice_chat/state.rs | 23 + bin/codescribe.rs | 166 ++- bin/codescribe_loop.rs | 7 + core/Cargo.toml | 14 +- core/audio/chunker.rs | 1180 +++++++++++++++++ core/audio/mod.rs | 1 + core/audio/recorder.rs | 182 +-- core/audio/streaming_recorder.rs | 1054 +++++---------- core/build.rs | 322 ++++- core/config/default_env.txt | 17 +- core/conversation/config.rs | 52 +- core/demux/mod.rs | 3 + core/demux/parser.rs | 488 +++++++ core/embedder/embedded.rs | 78 ++ core/embedder/engine.rs | 443 +++++-- core/embedder/mod.rs | 12 +- core/hf_cache.rs | 128 ++ core/lib.rs | 9 +- core/pipeline/contracts.rs | 334 +++++ core/pipeline/dedup.rs | 241 ++++ core/pipeline/mod.rs | 6 + core/pipeline/sinks.rs | 102 ++ core/pipeline/stream_postprocess.rs | 125 +- core/pipeline/streaming.rs | 926 +++++++++++++ core/pipeline/tests.rs | 139 ++ core/quality/quality_report.rs | 2 +- core/stt/adapter.rs | 111 ++ core/stt/mod.rs | 1 + core/stt/whisper/embedded.rs | 13 +- core/stt/whisper/engine.rs | 27 +- core/stt/whisper/params.rs | 28 +- core/stt/whisper/singleton.rs | 26 +- core/tts/embedded.rs | 14 +- core/tts/engine.rs | 110 +- core/tts/singleton.rs | 18 +- core/vad/config.rs | 50 +- core/vad/embedded.rs | 45 + core/vad/mod.rs | 3 +- core/vad/silero_ort.rs | 228 ++-- docs/ADR/2026-01-16-LICENSE_NOTES.md | 23 + docs/ADR/2026-01-18-WHISPER_LIVE.md | 152 +++ docs/ADR/2026-01-19-INSTALLATION.md | 194 +++ docs/ADR/2026-01-19-TEAM_SETUP.md | 181 +++ docs/ADR/2026-01-25-ARCHITECTURE.md | 247 ++++ docs/ADR/2026-01-25-ARCHITECTURE_VISION.md | 186 +++ docs/ADR/2026-01-25-BACKLOG.md | 114 ++ docs/ADR/2026-01-25-FEASIBILITY_ANALYSIS.md | 95 ++ docs/ADR/2026-01-25-README.md | 99 ++ docs/ADR/2026-01-25-chat-overlay.md | 199 +++ docs/ADR/2026-01-25-installation.md | 133 ++ docs/ADR/2026-01-25-modes.md | 187 +++ docs/ADR/2026-01-25-quality-loop-vision.md | 565 ++++++++ docs/ADR/2026-01-26-OVERLAY_STREAMING.md | 172 +++ docs/ADR/2026-01-26-QUICKSTART.md | 142 ++ docs/ADR/2026-01-26-SPEECH_TO_SPEECH_AGENT.md | 554 ++++++++ docs/ADR/2026-01-26-VOICE_STACK_WBS.md | 311 +++++ docs/ADR/2026-01-26-env.md | 275 ++++ docs/ADR/2026-01-26-privacy.md | 280 ++++ docs/ADR/2026-01-26-settings.md | 335 +++++ docs/ADR/2026-01-26-troubleshooting.md | 301 +++++ docs/ARCHITECTURE.md | 2 +- docs/BACKLOG.md | 2 +- docs/ENV_REGISTRY.toml | 758 +++++++++++ docs/HOTKEYS_CONTRACT.md | 418 ++++++ docs/INSTALLATION.md | 2 +- docs/OVERLAY_STREAMING.md | 125 +- docs/TEAM_SETUP.md | 2 +- docs/VOICE_STACK_WBS.md | 2 +- docs/env.md | 50 +- docs/future/SPEECH_TO_SPEECH_AGENT.md | 2 +- docs/guide/QUICKSTART.md | 142 ++ docs/guide/README.md | 4 +- docs/guide/chat-overlay.md | 2 +- docs/guide/installation.md | 2 +- docs/guide/modes.md | 6 +- docs/guide/privacy.md | 2 +- docs/guide/settings.md | 91 +- docs/guide/troubleshooting.md | 2 +- docs/quality-loop-vision.md | 2 +- examples/README.md | 7 +- examples/roundtrip_live.rs | 291 ++++ packaging/appwrap/build_wrapper_app.sh | 92 +- packaging/notarize.sh | 11 +- packaging/release.sh | 2 +- packaging/release_master.sh | 23 +- packaging/scripts/sign_and_notarize.sh | 16 +- packaging/validate_setup.sh | 24 +- scripts/download-csm.sh | 46 +- scripts/download-e5.sh | 72 + scripts/download-model.sh | 51 +- scripts/download-silero-vad.sh | 45 + scripts/download-silero.sh | 4 +- scripts/notarize.sh | 10 +- scripts/validate-envs.sh | 175 +++ tests/e2e_env_registry.rs | 26 + tests/e2e_round_trip.rs | 411 ++++++ tests/e2e_vad_auto_stop.rs | 310 ----- tests/e2e_vad_flow.rs | 2 +- tests/e2e_vad_gate_live.rs | 45 + {codescribe-tools => tools}/runner.sh | 0 125 files changed, 14976 insertions(+), 2373 deletions(-) create mode 100644 app/os/selection.rs create mode 100644 app/ui/bootstrap/handlers.rs create mode 100644 app/ui/bootstrap/mod.rs create mode 100644 core/audio/chunker.rs create mode 100644 core/demux/mod.rs create mode 100644 core/demux/parser.rs create mode 100644 core/embedder/embedded.rs create mode 100644 core/hf_cache.rs create mode 100644 core/pipeline/contracts.rs create mode 100644 core/pipeline/dedup.rs create mode 100644 core/pipeline/sinks.rs create mode 100644 core/pipeline/streaming.rs create mode 100644 core/pipeline/tests.rs create mode 100644 core/stt/adapter.rs create mode 100644 core/vad/embedded.rs create mode 100644 docs/ADR/2026-01-16-LICENSE_NOTES.md create mode 100644 docs/ADR/2026-01-18-WHISPER_LIVE.md create mode 100644 docs/ADR/2026-01-19-INSTALLATION.md create mode 100644 docs/ADR/2026-01-19-TEAM_SETUP.md create mode 100644 docs/ADR/2026-01-25-ARCHITECTURE.md create mode 100644 docs/ADR/2026-01-25-ARCHITECTURE_VISION.md create mode 100644 docs/ADR/2026-01-25-BACKLOG.md create mode 100644 docs/ADR/2026-01-25-FEASIBILITY_ANALYSIS.md create mode 100644 docs/ADR/2026-01-25-README.md create mode 100644 docs/ADR/2026-01-25-chat-overlay.md create mode 100644 docs/ADR/2026-01-25-installation.md create mode 100644 docs/ADR/2026-01-25-modes.md create mode 100644 docs/ADR/2026-01-25-quality-loop-vision.md create mode 100644 docs/ADR/2026-01-26-OVERLAY_STREAMING.md create mode 100644 docs/ADR/2026-01-26-QUICKSTART.md create mode 100644 docs/ADR/2026-01-26-SPEECH_TO_SPEECH_AGENT.md create mode 100644 docs/ADR/2026-01-26-VOICE_STACK_WBS.md create mode 100644 docs/ADR/2026-01-26-env.md create mode 100644 docs/ADR/2026-01-26-privacy.md create mode 100644 docs/ADR/2026-01-26-settings.md create mode 100644 docs/ADR/2026-01-26-troubleshooting.md create mode 100644 docs/ENV_REGISTRY.toml create mode 100644 docs/HOTKEYS_CONTRACT.md create mode 100644 docs/guide/QUICKSTART.md create mode 100644 examples/roundtrip_live.rs create mode 100755 scripts/download-e5.sh create mode 100755 scripts/download-silero-vad.sh create mode 100755 scripts/validate-envs.sh create mode 100644 tests/e2e_env_registry.rs create mode 100644 tests/e2e_round_trip.rs delete mode 100644 tests/e2e_vad_auto_stop.rs create mode 100644 tests/e2e_vad_gate_live.rs rename {codescribe-tools => tools}/runner.sh (100%) diff --git a/.env.debug.example b/.env.debug.example index 58a99ccb..5b0d7fef 100644 --- a/.env.debug.example +++ b/.env.debug.example @@ -77,7 +77,9 @@ AI_FORMATTING_ENABLED=1 # AI_PROVIDER=harmony # legacy (harmony|ollama) # AI_MAX_TOKENS=0 # 0 = no limit # AI_ASSISTIVE_MAX_TOKENS=0 # 0 = no limit -TRANSCRIPT_SEND_MODE=end_of_utterance # end_of_utterance | streaming + +# end_of_utterance | streaming: +TRANSCRIPT_SEND_MODE=end_of_utterance # Retry policy for AI calls # CODESCRIBE_AI_MAX_RETRIES=2 @@ -94,11 +96,13 @@ CODESCRIBE_BUFFER_DELAY_MS=3000 # Delay before emitting buffered text CODESCRIBE_TYPING_CPS=30 # Emission speed in buffered mode # VAD for buffered mode (Silero neural network) -CODESCRIBE_VAD_THRESHOLD=0.5 +# See docs/ENV_REGISTRY.toml for all options +CODESCRIBE_VAD_ENABLED=1 +CODESCRIBE_VAD_THRESHOLD=0.35 +CODESCRIBE_VAD_SILENCE_SEC=2.5 CODESCRIBE_VAD_MIN_SPEECH_SEC=0.1 -CODESCRIBE_VAD_MAX_SILENCE_SEC=1.2 CODESCRIBE_VAD_MAX_UTTERANCE_SEC=60 -CODESCRIBE_VAD_PRE_ROLL_SEC=0.3 +CODESCRIBE_VAD_PRE_ROLL_SEC=0.5 # ============================================================================= # STREAM POSTPROCESS (REPETITION / GATING) @@ -117,7 +121,7 @@ CODESCRIBE_VAD_PRE_ROLL_SEC=0.3 # AUDIO INPUT # ============================================================================= # AUDIO_INPUT_DEVICE=MacBook Pro Microphone -AUTO_SILENCE=0 +# NOTE: To disable VAD for debug, change CODESCRIBE_VAD_ENABLED=1 above to =0 # ============================================================================= # HOTKEYS @@ -131,7 +135,7 @@ HOLD_START_DELAY_MS=800 # UI / OVERLAYS / FEEDBACK # ============================================================================= SHOW_TRAY_GLYPH=1 -HOLD_INDICATOR=1 # bool (legacy string values ignored) +HOLD_INDICATOR=1# bool (legacy string values ignored) HOLD_BADGE_SIZE=12 HOLD_BADGE_OFFSET_X=10 HOLD_BADGE_OFFSET_Y=-10 @@ -161,11 +165,6 @@ DUMP_AUDIO_LOGS=1 # QUALITY_DEBUG_MODE=1 # CODESCRIBE_QUALITY_DISABLE_CLOUD=1 -# ============================================================================= -# VOICE CHAT BACKEND (LOCAL DEV) -# ============================================================================= -# CODESCRIBE_BACKEND_URL=http://127.0.0.1:8237 - # ============================================================================= # DEBUG / TEST FLAGS (E2E) # ============================================================================= diff --git a/.env.example b/.env.example index 6387f33d..204caf8c 100644 --- a/.env.example +++ b/.env.example @@ -1,13 +1,14 @@ # CodeScribe .env (minimal + recommended) -# Source of truth: docs/env.md +# Ground truth for E2E defaults (validated by scripts/validate-envs.sh) # Copy: cp .env.example ~/.codescribe/.env # ============================================================================= # LOCAL EMBEDDED (RECOMMENDED) # ============================================================================= USE_LOCAL_STT=1 -WHISPER_LANGUAGE=pl +# WHISPER_LANGUAGE=pl # leave unset for autodetect LOCAL_MODEL=whisper-large-v3-turbo-mlx-q8 +# WHISPER_INITIAL_PROMPT=CodeScribe, LibraxisAI, Docker, GitHub # ============================================================================= # AI FORMATTING (OPTIONAL) @@ -33,11 +34,33 @@ LOCAL_MODEL=whisper-large-v3-turbo-mlx-q8 # USE_LOCAL_STT=0 # STT_ENDPOINT=https://api.libraxis.cloud/v1/audio/transcriptions # STT_API_KEY=... +# Loop-only: force cloud STT for quality loop without changing app defaults +# CODESCRIBE_LOOP_USE_CLOUD_STT=1 + +# ============================================================================= +# VAD gate (Silero) +# ============================================================================= +# CODESCRIBE_VAD_GATE_MODE=iter # simple | iter (Silero vad_iter w/ temp_end/prev_end) +# CODESCRIBE_VAD_ITER=1 # shortcut for iter mode +# +# NOTE: Live gate uses hardcoded SoTA params (pre-roll + speech_pad). +# CODESCRIBE_VAD_* tuning is legacy/offline and ignored by live gate. # ============================================================================= # STREAMING TUNING (OPTIONAL) # ============================================================================= -# CODESCRIBE_STREAM_CHUNK_SEC=12 +# CODESCRIBE_STREAM_CHUNK_SEC=4 # CODESCRIBE_STREAM_OVERLAP_RATIO=0.25 # CODESCRIBE_STREAM_SIMILARITY=0.90 # CODESCRIBE_STREAM_NOVELTY=0.20 + +# ============================================================================= +# ASSISTIVE CONTEXT CAPTURE (OPTIONAL) +# ============================================================================= +# ASSISTIVE_CONTEXT_MAX_CHARS=5000 +# ASSISTIVE_CONTEXT_COPY_DELAY_MS=150 + +# ============================================================================= +# LOGGING (OPTIONAL) +# ============================================================================= +# CODESCRIBE_LOG_STDOUT=0 diff --git a/.gitignore b/.gitignore index 8a1e7ade..273fd1c9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,7 @@ __pycache__/ *.pyc .venv/ .env +.env.e2e .pids/ .ruff_cache/ .mypy_cache/ @@ -76,3 +77,6 @@ docs/runtime-diagnosis.md /logs/*.log *.debug.log /core/models +/tmp/*.log +codescribe.log +*.log diff --git a/Cargo.lock b/Cargo.lock index c1486f53..a2ce9162 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -69,12 +69,6 @@ dependencies = [ "equator", ] -[[package]] -name = "allocator-api2" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" - [[package]] name = "alsa" version = "0.10.0" @@ -539,7 +533,7 @@ dependencies = [ "rand 0.9.2", "rand_distr", "rayon", - "safetensors 0.4.5", + "safetensors", "thiserror 1.0.69", "ug", "ug-metal", @@ -572,7 +566,7 @@ dependencies = [ "metal 0.27.0", "num-traits", "rayon", - "safetensors 0.4.5", + "safetensors", "serde", "thiserror 1.0.69", ] @@ -759,7 +753,7 @@ dependencies = [ [[package]] name = "codescribe" -version = "0.7.8" +version = "0.7.10" dependencies = [ "anyhow", "arboard", @@ -789,6 +783,7 @@ dependencies = [ "tokio", "tokio-tungstenite", "tracing", + "tracing-appender", "tracing-subscriber", "tray-icon", "uuid", @@ -796,7 +791,7 @@ dependencies = [ [[package]] name = "codescribe-core" -version = "0.7.8" +version = "0.7.10" dependencies = [ "anyhow", "base64 0.22.1", @@ -810,11 +805,10 @@ dependencies = [ "directories", "dirs", "dotenvy", - "fastembed", "flate2", "futures-util", "hound", - "indicatif 0.18.3", + "indicatif", "lazy_static", "mockito", "moshi", @@ -885,19 +879,6 @@ dependencies = [ "static_assertions", ] -[[package]] -name = "console" -version = "0.15.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "054ccb5b10f9f2cbf51eb355ca1d05c2d279ce1804688d0db74b4733a5aeafd8" -dependencies = [ - "encode_unicode", - "libc", - "once_cell", - "unicode-width", - "windows-sys 0.59.0", -] - [[package]] name = "console" version = "0.16.2" @@ -1540,22 +1521,6 @@ dependencies = [ "regex-syntax", ] -[[package]] -name = "fastembed" -version = "5.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59a3f841f27a44bcc32214f8df75cc9b6cea55dbbebbfe546735690eab5bb2d2" -dependencies = [ - "anyhow", - "hf-hub", - "ndarray 0.17.2", - "ort", - "safetensors 0.7.0", - "serde", - "serde_json", - "tokenizers", -] - [[package]] name = "fastrand" version = "2.3.0" @@ -1623,12 +1588,6 @@ version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" -[[package]] -name = "foldhash" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" - [[package]] name = "foreign-types" version = "0.3.2" @@ -2359,13 +2318,6 @@ name = "hashbrown" version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" -dependencies = [ - "allocator-api2", - "equivalent", - "foldhash", - "serde", - "serde_core", -] [[package]] name = "heck" @@ -2391,27 +2343,6 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" -[[package]] -name = "hf-hub" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "629d8f3bbeda9d148036d6b0de0a3ab947abd08ce90626327fc3547a49d59d97" -dependencies = [ - "dirs", - "http", - "indicatif 0.17.11", - "libc", - "log", - "native-tls", - "rand 0.9.2", - "reqwest", - "serde", - "serde_json", - "thiserror 2.0.17", - "ureq 2.12.1", - "windows-sys 0.60.2", -] - [[package]] name = "hmac" version = "0.12.1" @@ -2515,7 +2446,7 @@ dependencies = [ "tokio", "tokio-rustls", "tower-service", - "webpki-roots 1.0.4", + "webpki-roots", ] [[package]] @@ -2755,26 +2686,13 @@ dependencies = [ "serde_core", ] -[[package]] -name = "indicatif" -version = "0.17.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "183b3088984b400f4cfac3620d5e076c84da5364016b4f49473de574b2586235" -dependencies = [ - "console 0.15.11", - "number_prefix", - "portable-atomic", - "unicode-width", - "web-time", -] - [[package]] name = "indicatif" version = "0.18.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9375e112e4b463ec1b1c6c011953545c65a30164fbab5b581df32b3abf0dcb88" dependencies = [ - "console 0.16.2", + "console", "portable-atomic", "unicode-width", "unit-prefix", @@ -3279,7 +3197,8 @@ dependencies = [ [[package]] name = "moshi" version = "0.6.4" -source = "git+https://github.com/kyutai-labs/moshi?branch=main#fa205be69cf69b1a7e8cd9f815dbb9c130f605ef" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5153be1c62a5f77bf31797931c5df6f155ad2df2f95c6f241720132046ed2259" dependencies = [ "candle-core", "candle-nn", @@ -3568,18 +3487,12 @@ version = "0.7.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff32365de1b6743cb203b710788263c44a03de03802daf96092f2da4fe6ba4d7" dependencies = [ - "proc-macro-crate 3.4.0", + "proc-macro-crate 2.0.2", "proc-macro2", "quote", "syn 2.0.111", ] -[[package]] -name = "number_prefix" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830b246a0e5f20af87141b25c173cd1b609bd7779a4617d6ec582abaf90870f3" - [[package]] name = "objc" version = "0.2.7" @@ -3928,7 +3841,7 @@ dependencies = [ "ort-sys", "smallvec", "tracing", - "ureq 3.1.4", + "ureq", ] [[package]] @@ -3939,7 +3852,7 @@ checksum = "06503bb33f294c5f1ba484011e053bfa6ae227074bdb841e9863492dc5960d4b" dependencies = [ "hmac-sha256", "lzma-rust2", - "ureq 3.1.4", + "ureq", ] [[package]] @@ -4182,19 +4095,10 @@ version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b00f26d3400549137f92511a46ac1cd8ce37cb5598a96d382381458b992a5d24" dependencies = [ - "toml_datetime 0.6.3", + "toml_datetime", "toml_edit 0.20.2", ] -[[package]] -name = "proc-macro-crate" -version = "3.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "219cb19e96be00ab2e37d6e299658a0cfa83e52429179969b0f0121b4ac46983" -dependencies = [ - "toml_edit 0.23.7", -] - [[package]] name = "proc-macro-error" version = "1.0.4" @@ -4727,7 +4631,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams", "web-sys", - "webpki-roots 1.0.4", + "webpki-roots", ] [[package]] @@ -4806,7 +4710,6 @@ version = "0.23.35" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "533f54bc6a7d4f647e46ad909549eda97bf5afc1585190ef692b4286b198bd8f" dependencies = [ - "log", "once_cell", "ring", "rustls-pki-types", @@ -4858,17 +4761,6 @@ dependencies = [ "serde_json", ] -[[package]] -name = "safetensors" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "675656c1eabb620b921efea4f9199f97fc86e36dd6ffd1fbbe48d0f59a4987f5" -dependencies = [ - "hashbrown 0.16.1", - "serde", - "serde_json", -] - [[package]] name = "same-file" version = "1.0.6" @@ -5799,7 +5691,7 @@ dependencies = [ "derive_builder", "esaxx-rs", "getrandom 0.3.4", - "indicatif 0.18.3", + "indicatif", "itertools", "log", "macro_rules_attribute", @@ -5903,7 +5795,7 @@ checksum = "185d8ab0dfbb35cf1399a6344d8484209c088f75f8f68230da55d48d95d43e3d" dependencies = [ "serde", "serde_spanned", - "toml_datetime 0.6.3", + "toml_datetime", "toml_edit 0.20.2", ] @@ -5916,15 +5808,6 @@ dependencies = [ "serde", ] -[[package]] -name = "toml_datetime" -version = "0.7.5+spec-1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347" -dependencies = [ - "serde_core", -] - [[package]] name = "toml_edit" version = "0.19.15" @@ -5932,8 +5815,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b5bb770da30e5cbfde35a2d7b9b8a2c4b8ef89548a7a6aeab5c9a576e3e7421" dependencies = [ "indexmap 2.12.1", - "toml_datetime 0.6.3", - "winnow 0.5.40", + "toml_datetime", + "winnow", ] [[package]] @@ -5945,29 +5828,8 @@ dependencies = [ "indexmap 2.12.1", "serde", "serde_spanned", - "toml_datetime 0.6.3", - "winnow 0.5.40", -] - -[[package]] -name = "toml_edit" -version = "0.23.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6485ef6d0d9b5d0ec17244ff7eb05310113c3f316f2d14200d4de56b3cb98f8d" -dependencies = [ - "indexmap 2.12.1", - "toml_datetime 0.7.5+spec-1.1.0", - "toml_parser", - "winnow 0.7.14", -] - -[[package]] -name = "toml_parser" -version = "1.0.6+spec-1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a3198b4b0a8e11f09dd03e133c0280504d0801269e9afa46362ffde1cbeebf44" -dependencies = [ - "winnow 0.7.14", + "toml_datetime", + "winnow", ] [[package]] @@ -6026,6 +5888,18 @@ dependencies = [ "tracing-core", ] +[[package]] +name = "tracing-appender" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "786d480bce6247ab75f005b14ae1624ad978d3029d9113f0a22fa1ac773faeaf" +dependencies = [ + "crossbeam-channel", + "thiserror 2.0.17", + "time", + "tracing-subscriber", +] + [[package]] name = "tracing-attributes" version = "0.1.31" @@ -6147,7 +6021,7 @@ dependencies = [ "num-traits", "num_cpus", "rayon", - "safetensors 0.4.5", + "safetensors", "serde", "thiserror 1.0.69", "tracing", @@ -6219,26 +6093,6 @@ version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" -[[package]] -name = "ureq" -version = "2.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d" -dependencies = [ - "base64 0.22.1", - "flate2", - "log", - "native-tls", - "once_cell", - "rustls", - "rustls-pki-types", - "serde", - "serde_json", - "socks", - "url", - "webpki-roots 0.26.11", -] - [[package]] name = "ureq" version = "3.1.4" @@ -6479,15 +6333,6 @@ dependencies = [ "rustls-pki-types", ] -[[package]] -name = "webpki-roots" -version = "0.26.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" -dependencies = [ - "webpki-roots 1.0.4", -] - [[package]] name = "webpki-roots" version = "1.0.4" @@ -6942,15 +6787,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "winnow" -version = "0.7.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a5364e9d77fcdeeaa6062ced926ee3381faa2ee02d3eb83a5c27a8825540829" -dependencies = [ - "memchr", -] - [[package]] name = "winx" version = "0.36.4" diff --git a/Cargo.toml b/Cargo.toml index e20e47bf..052a48c8 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = [".", "core"] [workspace.package] -version = "0.7.8" +version = "0.7.10" edition = "2024" authors = ["VetCoders "] @@ -16,7 +16,7 @@ tracing = "0.1" [package] name = "codescribe" -version = "0.7.8" +version = "0.7.10" edition = "2024" description = "Speech-to-text tray app for macOS - Pure Rust" authors = ["VetCoders "] @@ -79,6 +79,7 @@ directories = "6" # Logging tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } +tracing-appender = "0.2" # Error handling anyhow = "1" @@ -122,8 +123,8 @@ serial_test = "3" [lints.rust] # Allow unexpected_cfgs from objc crate's msg_send! macro (uses cargo-clippy cfg internally) -# embed_model is set by build.rs when model is available -unexpected_cfgs = { level = "allow", check-cfg = ['cfg(embed_model)'] } +# embed_model/embed_tts/embed_e5 are set by build.rs when models are available +unexpected_cfgs = { level = "allow", check-cfg = ['cfg(embed_model)', 'cfg(embed_tts)', 'cfg(embed_e5)'] } [profile.release] opt-level = "z" diff --git a/Makefile b/Makefile index 073a4353..73dbf2db 100644 --- a/Makefile +++ b/Makefile @@ -6,13 +6,18 @@ bump bump-patch bump-minor bump-major version \ lint format test test-quick test-e2e test-e2e-real test-sse test-formatting test-all \ demo demo-raw demo-assistive check fix clean help \ - dmg dmg-signed dmg-full notarize download-model \ + dmg dmg-signed dmg-full notarize download-model download-e5 \ hooks SHELL := /bin/bash VERSION_FILE := Cargo.toml EDITOR ?= $(shell command -v code || command -v nvim || command -v vim || echo nano) ENV_LOAD := set -a; [ -f $$HOME/.codescribe/.env ] && source $$HOME/.codescribe/.env; set +a +E2E_ENV_EXAMPLE := .env.example +E2E_ENV_FILE := ./.env.e2e +TEST_BUILD_JOBS ?= 2 +ENV_LOAD_E2E := set -a; [ -f $(E2E_ENV_FILE) ] && source $(E2E_ENV_FILE); set +a +E2E_ENV_GEN := ./scripts/validate-envs.sh --env-example --env-example-path $(E2E_ENV_EXAMPLE) --emit-e2e-env $(E2E_ENV_FILE) # Test defaults (reference/cloud unless forced local) TEST_USE_LOCAL_LLM ?= 0 @@ -50,7 +55,7 @@ release: @cargo build --release install: - @echo "Installing CodeScribe (with embedded model)..." + @echo "Installing CodeScribe (embedding optional via CODESCRIBE_EMBED_*)..." @cargo install --path . --force @mkdir -p ~/.codescribe @pwd > ~/.codescribe/repo_path @@ -165,41 +170,52 @@ lint: @cargo clippy --workspace -- -D warnings test: - @echo "=== Tests (workspace) ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test --workspace --all-targets -- --nocapture - @echo "=== Tests (ignored / real API) ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test --workspace --all-targets -- --ignored --nocapture + @echo "=== Tests (workspace, all incl. real API) ===" + @$(ENV_LOAD); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test --workspace --all-targets -- --include-ignored --nocapture test-quick: @echo "=== Tests (quick, no real API) ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test --workspace --all-targets -- --nocapture + @$(ENV_LOAD); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test --workspace --all-targets -- --nocapture test-e2e: @echo "=== E2E Tests (mock) ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test e2e --release -- --nocapture + @$(E2E_ENV_GEN) + @$(ENV_LOAD_E2E); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test e2e --release -- --nocapture test-e2e-real: @echo "=== E2E Tests (real API) ===" @echo "Requires: LLM_API_KEY, LLM_ASSISTIVE_API_KEY" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test e2e --release -- --ignored --nocapture + @$(E2E_ENV_GEN) + @$(ENV_LOAD_E2E); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test e2e --release -- --ignored --nocapture test-sse: @echo "=== SSE Streaming Tests ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test e2e_sse --release -- --ignored --nocapture + @$(E2E_ENV_GEN) + @$(ENV_LOAD_E2E); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test e2e_sse --release -- --ignored --nocapture test-formatting: @echo "=== AI Formatting Tests ===" - @$(ENV_LOAD); $(APPLY_TEST_LLM); cargo test formatting --release -- --nocapture + @$(ENV_LOAD); $(APPLY_TEST_LLM); CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test formatting --release -- --nocapture test-all: @echo "=== Full Test Suite ===" @$(MAKE) test @echo "Done." +test-roundtrip: + @echo "=== Round-Trip Tests (TTS→STT→Embeddings) ===" + @echo "Tests actual embedded models pipeline integrity" + @$(E2E_ENV_GEN) + @$(ENV_LOAD_E2E); CODESCRIBE_E2E_ROUNDTRIP=1 CARGO_BUILD_JOBS=$(TEST_BUILD_JOBS) cargo test --test e2e_round_trip --release -- --nocapture + demo: @echo "=== Full Pipeline Demo ===" @cargo run --release --example demo_full_pipeline -- $(AUDIO) +demo-roundtrip: + @echo "=== Round-Trip Interactive Demo ===" + @cargo run --release --example roundtrip_live -- --interactive + demo-raw: @echo "=== Raw Transcription Demo ===" @cargo run --release --example demo_full_pipeline -- --raw $(AUDIO) @@ -212,7 +228,7 @@ check: @echo "=== Format Check (Rust) ===" @cargo fmt --all -- --check @echo "=== Format Check (non-Rust) ===" - @npx --yes prettier@2.7.1 --check . --ignore-path .prettierignore --ignore-unknown || true + @git ls-files --cached --exclude-standard | xargs npx --yes prettier@2.7.1 --check --ignore-path .prettierignore --ignore-unknown || true @echo "=== Clippy (workspace, all targets) ===" @cargo clippy --workspace --all-targets -- -D warnings @echo "=== Semgrep ===" @@ -223,7 +239,7 @@ fix: @echo "=== Format Fix (Rust) ===" @cargo fmt --all @echo "=== Format Fix (non-Rust) ===" - @npx --yes prettier@2.7.1 --write . --ignore-path .prettierignore --ignore-unknown + @git ls-files --cached --exclude-standard | xargs npx --yes prettier@2.7.1 --write --ignore-path .prettierignore --ignore-unknown @echo "Formatting applied" # ============================================================================ @@ -254,19 +270,27 @@ help: @echo "" @echo "Build & Install:" @echo " make build Build debug binary" - @echo " make release Build release binary (with embedded model)" - @echo " make install Install CLI (~888MB with embedded model)" + @echo " make release Build release binary (embedding optional via env)" + @echo " make install Install CLI (embedding optional via env)" @echo " make install-no-embed Install without model (needs CODESCRIBE_MODEL_PATH)" @echo " make config Edit ~/.codescribe/.env" @echo " make bundle Create CodeScribe.app bundle" @echo " make install-app Install to /Applications" @echo "" @echo "Release & Distribution:" - @echo " make dmg Build DMG (ad-hoc signed)" - @echo " make dmg-signed Build DMG (Developer ID signed)" - @echo " make dmg-full Build DMG with embedded model (~888MB)" - @echo " make notarize Notarize DMG with Apple" + @echo " make dmg Create DMG from existing .app (unsigned)" + @echo " make dmg-signed Build .app + DMG + sign (NO notarize)" + @echo " make dmg-full Build .app + DMG + sign + notarize" + @echo " make notarize Notarize existing DMG (NOTARY_PROFILE required)" + @echo "" + @echo "Release env vars:" + @echo " SIGN_IDENTITY=... Codesign identity (Developer ID Application: ...)" + @echo " NOTARY_PROFILE=... notarytool profile name" + @echo " BUNDLE_WHISPER=1 bundle local Whisper into .app" + @echo " BUNDLE_FALLBACK_GIT=1 allow fallback git clone if no local model" @echo " make download-model Download Whisper model from HF" + @echo " make download-e5 Download E5 embedder model from HF" + @echo " make download-silero-vad Download Silero VAD model from HF" @echo "" @echo "Run:" @echo " make start Start CodeScribe" @@ -301,21 +325,31 @@ help: # ============================================================================ dmg: - @./scripts/build-release.sh + @./packaging/create_dmg.sh dmg-signed: - @./scripts/build-release.sh --sign + @SIGN_IDENTITY="$(SIGN_IDENTITY)" NOTARY_PROFILE="$(NOTARY_PROFILE)" \ + BUNDLE_WHISPER="$(BUNDLE_WHISPER)" BUNDLE_FALLBACK_GIT="$(BUNDLE_FALLBACK_GIT)" \ + ./packaging/release.sh --no-notary dmg-full: - @./scripts/build-release.sh --sign + @SIGN_IDENTITY="$(SIGN_IDENTITY)" NOTARY_PROFILE="$(NOTARY_PROFILE)" \ + BUNDLE_WHISPER="$(BUNDLE_WHISPER)" BUNDLE_FALLBACK_GIT="$(BUNDLE_FALLBACK_GIT)" \ + ./packaging/release.sh notarize: @if ls CodeScribe_*.dmg 1> /dev/null 2>&1; then \ DMG=$$(ls -t CodeScribe_*.dmg | head -1); \ - ./scripts/notarize.sh "$$DMG"; \ + NOTARY_PROFILE="$(NOTARY_PROFILE)" ./scripts/notarize.sh "$$DMG"; \ else \ echo "No DMG found. Run 'make dmg-signed' first."; \ fi download-model: @./scripts/download-model.sh + +download-e5: + @./scripts/download-e5.sh + +download-silero-vad: + @./scripts/download-silero-vad.sh diff --git a/README.md b/README.md index b5477170..5e2657af 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ LLM_ASSISTIVE_API_KEY=sk-proj-xxx | HTTP Client | reqwest | LLM API calls | | API Format | openai-harmony | Responses API support | | Security | cap-std | Path safety hardening | -| Embeddings | fastembed | Local vector utilities | +| Embeddings | candle-transformers (E5) | Local vector utilities | ## Installation diff --git a/app/controller/helpers.rs b/app/controller/helpers.rs index 655972f1..a3705bea 100644 --- a/app/controller/helpers.rs +++ b/app/controller/helpers.rs @@ -37,12 +37,15 @@ pub fn is_conversation_session() -> bool { IS_CONVERSATION_SESSION.load(Ordering::SeqCst) } -/// Route transcription delta to transcription overlay (ALWAYS) -/// Chat is a CONSUMER of transcription, not the display target. +/// Route transcription delta to the active overlay. +/// Assistive sessions stream into chat bubbles; non-assistive uses transcription overlay. pub fn route_transcription_delta(delta: &str) { - // Transcription ALWAYS goes to transcription_overlay - // Voice chat is a CONSUMER - user decides when to send to AI - crate::append_transcription_delta(delta); + if is_assistive_session() { + crate::voice_chat_ui::append_voice_chat_user_delta(delta); + } else { + // Non-assistive: transcription overlay preview + crate::append_transcription_delta(delta); + } } /// Setup the voice chat send callback with config @@ -51,7 +54,7 @@ pub fn setup_voice_chat_send_callback(config: Arc>) { crate::voice_chat_ui::set_voice_chat_send_callback(Some(Arc::new(move |text: String| { let config = Arc::clone(&callback_config); tokio::spawn(async move { - crate::voice_chat_ui::update_voice_chat_status("Wysyłam..."); + crate::voice_chat_ui::update_voice_chat_status("Sending..."); crate::voice_chat_ui::set_voice_chat_sending(true); let (lang_str, transcript_mode) = { diff --git a/app/controller/mod.rs b/app/controller/mod.rs index 829e0d99..78286465 100644 --- a/app/controller/mod.rs +++ b/app/controller/mod.rs @@ -42,6 +42,7 @@ use crate::audio::streaming_recorder::StreamingRecorder; use crate::config::Config; use crate::config::models::ModelManager; use crate::os::clipboard; +use crate::os::selection::{AssistiveContext, build_assistive_input, capture_assistive_context}; use crate::tray::{TrayStatus, update_tray_status}; use crate::{BadgeMode, hide_hold_badge, show_badge_for_mode}; @@ -64,7 +65,7 @@ pub fn register_overlay_controller(controller: Arc) { } } -/// Stop the current recording and enter decision mode without waiting for VAD. +/// Stop the current recording and enter decision mode. pub fn request_recording_commit() { let Some(controller) = OVERLAY_CONTROLLER.get().cloned() else { warn!("Overlay controller not registered; cannot commit recording"); @@ -107,8 +108,14 @@ pub struct RecordingController { /// Lock to serialize finish_recording calls serial_lock: Arc>, - /// Flag set by VAD (silence detection) when recording should auto-stop - vad_triggered: Arc, + /// Assistive hands-off loop active (Right Option toggle) + assistive_loop_active: Arc, + + /// Best-effort selected-text/app context captured for assistive sessions. + /// + /// Must be captured BEFORE showing any overlay window, because overlays + /// may steal focus and destroy the user's selection context. + assistive_context: Arc>>, // ═══════════════════════════════════════════════════════════ // Conversation mode (Moshi full-duplex) @@ -129,6 +136,12 @@ pub struct RecordingController { /// Task handle for conversation audio processing loop conversation_task: Arc>>>, + + /// Flag to skip Moshi init if models are not available (avoids spam) + moshi_unavailable: Arc, + + /// Background tasks (cloud saves, etc.) — aborted on reset + background_tasks: Arc>>>, } impl RecordingController { @@ -141,13 +154,20 @@ impl RecordingController { config.hold_start_delay_ms, config.beep_on_start, config.whisper_language ); - let mut recorder = - StreamingRecorder::new().expect("Failed to initialize streaming recorder"); - recorder.set_delta_callback(Some(Arc::new(|delta| { - route_transcription_delta(delta); - }))); - - let model_manager = ModelManager::new().expect("Failed to initialize model manager"); + let mut recorder = StreamingRecorder::new().unwrap_or_else(|e| { + error!("StreamingRecorder init failed: {e}"); + panic!("Cannot start CodeScribe without audio recorder: {e}"); + }); + recorder.set_delta_callback(Some(Arc::new( + codescribe_core::pipeline::sinks::CallbackSink::new(Arc::new(|delta: &str| { + route_transcription_delta(delta); + })), + ))); + + let model_manager = ModelManager::new().unwrap_or_else(|e| { + error!("ModelManager init failed: {e}"); + panic!("Cannot start CodeScribe without model manager: {e}"); + }); if let Ok(models) = model_manager.list_models() && !models.is_empty() { @@ -172,13 +192,16 @@ impl RecordingController { session_id: Arc::new(RwLock::new(None)), hold_start_task: Arc::new(Mutex::new(None)), serial_lock: Arc::new(Mutex::new(())), - vad_triggered: Arc::new(AtomicBool::new(false)), + assistive_loop_active: Arc::new(AtomicBool::new(false)), + assistive_context: Arc::new(RwLock::new(None)), // Conversation mode (lazy init) conversation_engine: Arc::new(Mutex::new(None)), audio_player: Arc::new(Mutex::new(None)), conversation_stop_flag: Arc::new(AtomicBool::new(false)), conversation_generation: Arc::new(AtomicU64::new(0)), conversation_task: Arc::new(Mutex::new(None)), + moshi_unavailable: Arc::new(AtomicBool::new(false)), + background_tasks: Arc::new(Mutex::new(Vec::new())), } } @@ -193,13 +216,20 @@ impl RecordingController { cfg.hold_start_delay_ms, cfg.beep_on_start, cfg.whisper_language ); - let mut recorder = - StreamingRecorder::new().expect("Failed to initialize streaming recorder"); - recorder.set_delta_callback(Some(Arc::new(|delta| { - route_transcription_delta(delta); - }))); - - let model_manager = ModelManager::new().expect("Failed to initialize model manager"); + let mut recorder = StreamingRecorder::new().unwrap_or_else(|e| { + error!("StreamingRecorder init failed: {e}"); + panic!("Cannot start CodeScribe without audio recorder: {e}"); + }); + recorder.set_delta_callback(Some(Arc::new( + codescribe_core::pipeline::sinks::CallbackSink::new(Arc::new(|delta: &str| { + route_transcription_delta(delta); + })), + ))); + + let model_manager = ModelManager::new().unwrap_or_else(|e| { + error!("ModelManager init failed: {e}"); + panic!("Cannot start CodeScribe without model manager: {e}"); + }); if let Ok(models) = model_manager.list_models() && !models.is_empty() { @@ -223,13 +253,16 @@ impl RecordingController { session_id: Arc::new(RwLock::new(None)), hold_start_task: Arc::new(Mutex::new(None)), serial_lock: Arc::new(Mutex::new(())), - vad_triggered: Arc::new(AtomicBool::new(false)), + assistive_loop_active: Arc::new(AtomicBool::new(false)), + assistive_context: Arc::new(RwLock::new(None)), // Conversation mode (lazy init) conversation_engine: Arc::new(Mutex::new(None)), audio_player: Arc::new(Mutex::new(None)), conversation_stop_flag: Arc::new(AtomicBool::new(false)), conversation_generation: Arc::new(AtomicU64::new(0)), conversation_task: Arc::new(Mutex::new(None)), + moshi_unavailable: Arc::new(AtomicBool::new(false)), + background_tasks: Arc::new(Mutex::new(Vec::new())), } } @@ -248,16 +281,6 @@ impl RecordingController { self.config.read().await.clone() } - /// Check if VAD (silence detection) has triggered auto-stop - pub fn is_vad_triggered(&self) -> bool { - self.vad_triggered.load(Ordering::SeqCst) - } - - /// Clear the VAD triggered flag - pub fn clear_vad_triggered(&self) { - self.vad_triggered.store(false, Ordering::SeqCst); - } - /// Cancel any pending delayed hold-start task async fn cancel_pending_hold_start(&self) { let mut task_guard = self.hold_start_task.lock().await; @@ -300,6 +323,7 @@ impl RecordingController { } else if matches!(event.action, HotkeyAction::Down | HotkeyAction::Press) { // Only reset on Down/Press, not Up (preserves upgrade during hold) *self.assistive_mode.write().await = false; + *self.assistive_context.write().await = None; match event.key_type { HotkeyType::Hold => { @@ -373,6 +397,7 @@ impl RecordingController { } State::RecToggle => { info!("Toggle pressed again; finishing recording"); + self.assistive_loop_active.store(false, Ordering::SeqCst); self.finish_recording().await?; } _ => { @@ -413,6 +438,12 @@ impl RecordingController { /// Initializes ConversationEngine and AudioPlayer, then starts the audio /// processing loop that feeds mic input to Moshi and plays responses. async fn start_conversation_mode(&self) -> Result<()> { + // Early exit if Moshi was already determined to be unavailable (no spam) + if self.moshi_unavailable.load(Ordering::SeqCst) { + debug!("Moshi unavailable (cached) - skipping conversation mode"); + return Ok(()); + } + info!("Starting conversation mode (Moshi full-duplex)"); // 1. Initialize ConversationEngine if needed (lazy init) @@ -421,11 +452,23 @@ impl RecordingController { if engine_guard.is_none() { info!("Lazy-initializing ConversationEngine..."); let config = MoshiConfig::default(); + + // Pre-validate: check if model files exist before trying to load + if let Err(e) = config.validate() { + warn!( + "Moshi models not available: {} (conversation mode disabled)", + e + ); + self.moshi_unavailable.store(true, Ordering::SeqCst); + return Ok(()); // Silent return - not an error, just unavailable + } + match ConversationEngine::new(config) { Ok(mut engine) => { // Pre-initialize to load models now (rather than on first audio) if let Err(e) = engine.init() { error!("ConversationEngine init failed: {}", e); + self.moshi_unavailable.store(true, Ordering::SeqCst); crate::voice_chat_ui::add_voice_chat_error_message(&format!( "Moshi init failed: {}", e @@ -437,6 +480,7 @@ impl RecordingController { } Err(e) => { error!("Failed to create ConversationEngine: {}", e); + self.moshi_unavailable.store(true, Ordering::SeqCst); crate::voice_chat_ui::add_voice_chat_error_message(&format!( "Moshi unavailable: {}", e @@ -785,6 +829,8 @@ impl RecordingController { /// Schedule delayed recording start for hold mode async fn schedule_hold_start(&self) -> Result<()> { + // Hold mode never runs the assistive loop + self.assistive_loop_active.store(false, Ordering::SeqCst); // Check backend health before starting (skip in tests: no backend available) if !cfg!(test) { match crate::client::check_health().await { @@ -819,14 +865,13 @@ impl RecordingController { // Cancel any existing delayed start self.cancel_pending_hold_start().await; - // Reset VAD flag for new session - self.vad_triggered.store(false, Ordering::SeqCst); + // HOLD MODE: User controls recording by releasing the key. let state = Arc::clone(&self.state); let session_id = Arc::clone(&self.session_id); let recorder = Arc::clone(&self.recorder); let delay = Duration::from_millis(delay_ms); - let vad_flag = Arc::clone(&self.vad_triggered); + let assistive_context = Arc::clone(&self.assistive_context); let task = tokio::spawn(async move { // Wait for the configured delay @@ -845,13 +890,19 @@ impl RecordingController { info!("Starting hold recording (session={})", new_session_id); + // Set session mode for delta routing BEFORE starting recorder + set_assistive_session(is_assistive); + // Start the recorder (skip in tests: no CoreAudio device needed) - // hang_sec is configured via CODESCRIBE_VAD_MAX_SILENCE_SEC env var (single source of truth) + // HOLD MODE: User controls recording by releasing the key. + // If user wants to hold in silence for 45 minutes - that's their choice. let mut rec = recorder.lock().await; - rec.recorder.set_on_vad_stop(move || { - info!("VAD callback: setting vad_triggered flag"); - vad_flag.store(true, Ordering::SeqCst); - }); + // Set streaming callback for overlay updates (routed by session mode) + rec.set_delta_callback(Some(Arc::new( + codescribe_core::pipeline::sinks::CallbackSink::new(Arc::new(|text: &str| { + route_transcription_delta(text); + })), + ))); if !cfg!(test) && let Err(e) = rec.start(Some(language.as_str().to_string())).await { @@ -872,17 +923,24 @@ impl RecordingController { }; show_badge_for_mode(badge_mode); - // Set session mode for delta routing - set_assistive_session(is_assistive); - - // ALWAYS show transcription overlay for live preview - crate::clear_transcription_text(); - crate::show_transcription_overlay(); if is_assistive { + // Capture context BEFORE showing any overlay (overlays can steal focus). + let ctx = tokio::task::spawn_blocking(capture_assistive_context) + .await + .unwrap_or_default(); + *assistive_context.write().await = Some(ctx); + + crate::hide_transcription_overlay(); crate::show_voice_chat_overlay(); crate::show_agent_tab(); + crate::voice_chat_ui::update_voice_chat_status("Listening..."); + } else { + *assistive_context.write().await = None; + // Non-assistive: transcription overlay preview + crate::clear_transcription_text(); + crate::show_transcription_overlay(); + crate::enter_recording_mode(); } - crate::enter_recording_mode(); // Transition to REC_HOLD *state.write().await = State::RecHold; @@ -935,29 +993,28 @@ impl RecordingController { let new_session_id = Uuid::new_v4().to_string(); *self.session_id.write().await = Some(new_session_id.clone()); + if is_assistive { + *self.assistive_mode.write().await = true; + *self.force_raw_mode.write().await = false; + *self.force_ai_mode.write().await = false; + } + self.assistive_loop_active + .store(is_assistive, Ordering::SeqCst); + info!("Starting toggle recording (session={})", new_session_id); let config = self.config.read().await; let language = config.whisper_language; drop(config); - // Reset VAD flag and set callback - self.vad_triggered.store(false, Ordering::SeqCst); - let vad_flag = Arc::clone(&self.vad_triggered); - - // Start the recorder with VAD callback - // hang_sec is configured via CODESCRIBE_VAD_MAX_SILENCE_SEC env var (single source of truth) let mut recorder = self.recorder.lock().await; - recorder.recorder.set_on_vad_stop(move || { - info!("VAD callback: setting vad_triggered flag"); - vad_flag.store(true, Ordering::SeqCst); - }); - // Set streaming callback for overlay updates (routed by session mode) - recorder.set_delta_callback(Some(Arc::new(|text: &str| { - route_transcription_delta(text); - }))); + recorder.set_delta_callback(Some(Arc::new( + codescribe_core::pipeline::sinks::CallbackSink::new(Arc::new(|text: &str| { + route_transcription_delta(text); + })), + ))); // Skip actual audio stream in tests (no CoreAudio device needed) if !cfg!(test) { @@ -981,14 +1038,24 @@ impl RecordingController { // Set session mode for delta routing set_assistive_session(is_assistive); - // ALWAYS show transcription overlay for live preview - crate::clear_transcription_text(); - crate::show_transcription_overlay(); if is_assistive { + // Capture context BEFORE showing any overlay (overlays can steal focus). + let ctx = tokio::task::spawn_blocking(capture_assistive_context) + .await + .unwrap_or_default(); + *self.assistive_context.write().await = Some(ctx); + + crate::hide_transcription_overlay(); crate::show_voice_chat_overlay(); crate::show_agent_tab(); + crate::voice_chat_ui::update_voice_chat_status("Listening..."); + } else { + *self.assistive_context.write().await = None; + // Non-assistive: transcription overlay preview + crate::clear_transcription_text(); + crate::show_transcription_overlay(); + crate::enter_recording_mode(); } - crate::enter_recording_mode(); // Transition to REC_TOGGLE *self.state.write().await = State::RecToggle; @@ -1041,6 +1108,7 @@ impl RecordingController { let assistive = *self.assistive_mode.read().await; let force_raw = *self.force_raw_mode.read().await; let force_ai = *self.force_ai_mode.read().await; + let keep_assistive_loop = assistive && self.assistive_loop_active.load(Ordering::SeqCst); // Switch badge to processing mode (orange, pulsing) show_badge_for_mode(BadgeMode::Processing); @@ -1055,6 +1123,13 @@ impl RecordingController { *self.force_raw_mode.write().await = false; *self.force_ai_mode.write().await = false; *self.session_id.write().await = None; + *self.assistive_context.write().await = None; + // Keep assistive loop alive only when explicitly enabled and no manual stop was requested + if result.is_ok() && keep_assistive_loop { + self.assistive_loop_active.store(true, Ordering::SeqCst); + } else { + self.assistive_loop_active.store(false, Ordering::SeqCst); + } // Hide red dot indicator hide_hold_badge(); @@ -1139,6 +1214,9 @@ impl RecordingController { } }; + let chat_active = crate::voice_chat_ui::is_voice_chat_overlay_visible(); + let assistive_loop = assistive && self.assistive_loop_active.load(Ordering::SeqCst); + let mut raw_text_opt = None; let mut cloud_text_opt = None; let mut cloud_handle: Option>> = None; @@ -1194,7 +1272,20 @@ impl RecordingController { } } - let raw_text = raw_text_opt.ok_or_else(|| anyhow::anyhow!("Empty transcript"))?; + let raw_text = match raw_text_opt { + Some(text) if !text.trim().is_empty() => text, + Some(_) | None => { + if assistive_loop { + if chat_active { + crate::voice_chat_ui::set_voice_chat_sending(false); + crate::voice_chat_ui::update_voice_chat_status("Listening..."); + } + warn!("Empty transcript in assistive loop; skipping"); + return Ok(()); + } + return Err(anyhow::anyhow!("Empty transcript")); + } + }; info!("Raw transcript captured ({} chars)", raw_text.len()); @@ -1231,7 +1322,7 @@ impl RecordingController { warn!("Detected repetition loop in transcription - will clean up"); } - let chat_active = crate::voice_chat_ui::is_voice_chat_overlay_visible(); + // chat_active already captured above // Determine final text based on mode (NEW architecture): // @@ -1244,17 +1335,27 @@ impl RecordingController { // - Quick dictation? → Ctrl (fast, raw) // - Need formatting? → Double Option (respects setting) // - Need AI help? → Ctrl+Shift (always AI) - let (formatted_text, output_kind) = if assistive { + let (formatted_text, output_kind, should_auto_paste) = if assistive { // Ctrl+Shift: ALWAYS augmentation mode (AI expands content) info!("Assistive mode (Ctrl+Shift): augmenting transcript via AI"); if chat_active { - crate::voice_chat_ui::add_voice_chat_user_message(&clean_text); + // Finalize the streaming user draft into a bubble + crate::show_voice_chat_overlay(); + crate::voice_chat_ui::set_voice_chat_user_text(&clean_text); crate::voice_chat_ui::show_agent_tab(); crate::voice_chat_ui::set_voice_chat_sending(true); crate::voice_chat_ui::update_voice_chat_status("Thinking..."); } + let ctx = self + .assistive_context + .read() + .await + .clone() + .unwrap_or_default(); + let assistive_input = build_assistive_input(&clean_text, &ctx); + let lang_str = language_opt.map(String::from); // Determine streaming mode from config @@ -1274,7 +1375,7 @@ impl RecordingController { }; let result = crate::ai_formatting::format_text_with_status( - &clean_text, + &assistive_input, lang_str.as_deref(), true, delta_callback, @@ -1284,6 +1385,7 @@ impl RecordingController { crate::ai_formatting::AiFormatStatus::Applied => { if chat_active { // Display AI response in overlay + crate::show_voice_chat_overlay(); crate::voice_chat_ui::update_voice_chat_status("AI Response:"); crate::voice_chat_ui::set_voice_chat_text(&result.text); info!( @@ -1295,6 +1397,7 @@ impl RecordingController { } crate::ai_formatting::AiFormatStatus::Failed => { if chat_active { + crate::show_voice_chat_overlay(); crate::voice_chat_ui::update_voice_chat_status("AI Failed"); crate::voice_chat_ui::add_voice_chat_error_message("AI Failed"); } @@ -1307,7 +1410,7 @@ impl RecordingController { crate::state::history::TranscriptKind::Raw } }; - (result.text, kind) + (result.text, kind, false) } else if force_raw { // Ctrl Hold: ALWAYS raw transcript (fast dictation mode) // Post-processed clean_text is used (lexicon + cleanup already applied) @@ -1316,12 +1419,14 @@ impl RecordingController { ( crate::ai_formatting::remove_simple_repetitions(&clean_text), crate::state::history::TranscriptKind::Raw, + true, ) } else { info!("Raw mode (Ctrl): using post-processed transcript"); ( clean_text.clone(), crate::state::history::TranscriptKind::Raw, + true, ) } } else if force_ai { @@ -1365,6 +1470,7 @@ impl RecordingController { crate::ai_formatting::AiFormatStatus::Applied => { if chat_active { // Display formatted text in overlay + crate::show_voice_chat_overlay(); crate::voice_chat_ui::update_voice_chat_status("Formatted:"); crate::voice_chat_ui::set_voice_chat_text(&result.text); info!( @@ -1388,12 +1494,13 @@ impl RecordingController { crate::state::history::TranscriptKind::Raw } }; - (result.text, kind) + (result.text, kind, false) } else if has_repetition { info!("Formatting mode (Left Option): AI unavailable, cleaning repetitions"); ( crate::ai_formatting::remove_simple_repetitions(&clean_text), crate::state::history::TranscriptKind::Raw, + false, ) } else { info!( @@ -1402,6 +1509,7 @@ impl RecordingController { ( clean_text.clone(), crate::state::history::TranscriptKind::Raw, + false, ) } } else { @@ -1448,6 +1556,7 @@ impl RecordingController { crate::ai_formatting::AiFormatStatus::Applied => { if chat_active { // Display formatted text in overlay + crate::show_voice_chat_overlay(); crate::voice_chat_ui::update_voice_chat_status("Formatted:"); crate::voice_chat_ui::set_voice_chat_text(&result.text); info!( @@ -1471,13 +1580,14 @@ impl RecordingController { crate::state::history::TranscriptKind::Raw } }; - (result.text, kind) + (result.text, kind, false) } else if has_repetition { // Toggle OFF with repetition: local cleanup only info!("Raw mode (Toggle OFF): applying local repetition cleanup"); ( crate::ai_formatting::remove_simple_repetitions(&clean_text), crate::state::history::TranscriptKind::Raw, + true, ) } else { // Toggle OFF: using post-processed transcript @@ -1485,6 +1595,7 @@ impl RecordingController { ( clean_text.clone(), crate::state::history::TranscriptKind::Raw, + true, ) } }; @@ -1516,10 +1627,15 @@ impl RecordingController { ); } - // Paste the text into the active application - clipboard::paste_text(&formatted_text).context("Failed to paste text")?; - - info!("Text pasted successfully"); + if cfg!(test) { + info!("Skipping paste in tests (mode={})", mode_label); + } else if should_auto_paste { + // Paste the text into the active application + clipboard::paste_text(&formatted_text).context("Failed to paste text")?; + info!("Text pasted successfully"); + } else { + info!("Auto-paste skipped (mode={})", mode_label); + } // Save final transcript (skip duplicate when RAW already stored and unchanged) let needs_final_save = !raw_save_enabled @@ -1549,7 +1665,7 @@ impl RecordingController { } else if let Some(handle) = cloud_handle { let slug_hint = raw_text.clone(); let timestamp = recording_timestamp; - tokio::spawn(async move { + let bg_handle = tokio::spawn(async move { match handle.await { Ok(Ok(text)) => { let entry = crate::state::history::save_entry_with_timestamp_and_slug( @@ -1564,6 +1680,11 @@ impl RecordingController { Err(e) => error!("Cloud transcription task failed: {}", e), } }); + // Track for cleanup on reset + if let Ok(mut tasks) = self.background_tasks.try_lock() { + tasks.retain(|h| !h.is_finished()); + tasks.push(bg_handle); + } } Ok(()) @@ -1585,6 +1706,14 @@ impl RecordingController { *self.force_raw_mode.write().await = false; *self.force_ai_mode.write().await = false; *self.session_id.write().await = None; + *self.assistive_context.write().await = None; + + // Abort outstanding background tasks + if let Ok(mut tasks) = self.background_tasks.try_lock() { + for handle in tasks.drain(..) { + handle.abort(); + } + } // Hide UI indicators hide_hold_badge(); diff --git a/app/lib.rs b/app/lib.rs index c1a36739..706cce2e 100644 --- a/app/lib.rs +++ b/app/lib.rs @@ -48,8 +48,13 @@ pub mod dev; #[cfg(target_os = "macos")] pub use ui::{ BadgeMode, HoldBadgeConfig, focused_element_accepts_text, get_caret_position, - get_cursor_position, hide_hold_badge, set_dock_icon, show_badge_for_mode, show_hold_badge, - show_hold_badge_with_config, + get_cursor_position, hide_hold_badge, install_basic_edit_menu, set_dock_icon, + show_badge_for_mode, show_hold_badge, show_hold_badge_with_config, +}; + +#[cfg(target_os = "macos")] +pub use ui::bootstrap::{ + hide_bootstrap_overlay, schedule_bootstrap, should_show_bootstrap, show_bootstrap_overlay, }; #[cfg(target_os = "macos")] @@ -58,11 +63,12 @@ pub use ui::tray; #[cfg(target_os = "macos")] pub use voice_chat_ui::{ VoiceChatOverlayConfig, add_voice_chat_error_message, add_voice_chat_user_message, - append_voice_chat_assistant_delta, clear_voice_chat_text, filter_drawer, - hide_voice_chat_overlay, is_auto_send_enabled, is_voice_chat_overlay_visible, refresh_drawer, - reset_voice_chat_activity, send_voice_chat_draft, set_voice_chat_send_callback, - set_voice_chat_sending, set_voice_chat_text, show_agent_tab, show_drawer_tab, - show_voice_chat_overlay, show_voice_chat_overlay_with_config, update_voice_chat_status, + append_voice_chat_assistant_delta, append_voice_chat_user_delta, clear_voice_chat_text, + filter_drawer, hide_voice_chat_overlay, is_auto_send_enabled, is_voice_chat_overlay_visible, + refresh_drawer, reset_voice_chat_activity, send_voice_chat_draft, set_voice_chat_send_callback, + set_voice_chat_sending, set_voice_chat_text, set_voice_chat_user_text, show_agent_tab, + show_drawer_tab, show_voice_chat_overlay, show_voice_chat_overlay_with_config, + update_voice_chat_status, }; #[cfg(target_os = "macos")] diff --git a/app/os/clipboard.rs b/app/os/clipboard.rs index 70837cc8..fb475ec9 100644 --- a/app/os/clipboard.rs +++ b/app/os/clipboard.rs @@ -29,6 +29,8 @@ use tracing::{debug, info, warn}; /// macOS virtual key code for 'V' key const KEYCODE_V: CGKeyCode = 9; +/// macOS virtual key code for 'C' key +const KEYCODE_C: CGKeyCode = 8; /// macOS virtual key code for Right Arrow const KEYCODE_RIGHT_ARROW: CGKeyCode = 124; @@ -231,6 +233,22 @@ fn simulate_cmd_v() -> Result<()> { Ok(()) } +/// Simulates Cmd+C keystroke using CGEvent +/// +/// Used for best-effort selection capture (clipboard snapshot+restore). +pub(crate) fn simulate_cmd_c() -> Result<()> { + let cmd_flag = CGEventFlags::CGEventFlagCommand; + + // Key down: C with Cmd modifier + simulate_key_event(KEYCODE_C, true, cmd_flag)?; + thread::sleep(Duration::from_millis(10)); + + // Key up: C with Cmd modifier + simulate_key_event(KEYCODE_C, false, cmd_flag)?; + + Ok(()) +} + /// Simulates Right Arrow keystroke using CGEvent fn simulate_right_arrow() -> Result<()> { // Key down: Right Arrow (no modifiers) diff --git a/app/os/hotkeys.rs b/app/os/hotkeys.rs index 8785b9e1..95bacc94 100644 --- a/app/os/hotkeys.rs +++ b/app/os/hotkeys.rs @@ -124,6 +124,8 @@ pub enum HotkeyEvent { ToggleNormal, /// Assistive toggle gesture (double-tap right Option) ToggleAssistive, + /// Conversation mode gesture (Ctrl+Option hold) - full-duplex Moshi + Conversation { action: HoldAction }, } /// Modifier flags for hold gesture detection @@ -468,15 +470,22 @@ mod macos { state.hold_event_sent = false; tracing::debug!( - "Hold combo activated ({:?}, assistive={}) - sending Hold Down event", + "Hold combo activated ({:?}, assistive={}) - sending event", hold_mods, is_assistive ); - // Send Hold Down immediately for responsiveness - let _ = state.tx.send(HotkeyEvent::Hold { - action: HoldAction::Down, - assistive: state.assistive_mode, - }); + // Send appropriate event based on hold mode + // CtrlAlt (Ctrl+Option) = Conversation mode (Moshi full-duplex) + if hold_mods == HoldMods::CtrlAlt { + let _ = state.tx.send(HotkeyEvent::Conversation { + action: HoldAction::Down, + }); + } else { + let _ = state.tx.send(HotkeyEvent::Hold { + action: HoldAction::Down, + assistive: state.assistive_mode, + }); + } state.hold_event_sent = true; } else if combo_active && state.hold_active && is_assistive && !state.assistive_mode { // Shift was added while combo active - upgrade to assistive mode @@ -493,10 +502,17 @@ mod macos { let elapsed = ts.elapsed(); tracing::debug!("Hold combo released after {:?}", elapsed); } - let _ = state.tx.send(HotkeyEvent::Hold { - action: HoldAction::Up, - assistive: state.assistive_mode, - }); + // Send appropriate event based on hold mode + if hold_mods == HoldMods::CtrlAlt { + let _ = state.tx.send(HotkeyEvent::Conversation { + action: HoldAction::Up, + }); + } else { + let _ = state.tx.send(HotkeyEvent::Hold { + action: HoldAction::Up, + assistive: state.assistive_mode, + }); + } } state.hold_active_ts = None; } diff --git a/app/os/mod.rs b/app/os/mod.rs index def6f481..304c9deb 100644 --- a/app/os/mod.rs +++ b/app/os/mod.rs @@ -4,3 +4,5 @@ pub mod clipboard; pub mod hotkeys; #[cfg(target_os = "macos")] pub mod permissions; +#[cfg(target_os = "macos")] +pub mod selection; diff --git a/app/os/selection.rs b/app/os/selection.rs new file mode 100644 index 00000000..e70629a0 --- /dev/null +++ b/app/os/selection.rs @@ -0,0 +1,252 @@ +//! Selection/context capture for assistive mode (macOS) +//! +//! POC goal: +//! - If user has selected text in the frontmost app, include it as context for Assistive mode. +//! - Avoid clipboard pollution by snapshot+restore. +//! - Best-effort only: failure should never break recording/transcription. + +use std::time::Duration; + +use tracing::{debug, warn}; + +use crate::os::clipboard::{self, ClipboardSnapshot}; + +#[derive(Debug, Clone, Default)] +pub struct AssistiveContext { + pub frontmost_app: Option, + pub selected_text: Option, +} + +fn env_flag(key: &str, default: bool) -> bool { + std::env::var(key) + .ok() + .map(|v| { + let v = v.to_lowercase(); + !matches!(v.as_str(), "0" | "false" | "no" | "off") + }) + .unwrap_or(default) +} + +fn env_usize(key: &str, default: usize) -> usize { + std::env::var(key) + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(default) +} + +fn env_u64(key: &str, default: u64) -> u64 { + std::env::var(key) + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(default) +} + +/// Capture best-effort context for assistive mode. +/// +/// Env knobs (POC): +/// - `ASSISTIVE_CONTEXT_ENABLED` (default: 1) +/// - `ASSISTIVE_CONTEXT_MAX_CHARS` (default: 5000) +/// - `ASSISTIVE_CONTEXT_INCLUDE_APP` (default: 1) +/// - `ASSISTIVE_CONTEXT_COPY_DELAY_MS` (default: 150) +/// - `ASSISTIVE_CONTEXT_COPY_FALLBACK` (default: 0) - enable Cmd+C fallback when AX selection is unavailable +pub fn capture_assistive_context() -> AssistiveContext { + // Unit tests should not trigger osascript / clipboard / event simulation. + if cfg!(test) { + return AssistiveContext::default(); + } + + if !env_flag("ASSISTIVE_CONTEXT_ENABLED", true) { + return AssistiveContext::default(); + } + + let max_chars = env_usize("ASSISTIVE_CONTEXT_MAX_CHARS", 5000); + let include_app = env_flag("ASSISTIVE_CONTEXT_INCLUDE_APP", true); + let copy_delay_ms = env_u64("ASSISTIVE_CONTEXT_COPY_DELAY_MS", 150); + + let frontmost_app = if include_app { + frontmost_app_name() + } else { + None + }; + + // Avoid capturing from ourselves (frontmost can temporarily become CodeScribe) + if matches!( + frontmost_app.as_deref(), + Some("CodeScribe") | Some("codescribe") + ) { + debug!("Assistive context: frontmost is CodeScribe, skipping selection capture"); + return AssistiveContext { + frontmost_app, + selected_text: None, + }; + } + + let selected_text = selected_text_from_frontmost(max_chars, copy_delay_ms); + + debug!( + "Assistive context captured (app_present={}, selected_chars={})", + frontmost_app.is_some(), + selected_text.as_ref().map(|s| s.len()).unwrap_or(0) + ); + + AssistiveContext { + frontmost_app, + selected_text, + } +} + +/// Capture only the frontmost app name (no selection, no clipboard). +/// +/// This is used to make paste actions (⇲) target the right app even when we're not in Assistive +/// selection mode. +pub fn capture_frontmost_app_only() -> AssistiveContext { + if cfg!(test) { + return AssistiveContext::default(); + } + + if !env_flag("ASSISTIVE_CONTEXT_ENABLED", true) { + return AssistiveContext::default(); + } + + let include_app = env_flag("ASSISTIVE_CONTEXT_INCLUDE_APP", true); + let frontmost_app = if include_app { + frontmost_app_name() + } else { + None + }; + + AssistiveContext { + frontmost_app, + selected_text: None, + } +} + +/// Build the LLM input for assistive mode, including optional selection context. +pub fn build_assistive_input(user_voice_text: &str, ctx: &AssistiveContext) -> String { + let instruction = user_voice_text.trim(); + let selected_text = ctx.selected_text.as_deref().unwrap_or("").trim(); + let frontmost_app = ctx.frontmost_app.as_deref().unwrap_or("").trim(); + + let mut out = String::new(); + + out.push_str("INSTRUKCJA_UŻYTKOWNIKA:\n<<<\n"); + out.push_str(instruction); + out.push_str("\n>\n\n"); + + out.push_str("ZAZNACZONY_TEKST:\n<<<\n"); + out.push_str(selected_text); + out.push_str("\n>\n"); + + if !frontmost_app.is_empty() { + out.push_str("\nKONTEKST:\n- frontmost_app: "); + out.push_str(frontmost_app); + out.push('\n'); + } + + out +} + +#[cfg(target_os = "macos")] +fn frontmost_app_name() -> Option { + use std::process::Command; + + // This is best-effort. It may fail if System Events is restricted. + let output = Command::new("osascript") + .args([ + "-e", + r#"tell application "System Events" to name of first application process whose frontmost is true"#, + ]) + .output() + .ok()?; + + if !output.status.success() { + return None; + } + + let s = String::from_utf8_lossy(&output.stdout).trim().to_string(); + (!s.is_empty()).then_some(s) +} + +#[cfg(not(target_os = "macos"))] +fn frontmost_app_name() -> Option { + None +} + +#[cfg(target_os = "macos")] +fn selected_text_from_frontmost(max_chars: usize, copy_delay_ms: u64) -> Option { + // Prefer Accessibility selection if available (doesn't depend on clipboard). + // + // Some apps report `AXSelectedTextRange.length == 0` even when `AXSelectedText` is non-empty, + // so we do *not* early-return on length==0 before checking `AXSelectedText`. + let sel_len = crate::ui::get_selected_text_length(); + if let Some(selected) = crate::ui::get_selected_text(max_chars) { + return Some(selected); + } + + // If we can reliably detect that selection length is zero, treat as "no selection" and + // never use any fallback that might touch the clipboard. + if matches!(sel_len, Some(0)) { + debug!("Assistive context: selection length is 0; skipping Cmd+C fallback"); + return None; + } + + // Cmd+C fallback is enabled by default for Selection mode. Some apps don't expose AX APIs. + // We snapshot+restore to avoid clipboard pollution and treat "unchanged clipboard" as no selection. + // + // NOTE: Default is OFF because Cmd+C can be surprising/privacy-sensitive (it touches clipboard) + // and some users explicitly want selection context to come only from AXSelectedText. + if !env_flag("ASSISTIVE_CONTEXT_COPY_FALLBACK", false) { + return None; + } + + // Fallback: snapshot clipboard + Cmd+C + restore. + // This can fail in some apps and can mis-detect "no selection" when clipboard doesn't change. + let snapshot = ClipboardSnapshot::capture().ok(); + let prev_text = snapshot.as_ref().and_then(|s| s.text.clone()); + + if let Err(e) = clipboard::simulate_cmd_c() { + warn!("Assistive context: failed to simulate Cmd+C: {}", e); + return None; + } + + std::thread::sleep(Duration::from_millis(copy_delay_ms)); + + let mut copied = match clipboard::get_clipboard() { + Ok(t) => t, + Err(e) => { + debug!("Assistive context: clipboard read failed: {}", e); + String::new() + } + }; + + if let Some(snapshot) = snapshot + && let Err(e) = snapshot.restore() + { + debug!("Assistive context: clipboard restore failed: {}", e); + } + + copied = copied.trim().to_string(); + if copied.is_empty() { + return None; + } + + // If clipboard didn't change, treat as "no selection" to avoid leaking arbitrary clipboard data. + if let Some(prev) = prev_text + && copied == prev.trim() + { + debug!("Assistive context: clipboard unchanged; treating as no selection"); + return None; + } + + if copied.len() > max_chars { + copied.truncate(max_chars); + copied.push('…'); + } + + Some(copied) +} + +#[cfg(not(target_os = "macos"))] +fn selected_text_from_frontmost(_max_chars: usize, _copy_delay_ms: u64) -> Option { + None +} diff --git a/app/ui/bootstrap/handlers.rs b/app/ui/bootstrap/handlers.rs new file mode 100644 index 00000000..92d5e26e --- /dev/null +++ b/app/ui/bootstrap/handlers.rs @@ -0,0 +1,56 @@ +use objc::declare::ClassDecl; +use objc::runtime::{Class, Object, Sel}; +use objc::{sel, sel_impl}; +use std::sync::Once; + +use super::{handle_finish, handle_hotkey_done, handle_show_overlay, handle_test_mic}; + +pub type Id = *mut Object; + +static ACTION_HANDLER_INIT: Once = Once::new(); +static mut ACTION_HANDLER_CLASS: *const Class = std::ptr::null(); + +pub fn action_handler_class() -> *const Class { + unsafe { + ACTION_HANDLER_INIT.call_once(|| { + let superclass = Class::get("NSObject").expect("NSObject not found"); + let mut decl = ClassDecl::new("BootstrapOverlayActionHandler", superclass) + .expect("Failed to declare handler class"); + decl.add_method( + sel!(onTestMic:), + on_test_mic as extern "C" fn(&Object, Sel, Id), + ); + decl.add_method( + sel!(onShowOverlay:), + on_show_overlay as extern "C" fn(&Object, Sel, Id), + ); + decl.add_method( + sel!(onHotkeyDone:), + on_hotkey_done as extern "C" fn(&Object, Sel, Id), + ); + decl.add_method( + sel!(onFinish:), + on_finish as extern "C" fn(&Object, Sel, Id), + ); + ACTION_HANDLER_CLASS = decl.register(); + }); + + ACTION_HANDLER_CLASS + } +} + +extern "C" fn on_test_mic(_this: &Object, _sel: Sel, _sender: Id) { + handle_test_mic(); +} + +extern "C" fn on_show_overlay(_this: &Object, _sel: Sel, _sender: Id) { + handle_show_overlay(); +} + +extern "C" fn on_hotkey_done(_this: &Object, _sel: Sel, _sender: Id) { + handle_hotkey_done(); +} + +extern "C" fn on_finish(_this: &Object, _sel: Sel, _sender: Id) { + handle_finish(); +} diff --git a/app/ui/bootstrap/mod.rs b/app/ui/bootstrap/mod.rs new file mode 100644 index 00000000..6c10b3d3 --- /dev/null +++ b/app/ui/bootstrap/mod.rs @@ -0,0 +1,421 @@ +use std::fs; +use std::io::{BufRead, BufReader, Write}; +use std::os::unix::net::UnixStream; +use std::path::PathBuf; +use std::sync::Mutex; +use std::thread; +use std::time::Duration; + +use dispatch::Queue; +use lazy_static::lazy_static; +use objc::runtime::{Class, Object}; +use objc::{msg_send, sel, sel_impl}; +use objc2_app_kit::{ + NSBackingStoreType, NSVisualEffectBlendingMode, NSVisualEffectMaterial, NSVisualEffectState, + NSWindowCollectionBehavior, NSWindowStyleMask, +}; +use tracing::{info, warn}; + +use crate::config::Config; +use crate::ipc::{IpcCommand, IpcResponse}; +use crate::ui::bootstrap::handlers::action_handler_class; +use crate::ui_helpers::{ + LabelConfig, NS_FLOATING_WINDOW_LEVEL, add_subview, button, button_set_action, color_rgba, + color_white, create_label, set_text_field_string, window_close, window_show, +}; + +mod handlers; + +// Type alias for Objective-C object pointers +type Id = *mut Object; + +const BOOTSTRAP_WIDTH: f64 = 480.0; +const BOOTSTRAP_HEIGHT: f64 = 260.0; + +const STEP_TEST_MIC: usize = 0; +const STEP_SHOW_OVERLAY: usize = 1; +const STEP_PRESS_HOTKEY: usize = 2; + +#[derive(Default)] +struct BootstrapState { + window: Option, + step_labels: [Option; 3], +} + +lazy_static! { + static ref BOOTSTRAP_STATE: Mutex = Mutex::new(BootstrapState::default()); +} + +fn bootstrap_done_path() -> PathBuf { + Config::config_dir().join("bootstrap_done") +} + +pub fn should_show_bootstrap() -> bool { + !bootstrap_done_path().exists() +} + +fn mark_bootstrap_done() { + let path = bootstrap_done_path(); + if let Some(parent) = path.parent() { + let _ = fs::create_dir_all(parent); + } + let _ = fs::write(path, "done"); +} + +pub fn schedule_bootstrap() { + if !should_show_bootstrap() { + return; + } + + thread::spawn(|| { + thread::sleep(Duration::from_millis(800)); + show_bootstrap_overlay(); + }); +} + +pub fn show_bootstrap_overlay() { + Queue::main().exec_async(|| { + show_bootstrap_overlay_impl(); + }); +} + +fn show_bootstrap_overlay_impl() { + unsafe { + let mut state = BOOTSTRAP_STATE.lock().unwrap_or_else(|e| e.into_inner()); + + if let Some(window_ptr) = state.window { + let window = window_ptr as Id; + let _: () = msg_send![window, setLevel: NS_FLOATING_WINDOW_LEVEL]; + window_show(window); + let nil: *mut Object = std::ptr::null_mut(); + let _: () = msg_send![window, makeKeyAndOrderFront: nil]; + return; + } + + let ns_window = Class::get("NSWindow").unwrap(); + let ns_screen = Class::get("NSScreen").unwrap(); + let ns_visual = Class::get("NSVisualEffectView").unwrap(); + + let main_screen: Id = msg_send![ns_screen, mainScreen]; + if main_screen.is_null() { + warn!("No main screen available for bootstrap overlay"); + return; + } + let visible_frame: core_graphics::geometry::CGRect = msg_send![main_screen, visibleFrame]; + + let x = visible_frame.origin.x + (visible_frame.size.width - BOOTSTRAP_WIDTH) / 2.0; + let y = visible_frame.origin.y + (visible_frame.size.height - BOOTSTRAP_HEIGHT) / 2.0; + + let frame = core_graphics::geometry::CGRect { + origin: core_graphics::geometry::CGPoint { x, y }, + size: core_graphics::geometry::CGSize { + width: BOOTSTRAP_WIDTH, + height: BOOTSTRAP_HEIGHT, + }, + }; + + let window: Id = msg_send![ns_window, alloc]; + let style_mask = NSWindowStyleMask::Borderless | NSWindowStyleMask::FullSizeContentView; + let backing = NSBackingStoreType::Buffered; + let window: Id = msg_send![ + window, + initWithContentRect: frame + styleMask: style_mask + backing: backing + defer: false + ]; + + let _: () = msg_send![window, setOpaque: false]; + let clear_color = color_rgba(0.0, 0.0, 0.0, 0.0); + let _: () = msg_send![window, setBackgroundColor: clear_color]; + let _: () = msg_send![window, setLevel: NS_FLOATING_WINDOW_LEVEL]; + let _: () = msg_send![window, setHasShadow: true]; + let _: () = msg_send![window, setMovableByWindowBackground: true]; + let collection_behavior = NSWindowCollectionBehavior::CanJoinAllSpaces + | NSWindowCollectionBehavior::FullScreenAuxiliary; + let _: () = msg_send![window, setCollectionBehavior: collection_behavior]; + + let content_view: Id = msg_send![window, contentView]; + if content_view.is_null() { + warn!("Failed to get content view for bootstrap overlay"); + return; + } + + let blur_frame = core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(0.0, 0.0), + &core_graphics::geometry::CGSize::new(BOOTSTRAP_WIDTH, BOOTSTRAP_HEIGHT), + ); + let blur_view: Id = msg_send![ns_visual, alloc]; + let blur_view: Id = msg_send![blur_view, initWithFrame: blur_frame]; + let _: () = msg_send![blur_view, setMaterial: NSVisualEffectMaterial::HUDWindow]; + let _: () = msg_send![blur_view, setBlendingMode: NSVisualEffectBlendingMode::BehindWindow]; + let _: () = msg_send![blur_view, setState: NSVisualEffectState::Active]; + let _: () = msg_send![blur_view, setWantsLayer: true]; + let layer: Id = msg_send![blur_view, layer]; + if !layer.is_null() { + let _: () = msg_send![layer, setCornerRadius: 16.0f64]; + let _: () = msg_send![layer, setMasksToBounds: true]; + } + add_subview(content_view, blur_view); + + let action_handler_class = action_handler_class(); + let action_handler: Id = msg_send![action_handler_class, new]; + + let title_label = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, BOOTSTRAP_HEIGHT - 40.0), + &core_graphics::geometry::CGSize::new(BOOTSTRAP_WIDTH - 40.0, 24.0), + ), + text: "Welcome to CodeScribe".to_string(), + font_size: 15.0, + bold: true, + text_color: color_white(0.95), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, title_label); + + let subtitle_label = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, BOOTSTRAP_HEIGHT - 64.0), + &core_graphics::geometry::CGSize::new(BOOTSTRAP_WIDTH - 40.0, 18.0), + ), + text: "3 quick steps (under 60s)".to_string(), + font_size: 12.0, + bold: false, + text_color: color_white(0.7), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, subtitle_label); + + // Step 1: Test mic + let step1_label = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, BOOTSTRAP_HEIGHT - 110.0), + &core_graphics::geometry::CGSize::new(260.0, 20.0), + ), + text: "1) Test mic".to_string(), + font_size: 13.0, + bold: true, + text_color: color_white(0.9), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step1_label); + + let step1_status = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(300.0, BOOTSTRAP_HEIGHT - 110.0), + &core_graphics::geometry::CGSize::new(80.0, 20.0), + ), + text: "pending".to_string(), + font_size: 11.0, + bold: false, + text_color: color_white(0.6), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step1_status); + + let step1_btn = button( + core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(380.0, BOOTSTRAP_HEIGHT - 114.0), + &core_graphics::geometry::CGSize::new(80.0, 26.0), + ), + "Test", + ); + button_set_action(step1_btn, action_handler, sel!(onTestMic:)); + add_subview(blur_view, step1_btn); + + // Step 2: Show overlay + let step2_label = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, BOOTSTRAP_HEIGHT - 145.0), + &core_graphics::geometry::CGSize::new(260.0, 20.0), + ), + text: "2) Show chat overlay".to_string(), + font_size: 13.0, + bold: true, + text_color: color_white(0.9), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step2_label); + + let step2_status = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(300.0, BOOTSTRAP_HEIGHT - 145.0), + &core_graphics::geometry::CGSize::new(80.0, 20.0), + ), + text: "pending".to_string(), + font_size: 11.0, + bold: false, + text_color: color_white(0.6), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step2_status); + + let step2_btn = button( + core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(380.0, BOOTSTRAP_HEIGHT - 149.0), + &core_graphics::geometry::CGSize::new(80.0, 26.0), + ), + "Show", + ); + button_set_action(step2_btn, action_handler, sel!(onShowOverlay:)); + add_subview(blur_view, step2_btn); + + // Step 3: Press hotkey + let step3_label = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, BOOTSTRAP_HEIGHT - 180.0), + &core_graphics::geometry::CGSize::new(260.0, 20.0), + ), + text: "3) Press hotkey (Ctrl+Shift)".to_string(), + font_size: 13.0, + bold: true, + text_color: color_white(0.9), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step3_label); + + let step3_status = create_label(LabelConfig { + frame: core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(300.0, BOOTSTRAP_HEIGHT - 180.0), + &core_graphics::geometry::CGSize::new(80.0, 20.0), + ), + text: "pending".to_string(), + font_size: 11.0, + bold: false, + text_color: color_white(0.6), + background_color: None, + selectable: false, + editable: false, + }); + add_subview(blur_view, step3_status); + + let step3_btn = button( + core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(380.0, BOOTSTRAP_HEIGHT - 184.0), + &core_graphics::geometry::CGSize::new(80.0, 26.0), + ), + "Done", + ); + button_set_action(step3_btn, action_handler, sel!(onHotkeyDone:)); + add_subview(blur_view, step3_btn); + + // Footer buttons + let finish_btn = button( + core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(BOOTSTRAP_WIDTH - 110.0, 16.0), + &core_graphics::geometry::CGSize::new(90.0, 28.0), + ), + "Finish", + ); + button_set_action(finish_btn, action_handler, sel!(onFinish:)); + add_subview(blur_view, finish_btn); + + let skip_btn = button( + core_graphics::geometry::CGRect::new( + &core_graphics::geometry::CGPoint::new(20.0, 16.0), + &core_graphics::geometry::CGSize::new(90.0, 28.0), + ), + "Skip", + ); + button_set_action(skip_btn, action_handler, sel!(onFinish:)); + add_subview(blur_view, skip_btn); + + // Store state + state.window = Some(window as usize); + state.step_labels = [ + Some(step1_status as usize), + Some(step2_status as usize), + Some(step3_status as usize), + ]; + + window_show(window); + let nil: *mut Object = std::ptr::null_mut(); + let _: () = msg_send![window, makeKeyAndOrderFront: nil]; + + info!("Bootstrap overlay shown"); + } +} + +pub(super) fn handle_test_mic() { + update_step_status(STEP_TEST_MIC, "recording…"); + + if let Err(e) = send_ipc(IpcCommand::StartRecording { assistive: false }) { + warn!("Bootstrap test mic failed to start: {}", e); + update_step_status(STEP_TEST_MIC, "failed"); + return; + } + + thread::spawn(|| { + thread::sleep(Duration::from_secs(3)); + let _ = send_ipc(IpcCommand::StopRecording); + update_step_status(STEP_TEST_MIC, "done"); + }); +} + +pub(super) fn handle_show_overlay() { + crate::show_voice_chat_overlay(); + crate::show_agent_tab(); + crate::voice_chat_ui::update_voice_chat_status("Listening..."); + update_step_status(STEP_SHOW_OVERLAY, "done"); +} + +pub(super) fn handle_hotkey_done() { + update_step_status(STEP_PRESS_HOTKEY, "done"); +} + +pub(super) fn handle_finish() { + mark_bootstrap_done(); + hide_bootstrap_overlay(); +} + +pub fn hide_bootstrap_overlay() { + Queue::main().exec_async(|| unsafe { + let mut state = BOOTSTRAP_STATE.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(window_ptr) = state.window.take() { + window_close(window_ptr as Id); + } + state.step_labels = [None, None, None]; + }); +} + +fn update_step_status(index: usize, text: &str) { + let text = text.to_string(); + Queue::main().exec_async(move || unsafe { + let state = BOOTSTRAP_STATE.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(label) = state.step_labels.get(index).and_then(|v| *v) { + set_text_field_string(label as Id, &text); + } + }); +} + +fn send_ipc(cmd: IpcCommand) -> Result { + let socket_path = crate::ipc::socket_path(); + let mut stream = + UnixStream::connect(socket_path).map_err(|e| format!("IPC connect failed: {e}"))?; + let payload = serde_json::to_string(&cmd).map_err(|e| e.to_string())?; + stream + .write_all(payload.as_bytes()) + .map_err(|e| e.to_string())?; + stream.write_all(b"\n").map_err(|e| e.to_string())?; + + let mut reader = BufReader::new(stream); + let mut line = String::new(); + reader.read_line(&mut line).map_err(|e| e.to_string())?; + + serde_json::from_str::(&line).map_err(|e| e.to_string()) +} diff --git a/app/ui/mod.rs b/app/ui/mod.rs index b4d24aea..046c7c88 100644 --- a/app/ui/mod.rs +++ b/app/ui/mod.rs @@ -1,3 +1,4 @@ +pub mod bootstrap; pub mod overlay; pub mod shared; pub mod tray; @@ -11,6 +12,7 @@ use core_foundation::base::TCFType; use core_foundation::string::CFString; use core_graphics::geometry::{CGPoint, CGRect, CGSize}; use dispatch::Queue; +use objc::runtime::Sel; use objc::runtime::{Class, Object}; use objc::{msg_send, sel, sel_impl}; use objc2_app_kit::{ @@ -19,6 +21,7 @@ use objc2_app_kit::{ use std::ptr; use crate::ui::shared::helpers::{add_subview, window_close, window_show}; +use crate::ui_helpers::ns_string; use std::sync::{Arc, Mutex}; use std::thread; use std::time::Duration; @@ -42,6 +45,7 @@ unsafe extern "C" { const AX_ERROR_SUCCESS: i32 = 0; const AX_FOCUSED_UIELEMENT_ATTRIBUTE: &str = "AXFocusedUIElement"; const AX_ROLE_ATTRIBUTE: &str = "AXRole"; +const AX_SELECTED_TEXT_ATTRIBUTE: &str = "AXSelectedText"; const AX_SELECTED_TEXT_RANGE_ATTRIBUTE: &str = "AXSelectedTextRange"; const AX_POSITION_ATTRIBUTE: &str = "AXPosition"; const AX_SIZE_ATTRIBUTE: &str = "AXSize"; @@ -329,6 +333,131 @@ pub fn get_caret_position() -> Option<(f64, f64)> { } } +/// Get currently selected text from the focused UI element (best-effort). +/// +/// Notes: +/// - Requires Accessibility permission. +/// - Many apps expose selected text via `AXSelectedText`, but not all. +/// - Returns `None` if there's no selection or the attribute isn't supported. +pub fn get_selected_text(max_chars: usize) -> Option { + unsafe { + let system_wide = AXUIElementCreateSystemWide(); + if system_wide.is_null() { + return None; + } + + let mut focused_element: AXId = ptr::null_mut(); + let attr_name = CFString::new(AX_FOCUSED_UIELEMENT_ATTRIBUTE); + let result = AXUIElementCopyAttributeValue( + system_wide, + attr_name.as_concrete_TypeRef() as AXId, + &mut focused_element, + ); + + CFRelease(system_wide); + + if result != AX_ERROR_SUCCESS || focused_element.is_null() { + return None; + } + + let mut selected_value: AXId = ptr::null_mut(); + let selected_attr = CFString::new(AX_SELECTED_TEXT_ATTRIBUTE); + let selected_result = AXUIElementCopyAttributeValue( + focused_element, + selected_attr.as_concrete_TypeRef() as AXId, + &mut selected_value, + ); + + CFRelease(focused_element); + + if selected_result != AX_ERROR_SUCCESS || selected_value.is_null() { + return None; + } + + let selected_str = CFString::wrap_under_get_rule(selected_value as *const _).to_string(); + CFRelease(selected_value); + + let mut s = selected_str.trim().to_string(); + if s.is_empty() { + return None; + } + + if max_chars > 0 && s.len() > max_chars { + s.truncate(max_chars); + s.push('…'); + } + + Some(s) + } +} + +/// Get the current selected text length (range length) from the focused UI element. +/// +/// Returns: +/// - `Some(0)` if the element supports `AXSelectedTextRange` but there's no selection +/// - `Some(n>0)` if there's a selection +/// - `None` if the attribute isn't available or any step fails +pub fn get_selected_text_length() -> Option { + unsafe { + let system_wide = AXUIElementCreateSystemWide(); + if system_wide.is_null() { + return None; + } + + let mut focused_element: AXId = ptr::null_mut(); + let attr_name = CFString::new(AX_FOCUSED_UIELEMENT_ATTRIBUTE); + let result = AXUIElementCopyAttributeValue( + system_wide, + attr_name.as_concrete_TypeRef() as AXId, + &mut focused_element, + ); + + CFRelease(system_wide); + + if result != AX_ERROR_SUCCESS || focused_element.is_null() { + return None; + } + + let mut range_value: AXId = ptr::null_mut(); + let range_attr = CFString::new(AX_SELECTED_TEXT_RANGE_ATTRIBUTE); + let range_result = AXUIElementCopyAttributeValue( + focused_element, + range_attr.as_concrete_TypeRef() as AXId, + &mut range_value, + ); + + CFRelease(focused_element); + + if range_result != AX_ERROR_SUCCESS || range_value.is_null() { + return None; + } + + #[repr(C)] + struct CFRange { + location: i64, + length: i64, + } + + let mut cf_range = CFRange { + location: 0, + length: 0, + }; + + let ok = AXValueGetValue( + range_value, + AX_VALUE_CFRANGE_TYPE, + &mut cf_range as *mut _ as *mut std::ffi::c_void, + ); + CFRelease(range_value); + + if !ok { + return None; + } + + Some(cf_range.length.max(0) as usize) + } +} + /// Get the current mouse cursor position in screen coordinates pub fn get_cursor_position() -> (f64, f64) { let mouse_location = NSEvent::mouseLocation(); @@ -673,6 +802,89 @@ pub fn set_dock_icon() { }); } +/// Install a minimal AppKit main menu with standard Edit key equivalents. +/// +/// CodeScribe runs as an `LSUIElement` agent app (no visible menu bar). In this mode AppKit still +/// relies on the app's `mainMenu` to resolve Command-key equivalents like Cmd+C / Cmd+V for text +/// controls (field editor). Without it, selectable text in bubbles and the Agent input field can +/// appear "dead" for copy/paste even though typing works. +/// +/// This is safe to call multiple times. +#[cfg(target_os = "macos")] +pub fn install_basic_edit_menu() { + use std::sync::Once; + + static INIT: Once = Once::new(); + INIT.call_once(|| { + Queue::main().exec_async(|| unsafe { + let ns_app_class = Class::get("NSApplication").expect("NSApplication class not found"); + let app: Id = msg_send![ns_app_class, sharedApplication]; + if app.is_null() { + warn!("install_basic_edit_menu: NSApplication sharedApplication is null"); + return; + } + + let ns_menu = Class::get("NSMenu").expect("NSMenu class not found"); + let ns_menu_item = Class::get("NSMenuItem").expect("NSMenuItem class not found"); + + let main_menu: Id = msg_send![ns_menu, alloc]; + let main_menu: Id = msg_send![main_menu, init]; + + // App menu (required for some key equivalent routing, even if hidden) + let app_item: Id = msg_send![ns_menu_item, alloc]; + let app_item: Id = msg_send![app_item, init]; + let app_menu: Id = msg_send![ns_menu, alloc]; + let app_menu: Id = msg_send![app_menu, init]; + + let quit_title = ns_string("Quit CodeScribe"); + let quit_key = ns_string("q"); + let quit_item: Id = msg_send![ns_menu_item, alloc]; + let quit_item: Id = + msg_send![quit_item, initWithTitle: quit_title action: sel!(terminate:) keyEquivalent: quit_key]; + let _: () = msg_send![app_menu, addItem: quit_item]; + let _: () = msg_send![app_item, setSubmenu: app_menu]; + let _: () = msg_send![main_menu, addItem: app_item]; + + // Edit menu with standard key equivalents + let edit_item: Id = msg_send![ns_menu_item, alloc]; + let edit_item: Id = msg_send![edit_item, init]; + let _: () = msg_send![edit_item, setTitle: ns_string("Edit")]; + + let edit_menu: Id = msg_send![ns_menu, alloc]; + let edit_menu: Id = msg_send![edit_menu, init]; + + let make_edit = |title: &str, sel: Sel, key: &str| -> Id { + let item: Id = msg_send![ns_menu_item, alloc]; + msg_send![ + item, + initWithTitle: ns_string(title) + action: sel + keyEquivalent: ns_string(key) + ] + }; + + let cut_item = make_edit("Cut", sel!(cut:), "x"); + let copy_item = make_edit("Copy", sel!(copy:), "c"); + let paste_item = make_edit("Paste", sel!(paste:), "v"); + let select_all_item = make_edit("Select All", sel!(selectAll:), "a"); + + let _: () = msg_send![edit_menu, addItem: cut_item]; + let _: () = msg_send![edit_menu, addItem: copy_item]; + let _: () = msg_send![edit_menu, addItem: paste_item]; + let _: () = msg_send![edit_menu, addItem: select_all_item]; + + let _: () = msg_send![edit_item, setSubmenu: edit_menu]; + let _: () = msg_send![main_menu, addItem: edit_item]; + + let _: () = msg_send![app, setMainMenu: main_menu]; + debug!("install_basic_edit_menu: mainMenu installed"); + }); + }); +} + +#[cfg(not(target_os = "macos"))] +pub fn install_basic_edit_menu() {} + #[cfg(test)] mod tests { use super::*; diff --git a/app/ui/overlay/mod.rs b/app/ui/overlay/mod.rs index b62182c2..a98969d0 100644 --- a/app/ui/overlay/mod.rs +++ b/app/ui/overlay/mod.rs @@ -31,7 +31,8 @@ use tracing::{debug, info, warn}; use crate::ui_helpers::{ add_subview, animate_fade, button_set_action, button_style, clamp_overlay_position, - color_white, create_button, set_hidden, set_text, window_close, window_set_alpha, window_show, + color_white, create_button, overlay_window_class, set_hidden, set_text, window_close, + window_set_alpha, window_show, }; use objc::declare::ClassDecl; use objc::runtime::Sel; @@ -525,6 +526,7 @@ fn update_overlay_text_and_layout(state: &mut TranscriptionOverlayState) { /// Show the transcription overlay window pub fn show_transcription_overlay() { + info!("show_transcription_overlay requested"); // Cancel any pending auto-hide AUTO_HIDE_GENERATION.fetch_add(1, Ordering::SeqCst); AUTO_HIDE_PENDING.store(false, Ordering::SeqCst); @@ -536,6 +538,7 @@ pub fn show_transcription_overlay() { fn show_transcription_overlay_impl() { unsafe { + info!("show_transcription_overlay_impl starting"); let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); // Reuse existing window if any @@ -576,7 +579,6 @@ fn show_transcription_overlay_impl() { return; } - let ns_window = ns_window_class.unwrap(); let ns_text_field = ns_text_field_class.unwrap(); let ns_screen = ns_screen_class.unwrap(); let ns_string = ns_string_class.unwrap(); @@ -642,7 +644,8 @@ fn show_transcription_overlay_impl() { }; // Create borderless window for modern look - let window: Id = msg_send![ns_window, alloc]; + let window_class = overlay_window_class(); + let window: Id = msg_send![window_class, alloc]; if window.is_null() { warn!("Failed to alloc NSWindow"); return; @@ -961,7 +964,7 @@ pub fn append_transcription_delta(delta: &str) { fn append_transcription_delta_impl(delta: &str) { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - state.accumulated_text.push_str(delta); + codescribe_core::contracts::TranscriptDelta::from_raw(delta).apply(&mut state.accumulated_text); update_overlay_text_and_layout(&mut state); } @@ -1102,6 +1105,7 @@ pub fn enter_recording_mode() { /// Hide the transcription overlay window (with fade-out animation) pub fn hide_transcription_overlay() { + info!("hide_transcription_overlay requested"); // Cancel any pending auto-hide AUTO_HIDE_PENDING.store(false, Ordering::SeqCst); @@ -1135,7 +1139,9 @@ fn hide_transcription_overlay_impl() { }); }); - debug!("Transcription overlay hidden"); + info!("Transcription overlay hidden"); + } else { + debug!("Transcription overlay hide: no window to close"); } state.text_field = None; state.status_field = None; diff --git a/app/ui/shared/helpers.rs b/app/ui/shared/helpers.rs index 74c6aefa..0c1c931a 100644 --- a/app/ui/shared/helpers.rs +++ b/app/ui/shared/helpers.rs @@ -11,10 +11,12 @@ use crate::os::clipboard; use core_graphics::geometry::{CGPoint, CGRect, CGSize}; -use objc::runtime::{Class, Object}; +use objc::declare::ClassDecl; +use objc::runtime::{Class, Object, Sel}; use objc::{msg_send, sel, sel_impl}; use objc2_app_kit::{NSBackingStoreType, NSWindowCollectionBehavior, NSWindowStyleMask}; use std::ffi::CString; +use std::sync::Once; /// Type alias for Objective-C object pointers pub type Id = *mut Object; @@ -23,6 +25,35 @@ pub type Id = *mut Object; pub const NS_FLOATING_WINDOW_LEVEL: i64 = 3; pub const NS_STATUS_WINDOW_LEVEL: i64 = 25; +// Custom overlay window class so borderless windows can receive input. +static OVERLAY_WINDOW_INIT: Once = Once::new(); +static mut OVERLAY_WINDOW_CLASS: *const Class = std::ptr::null(); + +extern "C" fn can_become_key(_this: &Object, _cmd: Sel) -> bool { + true +} + +/// Get a custom NSWindow subclass that can become key/main (for borderless overlays). +pub fn overlay_window_class() -> *const Class { + unsafe { + OVERLAY_WINDOW_INIT.call_once(|| { + let superclass = Class::get("NSWindow").expect("NSWindow not found"); + let mut decl = ClassDecl::new("CodeScribeOverlayWindow", superclass) + .expect("Failed to declare overlay window class"); + decl.add_method( + sel!(canBecomeKeyWindow), + can_become_key as extern "C" fn(&Object, Sel) -> bool, + ); + decl.add_method( + sel!(canBecomeMainWindow), + can_become_key as extern "C" fn(&Object, Sel) -> bool, + ); + OVERLAY_WINDOW_CLASS = decl.register(); + }); + OVERLAY_WINDOW_CLASS + } +} + // ============================================================================ // Color Helpers // ============================================================================ @@ -69,6 +100,16 @@ pub fn copy_to_clipboard(text: &str) { let _ = clipboard::copy(text); } +/// Set a tooltip on any NSView. +/// # Safety +/// `view` must be a valid Objective-C object that supports `setToolTip:`. +pub unsafe fn set_tooltip(view: Id, text: &str) { + unsafe { + let tip = ns_string(text); + let _: () = msg_send![view, setToolTip: tip]; + } +} + // ============================================================================ // Text Field Helpers // ============================================================================ @@ -497,6 +538,16 @@ pub unsafe fn window_show(window: Id) { } } +/// Hide window (order out) +/// # Safety +/// `window` must be a valid `NSWindow` instance. +pub unsafe fn window_hide(window: Id) { + unsafe { + let nil: *mut Object = std::ptr::null_mut(); + let _: () = msg_send![window, orderOut: nil]; + } +} + /// Close window /// # Safety /// `window` must be a valid `NSWindow` instance. @@ -682,21 +733,32 @@ pub unsafe fn set_enabled(view: Id, enabled: bool) { } // ============================================================================ -// Chat Bubble Helpers (loct.io dark theme) +// Chat Bubble Helpers (GlyphPulse / Quantum style) // ============================================================================ -/// loct.io brand colors for dark theme +/// GlyphPulse/Quantum palette adapted for native bubbles pub mod bubble_colors { - /// User bubble background - muted violet/purple accent - pub const USER_BG: (f64, f64, f64, f64) = (0.35, 0.28, 0.55, 0.9); - /// Assistant bubble background - dark gray - pub const ASSISTANT_BG: (f64, f64, f64, f64) = (0.22, 0.22, 0.24, 0.9); - /// System/error bubble background - dark red tint - pub const ERROR_BG: (f64, f64, f64, f64) = (0.4, 0.2, 0.2, 0.9); - /// Text color - white - pub const TEXT: (f64, f64, f64, f64) = (1.0, 1.0, 1.0, 1.0); - /// Streaming indicator color - subtle pulse - pub const STREAMING: (f64, f64, f64, f64) = (0.6, 0.6, 0.6, 1.0); + /// User bubble background - quantum cyan + pub const USER_BG: (f64, f64, f64, f64) = (0.0, 1.0, 1.0, 1.0); + /// User bubble text - CRT black + pub const USER_TEXT: (f64, f64, f64, f64) = (0.039, 0.039, 0.039, 1.0); + /// Assistant bubble background - deep navy glass + pub const ASSISTANT_BG: (f64, f64, f64, f64) = (0.086, 0.129, 0.243, 0.5); + /// Assistant bubble border - subtle white + pub const ASSISTANT_BORDER: (f64, f64, f64, f64) = (1.0, 1.0, 1.0, 0.08); + /// Assistant/system text - primary light + pub const ASSISTANT_TEXT: (f64, f64, f64, f64) = (0.886, 0.91, 0.941, 1.0); + /// Streaming text tint - muted + pub const STREAMING_TEXT: (f64, f64, f64, f64) = (0.58, 0.639, 0.722, 1.0); + /// System bubble background - slightly denser navy + pub const SYSTEM_BG: (f64, f64, f64, f64) = (0.086, 0.129, 0.243, 0.65); + /// System bubble border - subtle white + pub const SYSTEM_BORDER: (f64, f64, f64, f64) = (1.0, 1.0, 1.0, 0.08); + /// Error bubble background - soft red tint + pub const ERROR_BG: (f64, f64, f64, f64) = (1.0, 0.42, 0.42, 0.1); + /// Error bubble border/text - error red + pub const ERROR_BORDER: (f64, f64, f64, f64) = (1.0, 0.373, 0.341, 1.0); + pub const ERROR_TEXT: (f64, f64, f64, f64) = (1.0, 0.373, 0.341, 1.0); } /// Role for chat bubble styling @@ -729,25 +791,87 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { let ns_text_field = Class::get("NSTextField").unwrap(); let ns_color = Class::get("NSColor").unwrap(); let ns_font = Class::get("NSFont").unwrap(); + let ns_dict = Class::get("NSDictionary").unwrap(); - // Calculate text dimensions (approximate) let font_size = 13.0; - let padding = 12.0; + let padding_x = 12.0; + let padding_top = 10.0; let copy_button_height = if config.message_index.is_some() { - 20.0 + 16.0 } else { 0.0 }; - let chars_per_line = (config.max_width - padding * 2.0) / (font_size * 0.6); - let text_len = config.text.len() as f64; - let estimated_lines = (text_len / chars_per_line).ceil().max(1.0); + // Reserve space for the Copy button so it never overlaps text. + let padding_bottom = if copy_button_height > 0.0 { + copy_button_height + 8.0 + } else { + 10.0 + }; let line_height = font_size * 1.4; - let text_height = estimated_lines * line_height; - let bubble_height = text_height + padding * 2.0 + copy_button_height; - // Bubble width: content-aware but capped - let content_width = (text_len * font_size * 0.6).min(config.max_width - padding * 2.0); - let bubble_width = content_width + padding * 2.0; + // Font (prefer JetBrains Mono if installed) + let jb_name = ns_string("JetBrainsMono-Regular"); + let jb_font: Id = msg_send![ns_font, fontWithName: jb_name size: font_size]; + let font: Id = if jb_font.is_null() { + msg_send![ns_font, monospacedSystemFontOfSize: font_size weight: 0.0f64] + } else { + jb_font + }; + + // Set text (with streaming indicator if needed) + let display_text = if config.is_streaming && config.text.is_empty() { + "• • •".to_string() // Pulsing dots placeholder + } else if config.is_streaming { + format!("{} …", config.text) + } else { + config.text.clone() + }; + + // Measure text height/width using NSString boundingRectWithSize (handles newlines/wrapping). + // + // NOTE: `NSFontAttributeName` (key) has the string value "NSFont". AppKit expects that + // key, not the literal "NSFontAttributeName" string. + let text_str = ns_string(&display_text); + let font_key = ns_string("NSFont"); + let attrs: Id = msg_send![ns_dict, dictionaryWithObject: font forKey: font_key]; + let opts: u64 = 1 | 2; // NSStringDrawingUsesLineFragmentOrigin | NSStringDrawingUsesFontLeading + + // Keep a small side margin inside the container so full-width bubbles don't overflow. + let bubble_max_width = (config.max_width - 16.0).max(80.0); + let text_max_width = (bubble_max_width - padding_x * 2.0).max(40.0); + let rect_max: CGRect = msg_send![ + text_str, + boundingRectWithSize: CGSize::new(text_max_width, 10_000.0) + options: opts + attributes: attrs + ]; + + // Bubble width: content-aware but capped. + // If it wraps (or is long), keep the bubble full width for readability. + // + // We treat streaming messages as "wrap-prone" earlier to avoid the initial narrow bubble + // that later expands mid-stream. + let long_threshold = if config.is_streaming { 30 } else { 80 }; + let is_long = display_text.chars().count() > long_threshold; + let wraps_at_max = + rect_max.size.height > line_height * 1.6 || display_text.contains('\n') || is_long; + let bubble_width = if wraps_at_max { + bubble_max_width + } else { + let content_width = rect_max.size.width.min(text_max_width).max(1.0); + (content_width + padding_x * 2.0).min(bubble_max_width) + }; + + // Re-measure height for the final layout width (important when bubble_width < max). + let text_layout_width = (bubble_width - padding_x * 2.0).max(40.0); + let text_rect: CGRect = msg_send![ + text_str, + boundingRectWithSize: CGSize::new(text_layout_width, 10_000.0) + options: opts + attributes: attrs + ]; + let text_height = text_rect.size.height.ceil().max(line_height); + let bubble_height = text_height + padding_top + padding_bottom; // Container view (for alignment) let container: Id = msg_send![ns_view, alloc]; @@ -760,8 +884,8 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { // Bubble background view let bubble: Id = msg_send![ns_view, alloc]; let bubble_x = match config.role { - BubbleRole::User => config.max_width - bubble_width - 8.0, // Right-aligned - BubbleRole::Assistant | BubbleRole::System => 8.0, // Left-aligned + BubbleRole::User => (config.max_width - bubble_width - 8.0).max(8.0), // Right-aligned + BubbleRole::Assistant | BubbleRole::System => 8.0, // Left-aligned }; let bubble_frame = CGRect::new( &CGPoint::new(bubble_x, 0.0), @@ -776,7 +900,7 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { match config.role { BubbleRole::User => bubble_colors::USER_BG, BubbleRole::Assistant => bubble_colors::ASSISTANT_BG, - BubbleRole::System => bubble_colors::ERROR_BG, + BubbleRole::System => bubble_colors::SYSTEM_BG, } }; let bg_color: Id = msg_send![ns_color, colorWithRed: r green: g blue: b alpha: a]; @@ -789,12 +913,48 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { let cg_color: Id = msg_send![bg_color, CGColor]; let _: () = msg_send![layer, setBackgroundColor: cg_color]; let _: () = msg_send![layer, setCornerRadius: 12.0f64]; + let _: () = msg_send![layer, setMasksToBounds: false]; + // Border styling + let (br, bg, bb, ba, bw) = if config.is_error { + ( + bubble_colors::ERROR_BORDER.0, + bubble_colors::ERROR_BORDER.1, + bubble_colors::ERROR_BORDER.2, + bubble_colors::ERROR_BORDER.3, + 1.0f64, + ) + } else { + match config.role { + BubbleRole::Assistant => ( + bubble_colors::ASSISTANT_BORDER.0, + bubble_colors::ASSISTANT_BORDER.1, + bubble_colors::ASSISTANT_BORDER.2, + bubble_colors::ASSISTANT_BORDER.3, + 1.0f64, + ), + BubbleRole::System => ( + bubble_colors::SYSTEM_BORDER.0, + bubble_colors::SYSTEM_BORDER.1, + bubble_colors::SYSTEM_BORDER.2, + bubble_colors::SYSTEM_BORDER.3, + 1.0f64, + ), + BubbleRole::User => (0.0, 0.0, 0.0, 0.0, 0.0f64), + } + }; + if bw > 0.0 { + let border_color: Id = + msg_send![ns_color, colorWithRed: br green: bg blue: bb alpha: ba]; + let cg_border: Id = msg_send![border_color, CGColor]; + let _: () = msg_send![layer, setBorderColor: cg_border]; + let _: () = msg_send![layer, setBorderWidth: bw]; + } } // Text label inside bubble let text_frame = CGRect::new( - &CGPoint::new(padding, padding / 2.0), - &CGSize::new(bubble_width - padding * 2.0, text_height), + &CGPoint::new(padding_x, padding_bottom), + &CGSize::new((bubble_width - padding_x * 2.0).max(1.0), text_height), ); let text_label: Id = msg_send![ns_text_field, alloc]; let text_label: Id = msg_send![text_label, initWithFrame: text_frame]; @@ -804,28 +964,27 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { let _: () = msg_send![text_label, setSelectable: true]; let _: () = msg_send![text_label, setDrawsBackground: false]; - // Text color - let (tr, tg, tb, ta) = if config.is_streaming { - bubble_colors::STREAMING + // Text color (role-aware) + let (tr, tg, tb, ta) = if config.is_error { + bubble_colors::ERROR_TEXT } else { - bubble_colors::TEXT + match config.role { + BubbleRole::User => bubble_colors::USER_TEXT, + BubbleRole::Assistant => { + if config.is_streaming { + bubble_colors::STREAMING_TEXT + } else { + bubble_colors::ASSISTANT_TEXT + } + } + BubbleRole::System => bubble_colors::ASSISTANT_TEXT, + } }; let text_color: Id = msg_send![ns_color, colorWithRed: tr green: tg blue: tb alpha: ta]; let _: () = msg_send![text_label, setTextColor: text_color]; - // Font - let font: Id = msg_send![ns_font, systemFontOfSize: font_size]; let _: () = msg_send![text_label, setFont: font]; - // Set text (with streaming indicator if needed) - let display_text = if config.is_streaming && config.text.is_empty() { - "• • •".to_string() // Pulsing dots placeholder - } else if config.is_streaming { - format!("{} …", config.text) - } else { - config.text.clone() - }; - let text_str = ns_string(&display_text); let _: () = msg_send![text_label, setStringValue: text_str]; // Word wrap @@ -839,9 +998,9 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { let ns_button = Class::get("NSButton").unwrap(); let button_width = 40.0; - let button_height = 16.0; - let button_x = bubble_width - button_width - padding / 2.0; - let button_y = 2.0; // Bottom of bubble + let button_height = copy_button_height; + let button_x = bubble_width - button_width - padding_x; + let button_y = 4.0; // Bottom of bubble let button_frame = CGRect::new( &CGPoint::new(button_x, button_y), @@ -859,12 +1018,16 @@ pub fn create_bubble_view(config: BubbleConfig) -> (Id, Id) { let title = ns_string("Copy"); let _: () = msg_send![copy_button, setTitle: title]; - let small_font: Id = msg_send![ns_font, systemFontOfSize: 10.0f64]; + let small_font: Id = if jb_font.is_null() { + msg_send![ns_font, monospacedSystemFontOfSize: 10.0f64 weight: 0.0f64] + } else { + msg_send![ns_font, fontWithName: jb_name size: 10.0f64] + }; let _: () = msg_send![copy_button, setFont: small_font]; - // Subtle text color + // Match bubble text tint let button_color: Id = - msg_send![ns_color, colorWithRed: 0.7f64 green: 0.7f64 blue: 0.7f64 alpha: 1.0f64]; + msg_send![ns_color, colorWithRed: tr green: tg blue: tb alpha: ta]; let _: () = msg_send![copy_button, setContentTintColor: button_color]; // Store message index in tag for retrieval on click @@ -901,17 +1064,178 @@ pub unsafe fn update_bubble_text(text_label: Id, text: &str, is_streaming: bool) let text_str = ns_string(&display_text); let _: () = msg_send![text_label, setStringValue: text_str]; - // Update text color based on streaming state + // Update text color based on streaming state (assistant defaults) let (tr, tg, tb, ta) = if is_streaming { - bubble_colors::STREAMING + bubble_colors::STREAMING_TEXT } else { - bubble_colors::TEXT + bubble_colors::ASSISTANT_TEXT }; let text_color: Id = msg_send![ns_color, colorWithRed: tr green: tg blue: tb alpha: ta]; let _: () = msg_send![text_label, setTextColor: text_color]; } } +/// Update a stack view item (bubble container) height constraint if present. +/// +/// `stack_view_add` installs a fixed-height constraint on each arranged subview. +/// During streaming, the bubble text grows and we need to update that constraint +/// so the view doesn't clip. +/// +/// # Safety +/// `view` must be a valid `NSView` instance. +pub unsafe fn update_stack_item_height(view: Id, new_height: f64) { + unsafe { + let constraints: Id = msg_send![view, constraints]; + if constraints.is_null() { + return; + } + let count: usize = msg_send![constraints, count]; + for i in 0..count { + let c: Id = msg_send![constraints, objectAtIndex: i]; + if c.is_null() { + continue; + } + + // Prefer our tagged constraint. + let ident: Id = msg_send![c, identifier]; + if !ident.is_null() { + let c_str: *const i8 = msg_send![ident, UTF8String]; + if !c_str.is_null() { + let s = std::ffi::CStr::from_ptr(c_str).to_string_lossy(); + if s == "codescribe_height" { + let _: () = msg_send![c, setConstant: new_height]; + return; + } + } + } + + // Fallback: find a height constraint on this view. + let first: Id = msg_send![c, firstItem]; + if first != view { + continue; + } + let second: Id = msg_send![c, secondItem]; + if !second.is_null() { + continue; + } + let first_attr: isize = msg_send![c, firstAttribute]; + // NSLayoutAttributeHeight == 8 + if first_attr == 8 { + let _: () = msg_send![c, setConstant: new_height]; + return; + } + } + } +} + +/// Resize an existing bubble container + its internal views for the given text. +/// +/// Used for streaming updates to prevent clipping without rebuilding the whole view tree. +/// +/// # Safety +/// `container` must be the container returned by `create_bubble_view`. +/// `text_label` must be the label returned by `create_bubble_view`. +pub unsafe fn resize_bubble_container_for_text(container: Id, text_label: Id, display_text: &str) { + unsafe { + let ns_dict = Class::get("NSDictionary").unwrap(); + let ns_font = Class::get("NSFont").unwrap(); + + let font: Id = msg_send![text_label, font]; + let font = if font.is_null() { + msg_send![ns_font, systemFontOfSize: 13.0f64] + } else { + font + }; + + let container_frame: CGRect = msg_send![container, frame]; + let max_width = container_frame.size.width.max(80.0); + let bubble_max_width = (max_width - 16.0).max(80.0); + + // If the message is getting long, switch to full-width to avoid one-word-per-line bubbles. + // + // During streaming we append " …" so we can detect it and widen earlier to prevent + // the initial narrow bubble phase. + let streaming_like = display_text.ends_with('…'); + let long_threshold = if streaming_like { 30 } else { 80 }; + let is_long = display_text.chars().count() > long_threshold; + let force_full_width = display_text.contains('\n') || is_long; + + let label_frame: CGRect = msg_send![text_label, frame]; + let width = if force_full_width { + let padding_x = 12.0; + (bubble_max_width - padding_x * 2.0).max(40.0) + } else { + label_frame.size.width.max(1.0) + }; + + let text_str = ns_string(display_text); + let font_key = ns_string("NSFont"); + let attrs: Id = msg_send![ns_dict, dictionaryWithObject: font forKey: font_key]; + let opts: u64 = 1 | 2; // NSStringDrawingUsesLineFragmentOrigin | NSStringDrawingUsesFontLeading + let text_rect: CGRect = msg_send![ + text_str, + boundingRectWithSize: CGSize::new(width, 10_000.0) + options: opts + attributes: attrs + ]; + + // Approximate line-height floor to avoid tiny/bad measurements. + let point_size: f64 = msg_send![font, pointSize]; + let line_height = (point_size * 1.35).max(14.0); + + let text_height = text_rect.size.height.ceil().max(line_height); + + // Match `create_bubble_view` layout constants. + let padding_top = 10.0; + let copy_button_height = 16.0; + let padding_bottom = copy_button_height + 8.0; + let bubble_height = text_height + padding_top + padding_bottom; + + // Resize bubble background view (label's superview). + let bubble: Id = msg_send![text_label, superview]; + if !bubble.is_null() { + let bubble_frame: CGRect = msg_send![bubble, frame]; + let mut bubble_width = bubble_frame.size.width; + let mut bubble_x = bubble_frame.origin.x; + + if force_full_width { + bubble_width = bubble_max_width; + // Preserve alignment based on prior x (user bubbles are right-aligned). + let was_right_aligned = bubble_x > 20.0; + bubble_x = if was_right_aligned { + (max_width - bubble_width - 8.0).max(8.0) + } else { + 8.0 + }; + } + + // Resize label to match bubble width (keep in sync with create_bubble_view). + let padding_x = 12.0; + let new_label_w = (bubble_width - padding_x * 2.0).max(1.0); + let new_label_frame = CGRect::new( + &CGPoint::new(padding_x, padding_bottom), + &CGSize::new(new_label_w, text_height), + ); + let _: () = msg_send![text_label, setFrame: new_label_frame]; + + let new_bubble_frame = CGRect::new( + &CGPoint::new(bubble_x, bubble_frame.origin.y), + &CGSize::new(bubble_width, bubble_height), + ); + let _: () = msg_send![bubble, setFrame: new_bubble_frame]; + let _: () = msg_send![bubble, setNeedsDisplay: true]; + } + + // Resize container (stack arranged subview). + let _: () = msg_send![container, setFrameSize: CGSize::new(container_frame.size.width, bubble_height)]; + update_stack_item_height(container, bubble_height); + + let _: () = msg_send![container, setNeedsLayout: true]; + let _: () = msg_send![container, layoutSubtreeIfNeeded]; + let _: () = msg_send![container, setNeedsDisplay: true]; + } +} + // ============================================================================ // File Operations Helpers // ============================================================================ @@ -986,6 +1310,15 @@ pub fn create_vertical_stack_view(frame: CGRect) -> Id { pub unsafe fn stack_view_add(stack: Id, view: Id) { unsafe { let _: () = msg_send![stack, addArrangedSubview: view]; + + // Pin height to the initial frame height (good enough for our chat bubbles/cards). + let frame: CGRect = msg_send![view, frame]; + let height_anchor: Id = msg_send![view, heightAnchor]; + let height_constraint: Id = + msg_send![height_anchor, constraintEqualToConstant: frame.size.height]; + // Tag for later updates (streaming bubbles grow). + let _: () = msg_send![height_constraint, setIdentifier: ns_string("codescribe_height")]; + let _: () = msg_send![height_constraint, setActive: true]; } } diff --git a/app/ui/tray/handlers.rs b/app/ui/tray/handlers.rs index f133eccd..afd1bfc1 100644 --- a/app/ui/tray/handlers.rs +++ b/app/ui/tray/handlers.rs @@ -19,6 +19,8 @@ pub fn handle_menu_event(event_id: &MenuId, menu_ids: &MenuIds) { handle_copy_last(); } else if event_id == &menu_ids.show_overlay { crate::show_voice_chat_overlay(); + } else if event_id == &menu_ids.run_onboarding { + crate::show_bootstrap_overlay(); } else if event_id == &menu_ids.open_history { handle_open_history_folder(); } else if event_id == &menu_ids.help { diff --git a/app/ui/tray/menu.rs b/app/ui/tray/menu.rs index a1ed8ae0..3be74039 100644 --- a/app/ui/tray/menu.rs +++ b/app/ui/tray/menu.rs @@ -69,6 +69,11 @@ pub fn build_menu() -> Result<(Menu, MenuIds)> { let show_overlay_id = show_overlay_item.id().clone(); menu.append(&show_overlay_item)?; + // 2b. Run onboarding + let run_onboarding_item = MenuItem::new("Run Onboarding", true, None); + let run_onboarding_id = run_onboarding_item.id().clone(); + menu.append(&run_onboarding_item)?; + // 4. Separator menu.append(&PredefinedMenuItem::separator())?; @@ -138,6 +143,7 @@ pub fn build_menu() -> Result<(Menu, MenuIds)> { MenuIds { copy_last: copy_last_id, show_overlay: show_overlay_id, + run_onboarding: run_onboarding_id, open_history: open_history_id, help: help_id, about: about_id, diff --git a/app/ui/tray/types.rs b/app/ui/tray/types.rs index 4fc7b0f6..89983879 100644 --- a/app/ui/tray/types.rs +++ b/app/ui/tray/types.rs @@ -75,6 +75,9 @@ pub enum TrayMenuEvent { /// User clicked Quit - clean shutdown Quit, + /// Run onboarding (bootstrap) flow + RunOnboarding, + // Hold Hotkeys submenu SetHoldMods(HoldMods), ToggleHoldExclusive, @@ -115,6 +118,7 @@ pub struct MenuIds { // Top-level pub copy_last: MenuId, pub show_overlay: MenuId, + pub run_onboarding: MenuId, pub open_history: MenuId, pub help: MenuId, pub about: MenuId, diff --git a/app/ui/voice_chat/api.rs b/app/ui/voice_chat/api.rs index 9a5f6fca..e1fc8843 100644 --- a/app/ui/voice_chat/api.rs +++ b/app/ui/voice_chat/api.rs @@ -3,7 +3,7 @@ //! Contains all the public functions for controlling the overlay and //! internal helper functions for state updates. -use core_graphics::geometry::CGPoint; +use core_graphics::geometry::{CGPoint, CGRect}; use dispatch::Queue; use objc::runtime::{Class, Object}; use objc::{msg_send, sel, sel_impl}; @@ -12,8 +12,9 @@ use tracing::{debug, info, warn}; use crate::ui_helpers::{ BubbleConfig, BubbleRole, create_bubble_view, create_card_view, get_text_field_string, - list_draft_files, ns_string, open_file_in_editor, set_text_field_string, stack_view_add, - stack_view_clear, + get_text_view_string, list_draft_files, ns_string, open_file_in_editor, + resize_bubble_container_for_text, set_text_field_string, set_text_view_string, stack_view_add, + stack_view_clear, update_bubble_text, }; use super::handlers::{clear_search_field, copy_to_clipboard}; @@ -37,6 +38,22 @@ pub fn update_voice_chat_status(status: &str) { }); } +/// Append a delta to the user draft message (streaming transcription) +pub fn append_voice_chat_user_delta(delta: &str) { + let delta_owned = delta.to_string(); + Queue::main().exec_async(move || { + append_voice_chat_user_delta_impl(&delta_owned); + }); +} + +/// Finalize the user message text (stop streaming) +pub fn set_voice_chat_user_text(text: &str) { + let text_owned = text.to_string(); + Queue::main().exec_async(move || { + finalize_user_message_impl(&text_owned); + }); +} + /// Append a delta to the assistant response (streaming) pub fn append_voice_chat_assistant_delta(delta: &str) { let delta_owned = delta.to_string(); @@ -58,7 +75,7 @@ pub fn add_voice_chat_error_message(text: &str) { let text_owned = text.to_string(); Queue::main().exec_async(move || { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - state.messages.push(ChatMessage { + state.push_message(ChatMessage { role: ChatRole::System, text: text_owned.clone(), is_streaming: false, @@ -75,7 +92,7 @@ pub fn add_voice_chat_user_message(text: &str) { let text_owned = text.to_string(); Queue::main().exec_async(move || { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - state.messages.push(ChatMessage { + state.push_message(ChatMessage { role: ChatRole::User, text: text_owned, is_streaming: false, @@ -138,6 +155,7 @@ pub fn reset_voice_chat_activity() { /// Hide the voice chat overlay window pub fn hide_voice_chat_overlay() { + info!("hide_voice_chat_overlay requested"); Queue::main().exec_async(|| { hide_voice_chat_overlay_impl(); }); @@ -174,6 +192,14 @@ pub fn show_drawer_tab() { }); } +/// Store the name of the app the user was in when starting assistive mode. +pub fn set_voice_chat_target_app(app_name: Option) { + Queue::main().exec_async(move || { + let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); + state.last_target_app = app_name; + }); +} + /// Update the conversation mode state (Moshi full-duplex indicators) pub fn update_conversation_state(new_state: ConversationModeState) { Queue::main().exec_async(move || { @@ -222,17 +248,25 @@ pub fn update_active_tab_impl(tab: Tab) { if let Some(agent_view) = state.agent_scroll_view { crate::ui_helpers::set_hidden(agent_view as Id, show_drawer); } - if let Some(agent_input) = state.agent_input_field { - let superview: Id = msg_send![agent_input as Id, superview]; - if !superview.is_null() { - crate::ui_helpers::set_hidden(superview, show_drawer); - } else { - crate::ui_helpers::set_hidden(agent_input as Id, show_drawer); - } + if let Some(agent_input_bar) = state.agent_input_bar { + crate::ui_helpers::set_hidden(agent_input_bar as Id, show_drawer); } if let Some(agent_send) = state.agent_send_button { crate::ui_helpers::set_hidden(agent_send as Id, show_drawer); } + + // When switching to Agent, make sure the input field can actually receive text. + // We do NOT force activation (to avoid stealing focus), but if the window is already + // key, we nudge first responder to the input field for better UX. + if tab == Tab::Agent + && let (Some(window_ptr), Some(input_ptr)) = (state.window, state.agent_input_text_view) + { + let window = window_ptr as Id; + let is_key: bool = msg_send![window, isKeyWindow]; + if is_key { + let _: bool = msg_send![window, makeFirstResponder: input_ptr as Id]; + } + } } } @@ -247,14 +281,77 @@ fn update_voice_chat_status_impl(status: &str) { } } +fn append_voice_chat_user_delta_impl(delta: &str) { + let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); + ensure_streaming_user_message(&mut state); + if let Some(last) = state.messages.last_mut() { + codescribe_core::contracts::TranscriptDelta::from_raw(delta).apply(&mut last.text); + last.is_streaming = true; + } + update_chat_view_with_state(&mut state, false); +} + fn append_voice_chat_assistant_delta_impl(delta: &str) { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); ensure_streaming_assistant_message(&mut state); if let Some(last) = state.messages.last_mut() { - last.text.push_str(delta); + codescribe_core::contracts::TranscriptDelta::from_raw(delta).apply(&mut last.text); last.is_streaming = true; } - update_chat_view_with_state(&mut state, false); + if !try_update_last_message_view_in_place(&mut state) { + update_chat_view_with_state(&mut state, false); + } +} + +fn display_text_for_message(message: &ChatMessage) -> String { + if message.is_streaming && message.text.is_empty() { + "• • •".to_string() + } else if message.is_streaming { + format!("{} …", message.text) + } else { + message.text.clone() + } +} + +fn try_update_last_message_view_in_place(state: &mut VoiceChatOverlayState) -> bool { + unsafe { + // If the view list doesn't match messages, a full rebuild is safer. + if state.agent_bubble_views.len() != state.messages.len() { + return false; + } + + let Some(last_message) = state.messages.last() else { + return false; + }; + let Some((bubble_ptr, label_ptr)) = state.agent_bubble_views.last().copied() else { + return false; + }; + + let container = bubble_ptr as Id; + let label = label_ptr as Id; + update_bubble_text(label, &last_message.text, last_message.is_streaming); + let display_text = display_text_for_message(last_message); + resize_bubble_container_for_text(container, label, &display_text); + + // Keep the latest message in view while streaming. + if let Some(scroll_view_ptr) = state.agent_scroll_view { + let _ = scroll_view_ptr; + let bounds: CGRect = msg_send![container, bounds]; + let _: () = msg_send![container, scrollRectToVisible: bounds]; + } + true + } +} + +fn finalize_user_message_impl(text: &str) { + let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); + ensure_streaming_user_message(&mut state); + if let Some(last) = state.messages.last_mut() { + last.text = text.to_string(); + last.is_streaming = false; + last.is_error = false; + } + update_chat_view_with_state(&mut state, true); } fn finalize_assistant_message_impl(text: &str, is_error: bool) { @@ -276,10 +373,10 @@ pub(super) fn clear_voice_chat_text_impl() { state.manual_draft.clear(); state.is_sending = false; - if let Some(input_field) = state.agent_input_field { - unsafe { - set_text_field_string(input_field as Id, ""); - } + if let Some(input_view) = state.agent_input_text_view { + unsafe { set_text_view_string(input_view as Id, "") }; + } else if let Some(input_field) = state.agent_input_field { + unsafe { set_text_field_string(input_field as Id, "") }; } update_chat_view_with_state(&mut state, true); @@ -287,18 +384,25 @@ pub(super) fn clear_voice_chat_text_impl() { } /// Send the draft message (called from handlers) +/// +/// SAFETY: OVERLAY_STATE must be fully released before invoking the send +/// callback, which may re-acquire the lock from another thread/queue. pub fn send_draft_message_impl() { - let callback = { + // Phase 1: extract draft and update state (OVERLAY_STATE held) + let draft = { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - let Some(input_field) = state.agent_input_field else { + let draft = if let Some(text_view) = state.agent_input_text_view { + unsafe { get_text_view_string(text_view as Id) } + } else if let Some(input_field) = state.agent_input_field { + unsafe { get_text_field_string(input_field as Id) } + } else { return; }; - let draft = unsafe { get_text_field_string(input_field as Id) }; let draft = draft.trim().to_string(); if draft.is_empty() { return; } - state.messages.push(ChatMessage { + state.push_message(ChatMessage { role: ChatRole::User, text: draft.clone(), is_streaming: false, @@ -306,16 +410,25 @@ pub fn send_draft_message_impl() { }); state.manual_draft.clear(); state.is_sending = true; - unsafe { - set_text_field_string(input_field as Id, ""); + if let Some(text_view) = state.agent_input_text_view { + unsafe { set_text_view_string(text_view as Id, "") }; + } else if let Some(input_field) = state.agent_input_field { + unsafe { set_text_field_string(input_field as Id, "") }; } update_chat_view_with_state(&mut state, true); update_send_button_with_state(&mut state); - let handler = SEND_CALLBACK.lock().unwrap_or_else(|e| e.into_inner()); - (handler.clone(), draft) + draft + // OVERLAY_STATE released here + }; + + // Phase 2: read callback (separate lock scope) + let handler = { + let guard = SEND_CALLBACK.lock().unwrap_or_else(|e| e.into_inner()); + guard.clone() }; - if let (Some(handler), draft) = callback { + // Phase 3: invoke callback with NO locks held + if let Some(handler) = handler { handler(draft); } else { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); @@ -325,7 +438,8 @@ pub fn send_draft_message_impl() { } pub(super) fn commit_last_user_message_impl() { - let callback = { + // Phase 1: extract text and mark sending (OVERLAY_STATE held) + let text = { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); let Some(last_message) = state.messages.last() else { return; @@ -337,11 +451,18 @@ pub(super) fn commit_last_user_message_impl() { state.is_sending = true; update_chat_view_with_state(&mut state, true); update_send_button_with_state(&mut state); - let handler = SEND_CALLBACK.lock().unwrap_or_else(|e| e.into_inner()); - (handler.clone(), text) + text + // OVERLAY_STATE released here + }; + + // Phase 2: read callback (separate lock scope) + let handler = { + let guard = SEND_CALLBACK.lock().unwrap_or_else(|e| e.into_inner()); + guard.clone() }; - if let (Some(handler), text) = callback { + // Phase 3: invoke callback with NO locks held + if let Some(handler) = handler { handler(text); } else { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); @@ -363,7 +484,7 @@ fn ensure_streaming_assistant_message(state: &mut VoiceChatOverlayState) { .last() .is_none_or(|msg| msg.role != ChatRole::Assistant || !msg.is_streaming); if needs_new { - state.messages.push(ChatMessage { + state.push_message(ChatMessage { role: ChatRole::Assistant, text: String::new(), is_streaming: true, @@ -372,6 +493,21 @@ fn ensure_streaming_assistant_message(state: &mut VoiceChatOverlayState) { } } +fn ensure_streaming_user_message(state: &mut VoiceChatOverlayState) { + let needs_new = state + .messages + .last() + .is_none_or(|msg| msg.role != ChatRole::User || !msg.is_streaming); + if needs_new { + state.push_message(ChatMessage { + role: ChatRole::User, + text: String::new(), + is_streaming: true, + is_error: false, + }); + } +} + pub(super) fn update_chat_view_with_state( state: &mut VoiceChatOverlayState, scroll_to_bottom: bool, @@ -396,11 +532,7 @@ pub(super) fn update_chat_view_with_state( max_width: 390.0, is_streaming: message.is_streaming, is_error: message.is_error, - message_index: if message.role == ChatRole::Assistant { - Some(index) - } else { - None - }, + message_index: Some(index), copy_action_target: state.action_handler.map(|p| p as Id), }); stack_view_add(container, bubble); @@ -431,11 +563,24 @@ fn update_send_button_with_state(state: &mut VoiceChatOverlayState) { unsafe { if let Some(button_ptr) = state.agent_send_button { let btn = button_ptr as Id; - let enabled = !state.is_sending && state.auto_send_enabled; + // In auto-send mode: button shows "Auto" and is disabled (voice input sends + // automatically). In draft mode: button shows ">" and the user clicks to send. + let enabled = !state.is_sending; let _: () = msg_send![btn, setEnabled: enabled]; - let title = if state.is_sending { "…" } else { ">" }; - let title = ns_string(title); - let _: () = msg_send![btn, setTitle: title]; + let title = if state.is_sending { + "…" + } else if state.auto_send_enabled { + "Auto" + } else { + ">" + }; + let label = if state.auto_send_enabled { + "Auto-send enabled" + } else { + "Send message" + }; + let _: () = msg_send![btn, setTitle: ns_string(title)]; + let _: () = msg_send![btn, setAccessibilityLabel: ns_string(label)]; } } } @@ -501,9 +646,12 @@ fn hide_voice_chat_overlay_impl() { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); if let Some(window_ptr) = state.window { let window = window_ptr as Id; + info!("Voice chat overlay hide: closing window"); crate::ui_helpers::animate_fade(window, 0.0, 0.15); crate::ui_helpers::window_close(window); clear_overlay_state(&mut state); + } else { + debug!("Voice chat overlay hide: no window to close"); } } clear_search_field(); @@ -523,6 +671,9 @@ pub fn clear_overlay_state(state: &mut VoiceChatOverlayState) { state.agent_scroll_view = None; state.agent_container = None; state.agent_bubble_views.clear(); + state.agent_input_bar = None; + state.agent_input_scroll_view = None; + state.agent_input_text_view = None; state.agent_input_field = None; state.agent_send_button = None; state.active_tab = Tab::Drawer; @@ -541,28 +692,61 @@ fn refresh_drawer_impl() { render_drawer_entries(&mut state, &query); } +/// Check that a path is within the CodeScribe transcriptions directory. +/// Prevents accidental read/delete of files outside the sandbox. +fn is_safe_transcription_path(path: &std::path::Path) -> bool { + let config_dir = codescribe_core::config::Config::config_dir(); + let allowed_root = config_dir.join("transcriptions"); + match (path.canonicalize(), allowed_root.canonicalize()) { + (Ok(canon), Ok(root)) => canon.starts_with(root), + // If canonicalize fails (file doesn't exist yet), fall back to prefix check + _ => path.starts_with(&allowed_root), + } +} + pub fn handle_card_copy(index: usize) { let state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - if let Some(entry) = state.drawer_entries.get(index) - && let Ok(contents) = std::fs::read_to_string(&entry.path) - { - copy_to_clipboard(&contents); + if let Some(entry) = state.drawer_entries.get(index) { + if !is_safe_transcription_path(&entry.path) { + warn!( + "Blocked copy: path outside transcriptions dir: {}", + entry.path.display() + ); + return; + } + if let Ok(contents) = std::fs::read_to_string(&entry.path) { + copy_to_clipboard(&contents); + } } } pub fn handle_card_edit(index: usize) { let state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); if let Some(entry) = state.drawer_entries.get(index) { + if !is_safe_transcription_path(&entry.path) { + warn!( + "Blocked edit: path outside transcriptions dir: {}", + entry.path.display() + ); + return; + } let _ = open_file_in_editor(&entry.path); } } pub fn handle_card_delete(index: usize) { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - if let Some(entry) = state.drawer_entries.get(index) - && let Err(err) = std::fs::remove_file(&entry.path) - { - warn!("Failed to delete {}: {}", entry.path.display(), err); + if let Some(entry) = state.drawer_entries.get(index) { + if !is_safe_transcription_path(&entry.path) { + warn!( + "Blocked delete: path outside transcriptions dir: {}", + entry.path.display() + ); + return; + } + if let Err(err) = std::fs::remove_file(&entry.path) { + warn!("Failed to delete {}: {}", entry.path.display(), err); + } } state.drawer_entries = load_drawer_entries(); render_drawer_entries(&mut state, ""); diff --git a/app/ui/voice_chat/handlers.rs b/app/ui/voice_chat/handlers.rs index cd784ec1..6ab3d2f2 100644 --- a/app/ui/voice_chat/handlers.rs +++ b/app/ui/voice_chat/handlers.rs @@ -8,6 +8,7 @@ use objc::{msg_send, sel, sel_impl}; use std::sync::Once; use tracing::{debug, info}; +use crate::ui::bootstrap; use crate::ui_helpers::{get_text_field_string, ns_string, set_hidden, set_text_field_string}; use super::api::{ @@ -117,12 +118,21 @@ extern "C" fn on_send(_this: &Object, _cmd: Sel, _sender: Id) { extern "C" fn on_close(_this: &Object, _cmd: Sel, _sender: Id) { super::api::hide_voice_chat_overlay(); + if bootstrap::should_show_bootstrap() { + bootstrap::handle_hotkey_done(); + } } extern "C" fn on_window_will_close(_this: &Object, _cmd: Sel, _notification: Id) { - let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); - clear_overlay_state(&mut state); - debug!("Voice chat overlay closed by user"); + match OVERLAY_STATE.try_lock() { + Ok(mut state) => { + clear_overlay_state(&mut state); + debug!("Voice chat overlay closed by user"); + } + Err(_) => { + debug!("Voice chat overlay close: state lock busy, skipping clear"); + } + } } extern "C" fn on_tab_changed(_this: &Object, _cmd: Sel, sender: Id) { diff --git a/app/ui/voice_chat/mod.rs b/app/ui/voice_chat/mod.rs index ca1ebd8f..41e3b639 100644 --- a/app/ui/voice_chat/mod.rs +++ b/app/ui/voice_chat/mod.rs @@ -11,11 +11,12 @@ mod state; // Re-export public API pub use api::{ add_voice_chat_error_message, add_voice_chat_user_message, append_voice_chat_assistant_delta, - clear_voice_chat_text, filter_drawer, hide_voice_chat_overlay, is_auto_send_enabled, - is_conversation_active, is_voice_chat_overlay_visible, refresh_drawer, + append_voice_chat_user_delta, clear_voice_chat_text, filter_drawer, hide_voice_chat_overlay, + is_auto_send_enabled, is_conversation_active, is_voice_chat_overlay_visible, refresh_drawer, reset_voice_chat_activity, send_voice_chat_draft, set_voice_chat_send_callback, - set_voice_chat_sending, set_voice_chat_text, show_agent_tab, show_drawer_tab, - update_conversation_state, update_drawer_after_save, update_voice_chat_status, + set_voice_chat_sending, set_voice_chat_target_app, set_voice_chat_text, + set_voice_chat_user_text, show_agent_tab, show_drawer_tab, update_conversation_state, + update_drawer_after_save, update_voice_chat_status, }; pub use state::{ConversationModeState, VoiceChatOverlayConfig}; @@ -32,8 +33,9 @@ use tracing::{info, warn}; use crate::ui_helpers::{ NS_FLOATING_WINDOW_LEVEL, add_subview, button_set_action, button_style, color_clear, - create_button, create_segmented_control, create_vertical_stack_view, ns_string, set_hidden, - window_set_alpha, window_show, + create_button, create_scrollable_text_view, create_segmented_control, + create_vertical_stack_view, ns_string, overlay_window_class, set_hidden, window_set_alpha, + window_show, }; use api::update_active_tab_impl; @@ -45,6 +47,7 @@ pub type Id = *mut Object; /// Show the voice chat overlay window pub fn show_voice_chat_overlay() { + info!("show_voice_chat_overlay requested"); Queue::main().exec_async(|| { show_voice_chat_overlay_impl(); }); @@ -52,6 +55,7 @@ pub fn show_voice_chat_overlay() { /// Show the voice chat overlay with custom configuration pub fn show_voice_chat_overlay_with_config(_config: VoiceChatOverlayConfig) { + info!("show_voice_chat_overlay_with_config requested"); Queue::main().exec_async(|| { show_voice_chat_overlay_impl(); }); @@ -59,6 +63,7 @@ pub fn show_voice_chat_overlay_with_config(_config: VoiceChatOverlayConfig) { fn show_voice_chat_overlay_impl() { unsafe { + info!("show_voice_chat_overlay_impl starting"); let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); let ns_window = Class::get("NSWindow").unwrap(); @@ -128,7 +133,8 @@ fn show_voice_chat_overlay_impl() { }, }; - let window: Id = msg_send![ns_window, alloc]; + let window_class = overlay_window_class(); + let window: Id = msg_send![window_class, alloc]; let style_mask = NSWindowStyleMask::Borderless | NSWindowStyleMask::FullSizeContentView; let backing = NSBackingStoreType::Buffered; let window: Id = msg_send![ @@ -155,8 +161,12 @@ fn show_voice_chat_overlay_impl() { let content_view: Id = msg_send![window, contentView]; let ns_visual = Class::get("NSVisualEffectView").unwrap(); + let blur_frame = CGRect::new( + &CGPoint::new(0.0, 0.0), + &CGSize::new(window_width, window_height), + ); let blur_view: Id = msg_send![ns_visual, alloc]; - let blur_view: Id = msg_send![blur_view, initWithFrame: frame]; + let blur_view: Id = msg_send![blur_view, initWithFrame: blur_frame]; let _: () = msg_send![blur_view, setMaterial: NSVisualEffectMaterial::HUDWindow]; let _: () = msg_send![blur_view, setBlendingMode: NSVisualEffectBlendingMode::BehindWindow]; let _: () = msg_send![blur_view, setState: NSVisualEffectState::Active]; @@ -173,7 +183,7 @@ fn show_voice_chat_overlay_impl() { let header_height = 44.0; let footer_height = 44.0; - let agent_input_height = 52.0; + let agent_input_height = 96.0; // Header let header_frame = CGRect::new( @@ -191,10 +201,12 @@ fn show_voice_chat_overlay_impl() { } add_subview(blur_view, header_view); + let title_x = 16.0; + let title_w = 120.0; let title_label = crate::ui_helpers::create_label(crate::ui_helpers::LabelConfig { frame: CGRect::new( - &CGPoint::new(16.0, window_height - 30.0), - &CGSize::new(160.0, 20.0), + &CGPoint::new(title_x, window_height - 30.0), + &CGSize::new(title_w, 20.0), ), text: "CodeScribe".to_string(), font_size: 13.0, @@ -206,16 +218,49 @@ fn show_voice_chat_overlay_impl() { }); add_subview(blur_view, title_label); + // Keep the tab control between the title and the right-side icon cluster. + // The overlay window is typically ~450px wide; fixed coordinates can overlap. + let right_cluster_start_x = window_width - 192.0; + let tab_x = title_x + title_w + 10.0; + // Don't enforce a minimum width here; on narrower windows, forcing a min can make the + // segmented control overlap the right-side icon cluster. + let tab_w = (right_cluster_start_x - 8.0 - tab_x).max(0.0); let tab_control = create_segmented_control( CGRect::new( - &CGPoint::new(170.0, window_height - 34.0), - &CGSize::new(160.0, 24.0), + &CGPoint::new(tab_x, window_height - 34.0), + &CGSize::new(tab_w, 24.0), ), &["Drawer", "Agent"], ); button_set_action(tab_control, action_handler, sel!(onTabChanged:)); add_subview(blur_view, tab_control); + let paste_last_button = create_button( + CGRect::new( + &CGPoint::new(window_width - 160.0, window_height - 34.0), + &CGSize::new(24.0, 24.0), + ), + "⇲", + button_style::SMALL_SQUARE, + ); + button_set_action( + paste_last_button, + action_handler, + sel!(onPasteLastResponse:), + ); + add_subview(blur_view, paste_last_button); + + let copy_last_button = create_button( + CGRect::new( + &CGPoint::new(window_width - 128.0, window_height - 34.0), + &CGSize::new(24.0, 24.0), + ), + "⧉", + button_style::SMALL_SQUARE, + ); + button_set_action(copy_last_button, action_handler, sel!(onCopyLastResponse:)); + add_subview(blur_view, copy_last_button); + let new_thread_button = create_button( CGRect::new( &CGPoint::new(window_width - 96.0, window_height - 34.0), @@ -270,13 +315,25 @@ fn show_voice_chat_overlay_impl() { add_subview(blur_view, drawer_scroll); // Agent scroll view + stack + let agent_scroll_frame_bottom = agent_input_height + 18.0; + let agent_scroll_frame_top = window_height - header_height - 10.0; + let agent_scroll_frame = CGRect::new( + &CGPoint::new(16.0, agent_scroll_frame_bottom), + &CGSize::new( + window_width - 32.0, + (agent_scroll_frame_top - agent_scroll_frame_bottom).max(0.0), + ), + ); let agent_scroll: Id = msg_send![ns_scroll, alloc]; - let agent_scroll: Id = msg_send![agent_scroll, initWithFrame: drawer_frame]; + let agent_scroll: Id = msg_send![agent_scroll, initWithFrame: agent_scroll_frame]; let _: () = msg_send![agent_scroll, setHasVerticalScroller: true]; let _: () = msg_send![agent_scroll, setDrawsBackground: false]; let agent_container = create_vertical_stack_view(CGRect::new( &CGPoint::new(0.0, 0.0), - &CGSize::new(drawer_frame.size.width, drawer_frame.size.height), + &CGSize::new( + agent_scroll_frame.size.width, + agent_scroll_frame.size.height, + ), )); let _: () = msg_send![agent_scroll, setDocumentView: agent_container]; add_subview(blur_view, agent_scroll); @@ -332,18 +389,23 @@ fn show_voice_chat_overlay_impl() { } add_subview(blur_view, input_bar); - let ns_text_field = Class::get("NSTextField").unwrap(); - let agent_input: Id = msg_send![ns_text_field, alloc]; - let agent_input: Id = msg_send![agent_input, initWithFrame: CGRect::new(&CGPoint::new(12.0, 12.0), &CGSize::new(window_width - 90.0, 28.0))]; - let _: () = msg_send![agent_input, setBezeled: true]; - let _: () = msg_send![agent_input, setPlaceholderString: ns_string("Napisz polecenie...")]; - let _: () = msg_send![agent_input, setTarget: action_handler]; - let _: () = msg_send![agent_input, setAction: sel!(onInputSubmit:)]; - let _: () = msg_send![input_bar, addSubview: agent_input]; - + let text_area_frame = CGRect::new( + &CGPoint::new(12.0, 10.0), + &CGSize::new(window_width - 90.0, agent_input_height - 20.0), + ); + let (agent_input_scroll, agent_input_text_view) = + create_scrollable_text_view(text_area_frame, true); + let ns_font = Class::get("NSFont").unwrap(); + let text_font: Id = msg_send![ns_font, systemFontOfSize: 13.0f64]; + let _: () = msg_send![agent_input_text_view, setFont: text_font]; + // Plain text: avoid rich text / style surprises when pasting. + let _: () = msg_send![agent_input_text_view, setRichText: false]; + let _: () = msg_send![input_bar, addSubview: agent_input_scroll]; + + let send_y = ((agent_input_height - 32.0) / 2.0).max(8.0); let agent_send_button = create_button( CGRect::new( - &CGPoint::new(window_width - 76.0, 10.0), + &CGPoint::new(window_width - 76.0, send_y), &CGSize::new(36.0, 32.0), ), ">", @@ -352,6 +414,22 @@ fn show_voice_chat_overlay_impl() { button_set_action(agent_send_button, action_handler, sel!(onSend:)); let _: () = msg_send![input_bar, addSubview: agent_send_button]; + // Accessibility labels (VoiceOver) + let _: () = msg_send![close_button, setAccessibilityLabel: ns_string("Close overlay")]; + let _: () = msg_send![settings_button, setAccessibilityLabel: ns_string("Settings")]; + let _: () = msg_send![agent_send_button, setAccessibilityLabel: ns_string("Send message")]; + let _: () = msg_send![tab_control, setAccessibilityLabel: ns_string("View selector")]; + let _: () = + msg_send![search_field, setAccessibilityLabel: ns_string("Search transcriptions")]; + let _: () = + msg_send![agent_input_text_view, setAccessibilityLabel: ns_string("Message input")]; + let _: () = + msg_send![paste_last_button, setAccessibilityLabel: ns_string("Paste last response")]; + let _: () = + msg_send![copy_last_button, setAccessibilityLabel: ns_string("Copy last response")]; + let _: () = + msg_send![new_thread_button, setAccessibilityLabel: ns_string("New conversation")]; + // Initial visibility set_hidden(agent_scroll, true); set_hidden(input_bar, true); @@ -368,7 +446,10 @@ fn show_voice_chat_overlay_impl() { state.search_field = Some(search_field as usize); state.agent_scroll_view = Some(agent_scroll as usize); state.agent_container = Some(agent_container as usize); - state.agent_input_field = Some(agent_input as usize); + state.agent_input_bar = Some(input_bar as usize); + state.agent_input_scroll_view = Some(agent_input_scroll as usize); + state.agent_input_text_view = Some(agent_input_text_view as usize); + state.agent_input_field = None; state.agent_send_button = Some(agent_send_button as usize); state.action_handler = Some(action_handler as usize); state.active_tab = Tab::Drawer; @@ -380,14 +461,17 @@ fn show_voice_chat_overlay_impl() { crate::ui_helpers::animate_fade(window, 1.0, 0.2); let has_messages = !state.messages.is_empty(); + let desired_tab = if has_messages { + Tab::Agent + } else { + state.active_tab + }; drop(state); api::refresh_drawer(); - if has_messages { - update_active_tab_impl(Tab::Agent); + update_active_tab_impl(desired_tab); + if has_messages || matches!(desired_tab, Tab::Agent) { let mut state = OVERLAY_STATE.lock().unwrap_or_else(|e| e.into_inner()); api::update_chat_view_with_state(&mut state, true); - } else { - update_active_tab_impl(Tab::Drawer); } } } diff --git a/app/ui/voice_chat/state.rs b/app/ui/voice_chat/state.rs index 895b8094..11294446 100644 --- a/app/ui/voice_chat/state.rs +++ b/app/ui/voice_chat/state.rs @@ -89,6 +89,10 @@ pub struct DrawerEntry { pub is_favorite: bool, } +/// Maximum number of chat messages retained in memory. +/// Oldest messages are dropped when this limit is exceeded. +const MAX_CHAT_MESSAGES: usize = 100; + /// Voice chat overlay state pub struct VoiceChatOverlayState { // Window @@ -112,6 +116,9 @@ pub struct VoiceChatOverlayState { pub agent_scroll_view: Option, pub agent_container: Option, pub agent_bubble_views: Vec<(usize, usize)>, + pub agent_input_bar: Option, + pub agent_input_scroll_view: Option, + pub agent_input_text_view: Option, pub agent_input_field: Option, pub agent_send_button: Option, @@ -123,6 +130,7 @@ pub struct VoiceChatOverlayState { pub manual_draft: String, pub is_sending: bool, pub auto_send_enabled: bool, + pub last_target_app: Option, // Conversation mode (Moshi) pub conversation_state: ConversationModeState, @@ -148,6 +156,9 @@ impl Default for VoiceChatOverlayState { agent_scroll_view: None, agent_container: None, agent_bubble_views: Vec::new(), + agent_input_bar: None, + agent_input_scroll_view: None, + agent_input_text_view: None, agent_input_field: None, agent_send_button: None, active_tab: Tab::Drawer, @@ -155,12 +166,24 @@ impl Default for VoiceChatOverlayState { manual_draft: String::new(), is_sending: false, auto_send_enabled: true, + last_target_app: None, conversation_state: ConversationModeState::default(), action_handler: None, } } } +impl VoiceChatOverlayState { + /// Append a message, dropping the oldest if over the cap. + pub fn push_message(&mut self, msg: ChatMessage) { + self.messages.push(msg); + if self.messages.len() > MAX_CHAT_MESSAGES { + let excess = self.messages.len() - MAX_CHAT_MESSAGES; + self.messages.drain(..excess); + } + } +} + lazy_static::lazy_static! { pub static ref OVERLAY_STATE: Mutex = Mutex::new(VoiceChatOverlayState::default()); pub static ref SEND_CALLBACK: Mutex> = Mutex::new(None); diff --git a/bin/codescribe.rs b/bin/codescribe.rs index dbb06886..ccefb045 100644 --- a/bin/codescribe.rs +++ b/bin/codescribe.rs @@ -12,6 +12,9 @@ use codescribe::{ai_formatting, audio, whisper}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; +use tracing::{debug, info, warn}; +use tracing_subscriber::EnvFilter; +use tracing_subscriber::fmt::writer::MakeWriterExt; /// CodeScribe CLI - Local speech-to-text transcription /// @@ -92,6 +95,7 @@ enum MigrateKind { #[tokio::main] async fn main() -> Result<()> { + init_tracing(); let cli = Cli::parse(); // Handle --config flag @@ -110,6 +114,39 @@ async fn main() -> Result<()> { } } +static LOG_GUARD: std::sync::OnceLock = + std::sync::OnceLock::new(); + +fn init_tracing() { + // Ensure ~/.codescribe/.env is loaded before we read RUST_LOG. + let _ = codescribe::config::Config::load(); + + let filter = EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")); + + let log_dir = codescribe::config::Config::config_dir().join("logs"); + let _ = std::fs::create_dir_all(&log_dir); + let file_appender = tracing_appender::rolling::never(&log_dir, "codescribe.log"); + let (file_writer, guard) = tracing_appender::non_blocking(file_appender); + let _ = LOG_GUARD.set(guard); + + let log_to_stdout = std::env::var("CODESCRIBE_LOG_STDOUT") + .ok() + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(false); + + let subscriber = tracing_subscriber::fmt() + .with_env_filter(filter) + .with_target(false); + + let _ = if log_to_stdout { + subscriber + .with_writer(file_writer.and(std::io::stdout)) + .try_init() + } else { + subscriber.with_writer(file_writer).try_init() + }; +} + /// Handle --config flag: create default config and open in editor fn handle_config_command() -> Result<()> { use std::fs; @@ -327,42 +364,35 @@ async fn handle_transcribe_file( } async fn handle_transcribe_live(language: Option) -> Result<()> { - use tokio::sync::mpsc; - eprintln!("CodeScribe Live Transcription"); eprintln!("Press Ctrl+C to stop."); whisper::init()?; - let mut recorder = codescribe::audio::streaming_recorder::StreamingRecorder::new()?; - let (vad_tx, mut vad_rx) = mpsc::unbounded_channel::<()>(); - recorder.recorder.set_on_vad_stop({ - let vad_tx = vad_tx.clone(); - move || { - let _ = vad_tx.send(()); - } - }); + let config = codescribe::audio::recorder::RecorderConfig::default(); + let mut recorder = + codescribe::audio::streaming_recorder::StreamingRecorder::with_config(config)?; let emitter = StreamEmitter::new(); - recorder.set_delta_callback(Some(Arc::new({ - let emitter = Arc::clone(&emitter); - move |delta: &str| { - emitter.emit_raw(delta); - } - }))); + recorder.set_delta_callback(Some(Arc::new( + codescribe_core::pipeline::sinks::CallbackSink::new(Arc::new({ + let emitter = Arc::clone(&emitter); + move |delta: &str| { + emitter.emit_delta(delta); + } + })), + ))); - recorder.start(language).await?; + // Live CLI should stream continuously (no VAD-gated buffering). + recorder.start_with_buffered(language, false).await?; tokio::select! { _ = tokio::signal::ctrl_c() => { eprintln!("Stopping live transcription (Ctrl+C)..."); } - _ = vad_rx.recv() => { - eprintln!("Stopping live transcription (VAD)..."); - } } - let _ = recorder.stop().await?; + let _ = recorder.stop_without_saving().await?; emitter.finish(); Ok(()) @@ -382,11 +412,19 @@ async fn run_daemon() -> Result<()> { #[cfg(target_os = "macos")] codescribe::set_dock_icon(); + #[cfg(target_os = "macos")] + codescribe::install_basic_edit_menu(); codescribe::whisper::init().context("Failed to initialize Whisper")?; let controller = Arc::new(RecordingController::new()); #[cfg(target_os = "macos")] codescribe::controller::register_overlay_controller(Arc::clone(&controller)); + #[cfg(target_os = "macos")] + { + if codescribe::should_show_bootstrap() { + codescribe::schedule_bootstrap(); + } + } let config = Config::load(); sync_hotkey_config(&config); @@ -402,16 +440,32 @@ async fn run_daemon() -> Result<()> { let menu_controller = Arc::clone(&controller); let menu_handle = Handle::current(); std::thread::spawn(move || { + use tray::TrayMenuEvent; for event in menu_rx { + // Apply hotkey settings directly from event (avoids race condition with .env save) + match &event { + TrayMenuEvent::SetHoldMods(mods) => { + hotkeys::set_hold_mods(*mods); + } + TrayMenuEvent::SetToggleTrigger(trigger) => { + hotkeys::set_toggle_trigger(*trigger); + } + TrayMenuEvent::ToggleHoldExclusive => { + let config = Config::load(); + hotkeys::set_exclusive_mode(!config.hold_exclusive); + } + _ => {} + } + + // Update controller with fresh config for non-hotkey settings let controller = Arc::clone(&menu_controller); let handle = menu_handle.clone(); handle.spawn(async move { let config = Config::load(); - sync_hotkey_config(&config); controller.set_config(config).await; }); - if matches!(event, tray::TrayMenuEvent::Quit) { + if matches!(event, TrayMenuEvent::Quit) { break; } } @@ -434,21 +488,6 @@ async fn run_daemon() -> Result<()> { } }); - // VAD monitor task - auto-finish recording when silence detected - let vad_controller = Arc::clone(&controller); - tokio::spawn(async move { - loop { - tokio::time::sleep(tokio::time::Duration::from_millis(100)).await; - if vad_controller.is_vad_triggered() { - eprintln!("VAD triggered - auto-finishing recording"); - vad_controller.clear_vad_triggered(); - if let Err(e) = vad_controller.finish_recording().await { - eprintln!("VAD finish_recording error: {}", e); - } - } - } - }); - // Start Quality Loop daemon (always-on self-improvement) let quality_child = spawn_quality_daemon(); @@ -473,6 +512,15 @@ struct QualityDaemonHandle { fn spawn_quality_daemon() -> Option { use std::process::{Command, Stdio}; + if matches!( + std::env::var("CODESCRIBE_QUALITY_DAEMON").as_deref(), + Ok("0") | Ok("false") | Ok("off") + ) { + info!("Quality daemon disabled via CODESCRIBE_QUALITY_DAEMON=0"); + codescribe::quality_loop::mark_daemon_unavailable(); + return None; + } + // Strategy: find codescribe-loop binary next to current exe, or in PATH let loop_bin = find_sibling_binary("codescribe-loop"); @@ -483,7 +531,7 @@ fn spawn_quality_daemon() -> Option { if which_exists("codescribe-loop") { PathBuf::from("codescribe-loop") } else { - eprintln!("[quality-daemon] codescribe-loop not found; skipping auto-start"); + debug!("[quality-daemon] codescribe-loop not found; skipping auto-start"); codescribe::quality_loop::mark_daemon_unavailable(); return None; } @@ -499,7 +547,7 @@ fn spawn_quality_daemon() -> Option { && let Ok(pid) = pid_str.trim().parse::() && is_process_alive(pid) { - eprintln!( + debug!( "[quality-daemon] Already running (pid={}); skipping auto-start", pid ); @@ -515,7 +563,7 @@ fn spawn_quality_daemon() -> Option { { Ok(f) => f, Err(e) => { - eprintln!("[quality-daemon] Failed to open log file: {}", e); + warn!("[quality-daemon] Failed to open log file: {}", e); codescribe::quality_loop::mark_daemon_unavailable(); return None; } @@ -537,7 +585,7 @@ fn spawn_quality_daemon() -> Option { { Ok(child) => { let _ = std::fs::write(&pid_path, child.id().to_string()); - eprintln!( + info!( "[quality-daemon] Started (pid={}, bin={}, log={})", child.id(), bin_path.display(), @@ -546,7 +594,7 @@ fn spawn_quality_daemon() -> Option { Some(QualityDaemonHandle { child, pid_path }) } Err(e) => { - eprintln!("[quality-daemon] Failed to spawn: {}", e); + warn!("[quality-daemon] Failed to spawn: {}", e); codescribe::quality_loop::mark_daemon_unavailable(); None } @@ -600,6 +648,7 @@ fn emit_stdout(text: &str) -> Result<()> { struct StreamEmitter { last_len: Mutex, had_output: AtomicBool, + buffer: Mutex, } impl StreamEmitter { @@ -607,6 +656,7 @@ impl StreamEmitter { Arc::new(Self { last_len: Mutex::new(0), had_output: AtomicBool::new(false), + buffer: Mutex::new(String::new()), }) } @@ -619,6 +669,19 @@ impl StreamEmitter { } } + fn emit_delta(&self, delta: &str) { + if delta.is_empty() { + return; + } + + let snapshot = { + let mut buffer = self.buffer.lock().unwrap_or_else(|e| e.into_inner()); + apply_delta_to_string(&mut buffer, delta); + buffer.clone() + }; + self.emit_cumulative(&snapshot); + } + fn emit_cumulative(&self, cumulative: &str) { let mut last_len = self.last_len.lock().unwrap_or_else(|e| e.into_inner()); let total_len = cumulative.len(); @@ -637,6 +700,10 @@ impl StreamEmitter { } } +fn apply_delta_to_string(target: &mut String, delta: &str) { + codescribe_core::contracts::TranscriptDelta::from_raw(delta).apply(target); +} + fn sync_hotkey_config(config: &codescribe::config::Config) { codescribe::os::hotkeys::set_hold_mods(config.hold_mods); codescribe::os::hotkeys::set_toggle_trigger(config.toggle_trigger); @@ -682,6 +749,19 @@ async fn dispatch_hotkey_event( }; controller.handle_hotkey_event(input).await?; } + HotkeyEvent::Conversation { action } => { + let mapped_action = match action { + HoldAction::Down => HotkeyAction::Down, + HoldAction::Up => HotkeyAction::Up, + }; + let input = HotkeyInput { + key_type: HotkeyType::Conversation, + action: mapped_action, + assistive: false, + force_ai: false, + }; + controller.handle_hotkey_event(input).await?; + } } Ok(()) diff --git a/bin/codescribe_loop.rs b/bin/codescribe_loop.rs index 1a5d9901..16ebd4e6 100644 --- a/bin/codescribe_loop.rs +++ b/bin/codescribe_loop.rs @@ -142,6 +142,13 @@ enum ReferenceSourceArg { async fn main() -> Result<()> { let args = Args::parse(); + if env_bool("CODESCRIBE_LOOP_USE_CLOUD_STT") { + // Loop-only override: force cloud STT without changing app defaults. + unsafe { + std::env::set_var("USE_LOCAL_STT", "0"); + } + } + if args.no_embeddings { // SAFETY: this is a single-process CLI before any threads start. unsafe { diff --git a/core/Cargo.toml b/core/Cargo.toml index 5a47769b..d3442992 100644 --- a/core/Cargo.toml +++ b/core/Cargo.toml @@ -31,7 +31,7 @@ candle-nn = "0.9" tokenizers = "0.22" indicatif = "0.18" zip = "0.6" -# ndarray 0.17 required to match fastembed/ort dependency +# ndarray 0.17 required by ndarray-npy 0.10 ndarray = "0.17" # Disable default features to avoid zip dep (we only need .npy, not .npz archives) ndarray-npy = { version = "0.10", default-features = false } @@ -48,21 +48,19 @@ symphonia = { version = "0.5", features = ["all", "mp3", "pcm", "isomp4", "alac" # Text/embeddings regex = "1" -fastembed = { version = "5.8.1", default-features = false, features = ["hf-hub-native-tls", "ort-download-binaries"] } # Config/paths directories = "6" shellexpand = "3" dotenvy = "0.15" -# Voice Activity Detection - custom Silero wrapper (shares ort with fastembed) +# Voice Activity Detection - custom Silero wrapper # Model: silero_vad.onnx from https://github.com/snakers4/silero-vad -# NOTE: fastembed enables ort-download-binaries - we just need to match version -ort = { version = "=2.0.0-rc.11", default-features = false, features = ["download-binaries"] } +# NOTE: ort-download-binaries ensures runtime is available without manual install +ort = { version = "=2.0.0-rc.11", default-features = false, features = ["std", "download-binaries", "tls-native"] } # Moshi - conversational voice AI (Kyutai Labs) -# Git dependency for portability (path dep requires local clone) -moshi = { git = "https://github.com/kyutai-labs/moshi", branch = "main", features = ["metal"] } +moshi = { version = "0.6.4", features = ["metal"] } # Time/utilities chrono = "0.4" @@ -88,4 +86,4 @@ dirs = "6" # Allow unexpected_cfgs from objc crate's msg_send! macro (uses cargo-clippy cfg internally) # embed_model is set by build.rs when Whisper model is available # embed_tts is set by build.rs when TTS model is available (CODESCRIBE_EMBED_TTS=1) -unexpected_cfgs = { level = "allow", check-cfg = ['cfg(embed_model)', 'cfg(embed_tts)'] } +unexpected_cfgs = { level = "allow", check-cfg = ['cfg(embed_model)', 'cfg(embed_tts)', 'cfg(embed_e5)', 'cfg(embed_vad)'] } diff --git a/core/audio/chunker.rs b/core/audio/chunker.rs new file mode 100644 index 00000000..93406bb7 --- /dev/null +++ b/core/audio/chunker.rs @@ -0,0 +1,1180 @@ +//! Audio chunker — VAD-gated speech segmentation. +//! +//! Extracted from `streaming_recorder.rs` to decouple audio segmentation +//! from transcription logic. This module has **zero** dependency on Whisper/STT. +//! +//! ## Pipeline position +//! +//! ```text +//! Recorder (audio capture) → Chunker (this module) → STT adapter → PostProcessor → DeltaSink +//! ``` +//! +//! ## Key types +//! +//! - [`SpeechSession`] — stateful VAD gate + chunker (Silero neural VAD) +//! - [`SpeechEvent`] — emitted events: `Chunk` (streaming) or `Utterance` (complete) +//! - [`VadIterState`] — Silero VAD iterator state machine (start/end boundary detection) +//! +//! Created by M&K (c)2026 VetCoders + +use std::collections::VecDeque; + +use tokio::time::Instant; +use tracing::{debug, trace}; + +use crate::vad; + +// ═══════════════════════════════════════════════════════════ +// Public types +// ═══════════════════════════════════════════════════════════ + +pub(crate) enum SpeechEvent { + Chunk(Vec), + Utterance(Vec), +} + +pub(crate) enum SpeechMode { + Stream { + chunk_limit: usize, + overlap_size: usize, + }, + Utterance { + max_utterance_samples: usize, + }, +} + +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub(crate) enum VadGateMode { + /// Gate audio before it reaches Whisper (legacy). + Simple, + /// Silero VAD iter logic as a hard gate (legacy). + Iter, + /// Silero VAD is a supervisor: audio always flows, VAD only defines boundaries. + Supervisor, +} + +pub(crate) struct GateConfig { + pub vad: vad::VadConfig, + pub pre_roll_sec: f32, + pub speech_pad_sec: f32, + pub mode: VadGateMode, +} + +// ═══════════════════════════════════════════════════════════ +// SpeechSession +// ═══════════════════════════════════════════════════════════ + +pub(crate) struct SpeechSession { + mode: SpeechMode, + threshold: f32, + neg_threshold: f32, + min_speech_samples: usize, + min_silence_samples: usize, + in_speech: bool, + speech_samples: usize, + silence_samples: usize, + pending_speech: Vec, + pending_silence: Vec, + pending_samples: Vec, + pre_roll: VecDeque, + pre_roll_samples: usize, + speech_pad_samples: usize, + last_append_at: Instant, + vad: Option, + resampler: Option, + vad_resample_buf: Vec, + output_sample_rate: u32, + raw_sample_rate: u32, + gate_mode: VadGateMode, + iter_state: Option, + iter_speech_start: Option, + raw_buffer: VecDeque, + raw_buffer_start: usize, + raw_cursor: usize, + segment_start: Option, + pending_end: Option, + pre_roll_raw: usize, + speech_pad_raw: usize, + last_emit_raw: usize, +} + +impl SpeechSession { + pub fn new_stream(sample_rate: u32, chunk_duration_sec: f32, overlap_sec: f32) -> Self { + let config = hardcoded_gate_config(); + debug!("SpeechSession::new_stream gate_mode={:?}", config.mode); + let vad_sample_rate = vad::VAD_SAMPLE_RATE; + let output_sample_rate = match config.mode { + VadGateMode::Supervisor => sample_rate, + _ => vad_sample_rate, + }; + let min_speech_samples = (config.vad.min_speech_duration_sec * vad_sample_rate as f32) + .round() + .max(1.0) as usize; + let min_silence_samples = (config.vad.max_silence_duration_sec * vad_sample_rate as f32) + .round() + .max(1.0) as usize; + let neg_threshold = (config.vad.threshold - 0.15).max(0.05); + + let vad = init_silero_vad(vad_sample_rate, &config.vad); + let resampler = if sample_rate != vad_sample_rate { + Some(vad::Resampler::new(sample_rate)) + } else { + None + }; + let pre_roll_samples = (config.pre_roll_sec * output_sample_rate as f32) + .round() + .max(0.0) as usize; + let speech_pad_samples = (config.speech_pad_sec * output_sample_rate as f32) + .round() + .max(0.0) as usize; + let iter_state = match config.mode { + VadGateMode::Iter | VadGateMode::Supervisor => { + Some(VadIterState::new(&config, vad::VAD_SAMPLE_RATE)) + } + VadGateMode::Simple => None, + }; + let chunk_limit_raw = (sample_rate as f32 * chunk_duration_sec).round().max(1.0) as usize; + let overlap_raw = (sample_rate as f32 * overlap_sec).round().max(0.0) as usize; + let pre_roll_raw = (sample_rate as f32 * config.pre_roll_sec).round().max(0.0) as usize; + let speech_pad_raw = (sample_rate as f32 * config.speech_pad_sec) + .round() + .max(0.0) as usize; + + Self { + mode: SpeechMode::Stream { + chunk_limit: chunk_limit_raw, + overlap_size: overlap_raw, + }, + threshold: config.vad.threshold, + neg_threshold, + min_speech_samples, + min_silence_samples, + in_speech: false, + speech_samples: 0, + silence_samples: 0, + pending_speech: Vec::new(), + pending_silence: Vec::new(), + pending_samples: Vec::new(), + pre_roll: VecDeque::new(), + pre_roll_samples, + speech_pad_samples, + last_append_at: Instant::now(), + vad, + resampler, + vad_resample_buf: Vec::new(), + output_sample_rate, + raw_sample_rate: sample_rate, + gate_mode: config.mode, + iter_state, + iter_speech_start: None, + raw_buffer: VecDeque::new(), + raw_buffer_start: 0, + raw_cursor: 0, + segment_start: None, + pending_end: None, + pre_roll_raw, + speech_pad_raw, + last_emit_raw: 0, + } + } + + pub fn new_utterance(sample_rate: u32) -> Self { + let config = hardcoded_gate_config(); + debug!("SpeechSession::new_utterance gate_mode={:?}", config.mode); + let vad_sample_rate = vad::VAD_SAMPLE_RATE; + let output_sample_rate = match config.mode { + VadGateMode::Supervisor => sample_rate, + _ => vad_sample_rate, + }; + let min_speech_samples = (config.vad.min_speech_duration_sec * vad_sample_rate as f32) + .round() + .max(1.0) as usize; + let min_silence_samples = (config.vad.max_silence_duration_sec * vad_sample_rate as f32) + .round() + .max(1.0) as usize; + let neg_threshold = (config.vad.threshold - 0.15).max(0.05); + + let vad = init_silero_vad(vad_sample_rate, &config.vad); + let resampler = if sample_rate != vad_sample_rate { + Some(vad::Resampler::new(sample_rate)) + } else { + None + }; + + let max_utterance_samples = + (config.vad.max_utterance_sec * output_sample_rate as f32) as usize; + let pre_roll_samples = (config.pre_roll_sec * output_sample_rate as f32) + .round() + .max(0.0) as usize; + let speech_pad_samples = (config.speech_pad_sec * output_sample_rate as f32) + .round() + .max(0.0) as usize; + let iter_state = match config.mode { + VadGateMode::Iter | VadGateMode::Supervisor => { + Some(VadIterState::new(&config, vad::VAD_SAMPLE_RATE)) + } + VadGateMode::Simple => None, + }; + + Self { + mode: SpeechMode::Utterance { + max_utterance_samples, + }, + threshold: config.vad.threshold, + neg_threshold, + min_speech_samples, + min_silence_samples, + in_speech: false, + speech_samples: 0, + silence_samples: 0, + pending_speech: Vec::new(), + pending_silence: Vec::new(), + pending_samples: Vec::new(), + pre_roll: VecDeque::new(), + pre_roll_samples, + speech_pad_samples, + last_append_at: Instant::now(), + vad, + resampler, + vad_resample_buf: Vec::new(), + output_sample_rate, + raw_sample_rate: sample_rate, + gate_mode: config.mode, + iter_state, + iter_speech_start: None, + raw_buffer: VecDeque::new(), + raw_buffer_start: 0, + raw_cursor: 0, + segment_start: None, + pending_end: None, + pre_roll_raw: (sample_rate as f32 * config.pre_roll_sec).round().max(0.0) as usize, + speech_pad_raw: (sample_rate as f32 * config.speech_pad_sec) + .round() + .max(0.0) as usize, + last_emit_raw: 0, + } + } + + pub fn feed(&mut self, audio: &[f32], _sample_rate: u32) -> Vec { + let mut events = Vec::new(); + if audio.is_empty() { + return events; + } + + if self.gate_mode == VadGateMode::Supervisor { + return self.feed_supervisor(audio); + } + + let resampled = if let Some(resampler) = self.resampler.as_mut() { + resampler.resample(audio) + } else { + audio.to_vec() + }; + self.vad_resample_buf.extend_from_slice(&resampled); + + while self.vad_resample_buf.len() >= vad::CHUNK_SIZE { + let frame: Vec = self.vad_resample_buf.drain(..vad::CHUNK_SIZE).collect(); + let speech_prob = match self.vad.as_mut() { + Some(vad) => vad.predict(&frame).unwrap_or(0.0), + None => 1.0, + }; + let decision = match self.gate_mode { + VadGateMode::Simple => self.gate_with_prob(&frame, speech_prob), + VadGateMode::Iter => self.gate_with_iter(&frame, speech_prob), + VadGateMode::Supervisor => self.gate_with_iter(&frame, speech_prob), + }; + if let Some(mut gated) = decision.audio { + self.pending_samples.append(&mut gated); + self.last_append_at = Instant::now(); + } + + if decision.ended { + if !self.pending_samples.is_empty() { + events.push(self.emit_final()); + } + return events; + } + + match self.mode { + SpeechMode::Stream { + chunk_limit, + overlap_size: _, + } => { + if self.pending_samples.len() >= chunk_limit { + events.push(self.emit_chunk()); + } + } + SpeechMode::Utterance { + max_utterance_samples, + } => { + if self.pending_samples.len() >= max_utterance_samples { + events.push(self.emit_final()); + } + } + } + } + events + } + + fn feed_supervisor(&mut self, audio: &[f32]) -> Vec { + let mut events = Vec::new(); + if audio.is_empty() { + return events; + } + + // Always keep raw audio flowing. + self.raw_buffer.extend(audio); + self.raw_cursor = self.raw_cursor.saturating_add(audio.len()); + + // Run Silero on 16kHz view of the audio, then map boundaries to raw. + let resampled = if let Some(resampler) = self.resampler.as_mut() { + resampler.resample(audio) + } else { + audio.to_vec() + }; + self.vad_resample_buf.extend_from_slice(&resampled); + + while self.vad_resample_buf.len() >= vad::CHUNK_SIZE { + let frame: Vec = self.vad_resample_buf.drain(..vad::CHUNK_SIZE).collect(); + let speech_prob = match self.vad.as_mut() { + Some(vad) => vad.predict(&frame).unwrap_or(0.0), + None => 1.0, + }; + + trace!( + "VAD frame: prob={:.3} threshold={:.3} triggered={} segment_start={:?}", + speech_prob, + self.threshold, + self.iter_state + .as_ref() + .map_or(self.in_speech, |s| s.triggered()), + self.segment_start, + ); + + let mut start_event: Option = None; + let mut end_event: Option = None; + + debug_assert!( + self.iter_state.is_some(), + "Supervisor gate requires VadIterState" + ); + if let Some(iter_state) = self.iter_state.as_mut() { + let event = iter_state.update(speech_prob); + match event { + VadIterEvent::Start { start_sample } => { + start_event = Some(start_sample); + } + VadIterEvent::End { end_sample } => { + end_event = Some(end_sample); + } + VadIterEvent::None => {} + } + } else { + trace!("Supervisor gate missing VadIterState; skipping frame"); + continue; + } + + if let Some(start_sample) = start_event { + let raw_start = self + .vad_to_raw_index(start_sample) + .saturating_sub(self.pre_roll_raw); + self.segment_start = Some(raw_start); + self.last_emit_raw = raw_start; + } + + if let Some(end_sample) = end_event { + let raw_end = self + .vad_to_raw_index(end_sample) + .saturating_add(self.speech_pad_raw); + self.pending_end = Some(raw_end); + } + + if let Some(iter_state) = self.iter_state.as_ref() { + trace!( + "VAD index sync: vad_current={} raw_cursor={} mapped={}", + iter_state.current_sample, + self.raw_cursor, + self.vad_to_raw_index(iter_state.current_sample) + ); + } + + if let SpeechMode::Stream { + chunk_limit, + overlap_size, + } = self.mode + && self.segment_start.is_some() + && self.pending_end.is_none() + && self.raw_cursor.saturating_sub(self.last_emit_raw) >= chunk_limit + { + let end = self.raw_cursor; + if let Some(chunk) = self.raw_slice(self.last_emit_raw, end) { + events.push(SpeechEvent::Chunk(chunk)); + } + if overlap_size > 0 { + self.last_emit_raw = end.saturating_sub(overlap_size); + } else { + self.last_emit_raw = end; + } + // Trim raw buffer to prevent unbounded growth during long speech. + // Keep from last_emit_raw minus pre-roll so overlap slicing still works. + self.trim_raw_buffer(self.last_emit_raw.saturating_sub(self.pre_roll_raw)); + } + } + + if let Some(end) = self.pending_end + && self.raw_cursor >= end + { + if let Some(start) = self.segment_start.take() + && let Some(chunk) = self.raw_slice(start, end) + { + match self.mode { + SpeechMode::Stream { .. } => events.push(SpeechEvent::Chunk(chunk)), + SpeechMode::Utterance { .. } => events.push(SpeechEvent::Utterance(chunk)), + } + } + self.pending_end = None; + self.last_emit_raw = end; + self.trim_raw_buffer(end.saturating_sub(self.pre_roll_raw)); + } + + // Safety net: when no segment is active, cap raw_buffer to pre-roll size. + // This prevents unbounded growth during silence or when VAD never fires Start. + if self.segment_start.is_none() && self.pending_end.is_none() { + self.trim_raw_buffer(self.raw_cursor.saturating_sub(self.pre_roll_raw)); + } + + events + } + + pub fn flush(&mut self) -> Option { + if self.gate_mode == VadGateMode::Supervisor { + if let Some(start) = self.segment_start.take() { + // VAD fired Start but recording ended before End — emit what we have. + let end = self.pending_end.take().unwrap_or(self.raw_cursor); + let end = end.min(self.raw_cursor); + self.last_emit_raw = end; + if let Some(chunk) = self.raw_slice(start, end) { + debug!( + "Supervisor flush: open segment {}..{} ({} samples)", + start, + end, + chunk.len() + ); + return Some(match self.mode { + SpeechMode::Stream { .. } => SpeechEvent::Chunk(chunk), + SpeechMode::Utterance { .. } => SpeechEvent::Utterance(chunk), + }); + } + } + // VAD never triggered Start — no speech detected, intentionally drop. + return None; + } + if self.pending_samples.is_empty() { + return None; + } + Some(self.emit_final()) + } + + fn emit_chunk(&mut self) -> SpeechEvent { + let chunk = std::mem::take(&mut self.pending_samples); + if let SpeechMode::Stream { overlap_size, .. } = self.mode + && overlap_size > 0 + && chunk.len() > overlap_size + { + let start = chunk.len() - overlap_size; + self.pending_samples.extend_from_slice(&chunk[start..]); + } + self.last_append_at = Instant::now(); + SpeechEvent::Chunk(chunk) + } + + fn emit_final(&mut self) -> SpeechEvent { + let chunk = std::mem::take(&mut self.pending_samples); + self.pending_speech.clear(); + self.pending_silence.clear(); + self.speech_samples = 0; + self.silence_samples = 0; + self.in_speech = false; + self.pre_roll.clear(); + self.iter_speech_start = None; + if let Some(iter_state) = self.iter_state.as_mut() { + iter_state.reset(); + } + self.last_append_at = Instant::now(); + match self.mode { + SpeechMode::Stream { .. } => SpeechEvent::Chunk(chunk), + SpeechMode::Utterance { .. } => SpeechEvent::Utterance(chunk), + } + } + + fn gate_with_prob(&mut self, audio: &[f32], speech_prob: f32) -> GateDecision { + let is_speech = speech_prob >= self.threshold; + + if self.in_speech { + if speech_prob >= self.neg_threshold { + self.silence_samples = 0; + if !self.pending_silence.is_empty() { + let mut out = Vec::with_capacity(self.pending_silence.len() + audio.len()); + out.append(&mut self.pending_silence); + out.extend_from_slice(audio); + return GateDecision { + audio: Some(out), + ended: false, + }; + } + return GateDecision { + audio: Some(audio.to_vec()), + ended: false, + }; + } + + self.pending_silence.extend_from_slice(audio); + self.silence_samples = self.silence_samples.saturating_add(audio.len()); + if self.silence_samples >= self.min_silence_samples { + self.in_speech = false; + self.silence_samples = 0; + if self.speech_pad_samples > 0 && !self.pending_silence.is_empty() { + let pad = self + .pending_silence + .drain(..self.speech_pad_samples.min(self.pending_silence.len())) + .collect::>(); + if !pad.is_empty() { + self.pending_samples.extend_from_slice(&pad); + } + } + self.pending_silence.clear(); + self.speech_samples = 0; + self.pending_speech.clear(); + return GateDecision { + audio: None, + ended: true, + }; + } + return GateDecision { + audio: None, + ended: false, + }; + } + + if is_speech { + self.pending_speech.extend_from_slice(audio); + self.speech_samples = self.speech_samples.saturating_add(audio.len()); + if self.speech_samples >= self.min_speech_samples { + self.in_speech = true; + self.speech_samples = 0; + let mut out = Vec::new(); + if self.pre_roll_samples > 0 && !self.pre_roll.is_empty() { + out.extend(self.pre_roll.drain(..)); + } + out.extend(std::mem::take(&mut self.pending_speech)); + return GateDecision { + audio: Some(out), + ended: false, + }; + } + return GateDecision { + audio: None, + ended: false, + }; + } + + self.pending_speech.clear(); + self.speech_samples = 0; + self.push_pre_roll(audio); + GateDecision { + audio: None, + ended: false, + } + } + + fn gate_with_iter(&mut self, audio: &[f32], speech_prob: f32) -> GateDecision { + let Some(iter_state) = self.iter_state.as_mut() else { + return self.gate_with_prob(audio, speech_prob); + }; + + let was_triggered = iter_state.triggered(); + let event = iter_state.update(speech_prob); + let is_triggered = iter_state.triggered(); + + if !is_triggered { + self.push_pre_roll(audio); + } else { + if !was_triggered { + if let VadIterEvent::Start { start_sample } = event { + self.iter_speech_start = Some(start_sample); + } else { + self.iter_speech_start = Some(iter_state.current_speech_start()); + } + if self.pre_roll_samples > 0 && !self.pre_roll.is_empty() { + self.pending_samples.extend(self.pre_roll.drain(..)); + } + } + self.pending_samples.extend_from_slice(audio); + } + + if let VadIterEvent::End { end_sample } = event { + if let Some(start_sample) = self.iter_speech_start.take() { + let speech_len = end_sample.saturating_sub(start_sample); + let mut target_len = self + .pre_roll_samples + .saturating_add(speech_len) + .saturating_add(self.speech_pad_samples); + if target_len == 0 { + target_len = self.pending_samples.len(); + } + if self.pending_samples.len() > target_len { + self.pending_samples.truncate(target_len); + } + } + self.in_speech = false; + self.speech_samples = 0; + self.silence_samples = 0; + self.pending_speech.clear(); + self.pending_silence.clear(); + return GateDecision { + audio: None, + ended: true, + }; + } + + GateDecision { + audio: None, + ended: false, + } + } + + fn push_pre_roll(&mut self, audio: &[f32]) { + if self.pre_roll_samples == 0 { + return; + } + for &sample in audio { + if self.pre_roll.len() >= self.pre_roll_samples { + self.pre_roll.pop_front(); + } + self.pre_roll.push_back(sample); + } + } + + pub fn output_sample_rate(&self) -> u32 { + self.output_sample_rate + } + + /// Current absolute raw sample cursor position (test-only accessor). + #[cfg(test)] + pub fn raw_cursor(&self) -> usize { + self.raw_cursor + } + + /// Current gate mode (test-only accessor). + #[cfg(test)] + pub fn gate_mode(&self) -> VadGateMode { + self.gate_mode + } + + /// Mapped VAD index to raw sample index (test-only accessor). + #[cfg(test)] + pub fn vad_to_raw_index_pub(&self, vad_index: usize) -> usize { + self.vad_to_raw_index(vad_index) + } + + /// Current VAD iterator sample counter (test-only accessor). + #[cfg(test)] + pub fn vad_current_sample(&self) -> Option { + self.iter_state.as_ref().map(|s| s.current_sample) + } + + /// Residual VAD resample buffer length (test-only accessor). + #[cfg(test)] + pub fn vad_resample_buf_len(&self) -> usize { + self.vad_resample_buf.len() + } + + /// Pre-roll size in raw samples (test-only accessor). + #[cfg(test)] + pub fn pre_roll_raw(&self) -> usize { + self.pre_roll_raw + } + + /// Raw audio buffer length (test-only accessor). + #[cfg(test)] + pub fn raw_buffer_len(&self) -> usize { + self.raw_buffer.len() + } + + fn vad_to_raw_index(&self, vad_index: usize) -> usize { + if self.raw_sample_rate == 0 { + return vad_index; + } + ((vad_index as f32 * self.raw_sample_rate as f32) / vad::VAD_SAMPLE_RATE as f32) + .round() + .max(0.0) as usize + } + + fn raw_slice(&self, start: usize, end: usize) -> Option> { + if end <= start { + return None; + } + if start < self.raw_buffer_start || end > self.raw_cursor { + return None; + } + let start_idx = start - self.raw_buffer_start; + let end_idx = end - self.raw_buffer_start; + if end_idx <= start_idx { + return None; + } + Some( + self.raw_buffer + .iter() + .skip(start_idx) + .take(end_idx.saturating_sub(start_idx)) + .cloned() + .collect(), + ) + } + + fn trim_raw_buffer(&mut self, keep_from: usize) { + if keep_from <= self.raw_buffer_start { + return; + } + let drop = keep_from - self.raw_buffer_start; + for _ in 0..drop.min(self.raw_buffer.len()) { + self.raw_buffer.pop_front(); + } + self.raw_buffer_start = keep_from; + } +} + +// ═══════════════════════════════════════════════════════════ +// Configuration helpers +// ═══════════════════════════════════════════════════════════ + +pub(crate) fn hardcoded_gate_config() -> GateConfig { + // Start from env-aware VadConfig::default() so CODESCRIBE_VAD_* env vars + // are always respected. Then apply streaming-specific overrides only for + // fields that don't have an explicit env override set. + let mut vad_cfg = vad::VadConfig::default(); + + // Streaming needs a tighter threshold than the batch default (0.35). + if std::env::var("CODESCRIBE_VAD_THRESHOLD").is_err() { + vad_cfg.threshold = 0.50; + } + // Short min-speech for responsive streaming. + if std::env::var("CODESCRIBE_VAD_MIN_SPEECH_SEC").is_err() { + vad_cfg.min_speech_duration_sec = 0.05; + } + // Short silence tolerance for streaming chunk boundaries. + if std::env::var("CODESCRIBE_VAD_SILENCE_SEC").is_err() { + vad_cfg.max_silence_duration_sec = 0.20; + } + // Silero reference pre-roll (64ms) for tight boundary padding. + if std::env::var("CODESCRIBE_VAD_PRE_ROLL_SEC").is_err() { + vad_cfg.pre_roll_sec = 0.064; + } + + // Derive gate-level pre_roll/speech_pad from the (env-aware) vad config, + // with a dedicated env override for speech_pad (not in VadConfig). + let pre_roll = vad_cfg.pre_roll_sec; + let speech_pad = std::env::var("CODESCRIBE_VAD_SPEECH_PAD_SEC") + .ok() + .and_then(|v| v.parse::().ok()) + .map(|v| v.clamp(0.0, 2.0)) + .unwrap_or(pre_roll); // default: mirror pre_roll + + GateConfig { + vad: vad_cfg, + pre_roll_sec: pre_roll, + speech_pad_sec: speech_pad, + mode: gate_mode_from_env(), + } +} + +pub(crate) fn init_silero_vad(sample_rate: u32, config: &vad::VadConfig) -> Option { + let model_path = vad::default_model_path(); + match vad::SileroVad::new(&model_path, config.clone()) { + Ok(mut vad) => { + vad.set_input_sample_rate(sample_rate); + tracing::info!("Silero VAD ready (model: {})", model_path.display()); + Some(vad) + } + Err(e) => { + tracing::warn!("Silero VAD init failed ({}): {}", model_path.display(), e); + None + } + } +} + +fn gate_mode_from_env() -> VadGateMode { + if env_bool("CODESCRIBE_VAD_ITER") { + return VadGateMode::Iter; + } + match std::env::var("CODESCRIBE_VAD_GATE_MODE") + .ok() + .map(|v| v.to_lowercase()) + .as_deref() + { + Some("supervisor") | Some("quality") | Some("managed") => VadGateMode::Supervisor, + Some("iter") | Some("vad_iter") | Some("silero_iter") => VadGateMode::Iter, + Some("simple") | Some("gate") | Some("basic") => VadGateMode::Simple, + _ => VadGateMode::Supervisor, + } +} + +fn env_bool(key: &str) -> bool { + std::env::var(key) + .ok() + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(false) +} + +// ═══════════════════════════════════════════════════════════ +// VAD iterator state machine +// ═══════════════════════════════════════════════════════════ + +struct GateDecision { + audio: Option>, + ended: bool, +} + +#[derive(Debug)] +struct VadIterState { + params: VadIterParams, + current_sample: usize, + temp_end: usize, + next_start: usize, + prev_end: usize, + triggered: bool, + speech_start: usize, +} + +#[derive(Debug)] +struct VadIterParams { + threshold: f32, + min_silence_samples: usize, + min_speech_samples: usize, + max_speech_samples: f32, + frame_size_samples: usize, + min_silence_samples_at_max_speech: usize, +} + +#[derive(Debug, Copy, Clone)] +enum VadIterEvent { + None, + Start { start_sample: usize }, + End { end_sample: usize }, +} + +impl VadIterState { + fn new(config: &GateConfig, sample_rate: u32) -> Self { + let sr_per_ms = sample_rate as f32 / 1000.0; + let frame_size_samples = vad::CHUNK_SIZE; + let min_silence_samples = (config.vad.max_silence_duration_sec * sample_rate as f32) + .round() + .max(1.0) as usize; + let min_speech_samples = (config.vad.min_speech_duration_sec * sample_rate as f32) + .round() + .max(1.0) as usize; + let speech_pad_samples = (config.speech_pad_sec * sample_rate as f32) + .round() + .max(0.0) as usize; + let max_speech_samples = config.vad.max_utterance_sec * sample_rate as f32 + - frame_size_samples as f32 + - 2.0 * speech_pad_samples as f32; + let min_silence_samples_at_max_speech = (sr_per_ms * 98.0).round() as usize; + + Self { + params: VadIterParams { + threshold: config.vad.threshold, + min_silence_samples, + min_speech_samples, + max_speech_samples, + frame_size_samples, + min_silence_samples_at_max_speech, + }, + current_sample: 0, + temp_end: 0, + next_start: 0, + prev_end: 0, + triggered: false, + speech_start: 0, + } + } + + fn reset(&mut self) { + self.current_sample = 0; + self.temp_end = 0; + self.next_start = 0; + self.prev_end = 0; + self.triggered = false; + self.speech_start = 0; + } + + fn triggered(&self) -> bool { + self.triggered + } + + fn current_speech_start(&self) -> usize { + self.speech_start + } + + fn update(&mut self, speech_prob: f32) -> VadIterEvent { + self.current_sample = self + .current_sample + .saturating_add(self.params.frame_size_samples); + let frame_start = self + .current_sample + .saturating_sub(self.params.frame_size_samples); + + if speech_prob > self.params.threshold { + if self.temp_end != 0 { + self.temp_end = 0; + if self.next_start < self.prev_end { + self.next_start = frame_start; + } + } + if !self.triggered { + self.triggered = true; + self.speech_start = frame_start; + return VadIterEvent::Start { + start_sample: frame_start, + }; + } + return VadIterEvent::None; + } + + if self.triggered + && (self.current_sample.saturating_sub(self.speech_start) as f32) + > self.params.max_speech_samples + { + if self.prev_end > 0 { + let end = self.prev_end; + if self.next_start < self.prev_end { + self.triggered = false; + } else { + self.speech_start = self.next_start; + } + self.prev_end = 0; + self.next_start = 0; + self.temp_end = 0; + return VadIterEvent::End { end_sample: end }; + } + + let end = self.current_sample; + self.triggered = false; + self.prev_end = 0; + self.next_start = 0; + self.temp_end = 0; + return VadIterEvent::End { end_sample: end }; + } + + let neg_threshold = (self.params.threshold - 0.15).max(0.05); + if self.triggered && speech_prob < neg_threshold { + if self.temp_end == 0 { + self.temp_end = self.current_sample; + } + if self.current_sample.saturating_sub(self.temp_end) + > self.params.min_silence_samples_at_max_speech + { + self.prev_end = self.temp_end; + } + if self.current_sample.saturating_sub(self.temp_end) >= self.params.min_silence_samples + { + let end = self.temp_end; + if end.saturating_sub(self.speech_start) > self.params.min_speech_samples { + self.triggered = false; + self.prev_end = 0; + self.next_start = 0; + self.temp_end = 0; + return VadIterEvent::End { end_sample: end }; + } + } + } + + VadIterEvent::None + } +} + +// ═══════════════════════════════════════════════════════════ +// Tests +// ═══════════════════════════════════════════════════════════ + +#[cfg(test)] +mod tests { + use super::*; + use serial_test::serial; + + #[test] + fn vad_iter_state_basic_lifecycle() { + let config = GateConfig { + vad: vad::VadConfig { + threshold: 0.50, + min_speech_duration_sec: 0.05, + max_silence_duration_sec: 0.20, + max_utterance_sec: 300.0, + pre_roll_sec: 0.064, + }, + pre_roll_sec: 0.064, + speech_pad_sec: 0.064, + mode: VadGateMode::Supervisor, + }; + let mut state = VadIterState::new(&config, 16000); + + // Initially not triggered + assert!(!state.triggered()); + + // Feed high speech probability — should trigger Start + let event = state.update(0.9); + assert!(state.triggered()); + assert!(matches!(event, VadIterEvent::Start { .. })); + + // Continue speech — should be None + let event = state.update(0.8); + assert!(matches!(event, VadIterEvent::None)); + + // Feed silence below neg_threshold for long enough → End + // min_silence_samples at 16kHz with 0.20s = 3200 samples + // Each update advances by frame_size_samples (512) + // So we need 3200/512 ≈ 7 frames of silence + for _ in 0..10 { + let event = state.update(0.01); + if matches!(event, VadIterEvent::End { .. }) { + assert!(!state.triggered()); + return; // Success + } + } + // If we got here, the speech was too short for min_speech check + // That's OK — the state machine correctly filters short bursts. + } + + #[test] + fn vad_iter_state_reset() { + let config = GateConfig { + vad: vad::VadConfig { + threshold: 0.50, + min_speech_duration_sec: 0.05, + max_silence_duration_sec: 0.20, + max_utterance_sec: 300.0, + pre_roll_sec: 0.064, + }, + pre_roll_sec: 0.064, + speech_pad_sec: 0.064, + mode: VadGateMode::Supervisor, + }; + let mut state = VadIterState::new(&config, 16000); + state.update(0.9); // trigger + assert!(state.triggered()); + state.reset(); + assert!(!state.triggered()); + assert_eq!(state.current_sample, 0); + } + + #[test] + #[serial] + fn gate_mode_default_is_supervisor() { + // Without env vars set, default should be Supervisor + let config = hardcoded_gate_config(); + assert_eq!(config.mode, VadGateMode::Supervisor); + } + + #[test] + #[serial] + fn test_gate_mode_respects_env() { + // Set env to Iter — constructors must NOT override to Supervisor. + // SAFETY: test runs single-threaded (cargo test default); no concurrent env reads. + unsafe { + std::env::set_var("CODESCRIBE_VAD_GATE_MODE", "iter"); + } + let stream = SpeechSession::new_stream(16000, 6.0, 1.0); + assert_eq!(stream.gate_mode(), VadGateMode::Iter); + let utterance = SpeechSession::new_utterance(16000); + assert_eq!(utterance.gate_mode(), VadGateMode::Iter); + unsafe { + std::env::remove_var("CODESCRIBE_VAD_GATE_MODE"); + } + } + + #[test] + fn test_utterance_pre_roll_nonzero() { + // new_utterance() must calculate pre_roll from config, not hardcode 0. + let session = SpeechSession::new_utterance(16000); + // Config has pre_roll_sec=0.064 → 16000*0.064 = 1024 samples + assert!( + session.pre_roll_raw() > 0, + "pre_roll_raw should be > 0, got {}", + session.pre_roll_raw() + ); + } + + #[test] + fn speech_event_variants() { + let chunk = SpeechEvent::Chunk(vec![1.0, 2.0]); + assert!(matches!(chunk, SpeechEvent::Chunk(v) if v.len() == 2)); + + let utt = SpeechEvent::Utterance(vec![3.0]); + assert!(matches!(utt, SpeechEvent::Utterance(v) if v.len() == 1)); + } + + /// Verify that raw_buffer is trimmed during long continuous speech in stream mode. + /// Without the trim fix, raw_buffer grows without bound. + #[test] + fn test_supervisor_stream_raw_buffer_bounded() { + // Use 16kHz to avoid resampling complexity (VAD native rate). + let sr = 16000u32; + let chunk_sec = 2.0f32; + let mut session = SpeechSession::new_stream(sr, chunk_sec, 0.0); + + if session.gate_mode() != VadGateMode::Supervisor { + eprintln!("Skipping: gate mode is not Supervisor"); + return; + } + + // Feed 30 seconds of "speech-like" audio (loud sine wave). + // With chunk_sec=2s this should produce ~15 chunk emissions. + let total_samples = sr as usize * 30; + let callback_size = 512usize; // Match VAD CHUNK_SIZE + let freq = 300.0f32; + let mut phase = 0.0f32; + let phase_inc = 2.0 * std::f32::consts::PI * freq / sr as f32; + + let mut total_chunks = 0usize; + let mut max_buffer_len = 0usize; + + for _ in 0..(total_samples / callback_size) { + let mut buf = Vec::with_capacity(callback_size); + for _ in 0..callback_size { + buf.push(phase.sin() * 0.8); // Loud enough for VAD + phase += phase_inc; + } + for event in session.feed(&buf, sr) { + if matches!(event, SpeechEvent::Chunk(_)) { + total_chunks += 1; + } + } + let current_len = session.raw_buffer_len(); + if current_len > max_buffer_len { + max_buffer_len = current_len; + } + } + + // The buffer should be bounded: at most ~chunk_limit + pre_roll + some margin. + // chunk_limit = 2s * 16000 = 32000 samples. + // Without the fix, max_buffer_len would be ~480000 (30s * 16000). + let chunk_limit = (chunk_sec * sr as f32) as usize; + let max_expected = chunk_limit * 3; // Generous bound: 3x chunk size + assert!( + max_buffer_len <= max_expected, + "raw_buffer grew to {} samples (max expected {}); memory leak likely", + max_buffer_len, + max_expected + ); + + // Should have produced at least a few chunks if VAD triggered. + // (VAD may not trigger on pure sine — this is a structural test, + // not a VAD accuracy test. If 0 chunks, the trim path wasn't exercised.) + if total_chunks > 0 { + println!( + "Supervisor stream: {} chunks emitted, max buffer {} samples (limit {})", + total_chunks, max_buffer_len, max_expected + ); + } + } +} diff --git a/core/audio/mod.rs b/core/audio/mod.rs index 369e1461..852d5404 100644 --- a/core/audio/mod.rs +++ b/core/audio/mod.rs @@ -8,6 +8,7 @@ //! //! Created by M&K (c)2026 VetCoders +pub mod chunker; pub mod loader; pub mod playback; pub mod recorder; diff --git a/core/audio/recorder.rs b/core/audio/recorder.rs index ba70787c..e91d976e 100644 --- a/core/audio/recorder.rs +++ b/core/audio/recorder.rs @@ -17,9 +17,9 @@ // snapshot_wav method (save current buffer without stopping) // // design rationale: uses cpal for cross-platform audio input. silence detection -// is based on root mean square (rms) of audio chunks compared -// to a db threshold. tokio is used for async collection to avoid -// blocking. saving to a temp file simplifies passing audio data +// uses Silero VAD neural network via centralized vad::VadConfig. +// tokio is used for async collection to avoid blocking. +// saving to a temp file simplifies passing audio data // to the transcription api. // // usage example: @@ -51,10 +51,10 @@ // } // ``` // -// configuration via environment variables: -// - CODESCRIBE_VAD_THRESHOLD: speech probability threshold 0.0-1.0 (default: 0.5) -// - CODESCRIBE_VAD_MAX_SILENCE_SEC: silence duration before auto-stop (default: 1.2) -// - AUTO_SILENCE: enable/disable silence detection (default: true) +// Configuration via environment variables (VAD segmentation lives in streaming recorder): +// - CODESCRIBE_VAD_THRESHOLD: speech probability 0.0-1.0 (default: 0.35) +// - CODESCRIBE_VAD_SILENCE_SEC: silence before utterance flush (default: 2.5s) +// - (no VAD enable flag; segmentation is always handled by the streaming recorder) use crate::vad; use anyhow::{Context, Result}; @@ -64,7 +64,6 @@ use hound::{WavSpec, WavWriter}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; -use tokio::sync::mpsc; use tracing::{debug, error, info, warn}; // --- constants --- @@ -76,14 +75,8 @@ const SAMPLE_RATE: u32 = 16000; /// Number of channels (1 for mono) const CHANNELS: u16 = 1; -/// Speech probability threshold (0.0-1.0) for VAD -/// Below this is considered silence. Default: 0.5 -const DEFAULT_SPEECH_THRESHOLD: f32 = 0.5; - -/// Silence duration threshold (seconds) -/// Recording stops automatically after this duration of continuous silence. -/// Synced with VadConfig::max_silence_duration_sec default (1.2s) -const DEFAULT_HANG_SEC: f32 = 1.2; +// VAD defaults are sourced from vad::VadConfig - no local constants needed. +// See core/vad/config.rs for threshold (0.35) and max_silence (2.5s) defaults. /// Size of audio chunks to read from stream (samples) const BLOCK_SIZE: usize = 1024; @@ -96,13 +89,6 @@ pub struct RecorderConfig { pub sample_rate: u32, /// Number of audio channels pub channels: u16, - /// Speech probability threshold (0.0-1.0) for VAD - /// Below this is considered silence - pub speech_threshold: f32, - /// Hang time - silence duration before auto-stop (seconds) - pub hang_sec: f32, - /// Enable automatic silence detection - pub auto_silence: bool, /// Block size for audio chunks pub block_size: usize, } @@ -112,19 +98,6 @@ impl Default for RecorderConfig { Self { sample_rate: SAMPLE_RATE, channels: CHANNELS, - speech_threshold: std::env::var("CODESCRIBE_VAD_THRESHOLD") - .ok() - .and_then(|s| s.parse().ok()) - .unwrap_or(DEFAULT_SPEECH_THRESHOLD) - .clamp(0.1, 0.9), - hang_sec: std::env::var("CODESCRIBE_VAD_MAX_SILENCE_SEC") - .ok() - .and_then(|s| s.parse().ok()) - .unwrap_or(DEFAULT_HANG_SEC) - .clamp(0.1, 10.0), - auto_silence: std::env::var("AUTO_SILENCE") - .map(|v| !matches!(v.to_lowercase().as_str(), "0" | "false" | "no" | "off")) - .unwrap_or(true), block_size: BLOCK_SIZE, } } @@ -153,14 +126,11 @@ pub struct Recorder { stream: Option, device: Option, is_recording: Arc, - stop_tx: Option>, last_duration: f32, diagnostics: RecorderDiagnostics, /// Actual sample rate used for recording (may differ from config) actual_sample_rate: u32, on_data: Option, - /// Callback invoked when VAD (silence detection) stops recording - on_vad_stop: Option>, } // Safety: Recorder can be sent between threads because: @@ -195,12 +165,10 @@ impl Recorder { stream: None, device: None, is_recording: Arc::new(AtomicBool::new(false)), - stop_tx: None, last_duration: 0.0, diagnostics: RecorderDiagnostics::default(), actual_sample_rate: config.sample_rate, // Will be updated in start() on_data: None, - on_vad_stop: None, }) } @@ -209,14 +177,6 @@ impl Recorder { self.on_data = Some(callback); } - /// Set a callback invoked when VAD (silence detection) stops recording - pub fn set_on_vad_stop(&mut self, callback: F) - where - F: Fn() + Send + Sync + 'static, - { - self.on_vad_stop = Some(Arc::new(callback)); - } - /// Actual sample rate used by the underlying input stream. /// /// Note: This may differ from `config.sample_rate` because we always open @@ -241,7 +201,7 @@ impl Recorder { let model_path = vad::default_model_path(); if let Err(e) = vad::init(&model_path) { warn!( - "VAD init failed ({}): {} - auto-stop disabled, speech_probability will return 1.0", + "VAD init failed ({}): {} - segmentation disabled, speech_probability will return 1.0", model_path.display(), e ); @@ -318,24 +278,10 @@ impl Recorder { // Store actual sample rate for WAV file and duration calculations self.actual_sample_rate = native_sample_rate; - // Use actual sample rate for silence detection calculations - let sample_rate = native_sample_rate; - - // Create channel for stopping - let (stop_tx, mut stop_rx) = mpsc::channel::<()>(1); - self.stop_tx = Some(stop_tx); - // Setup stream callback let buffer = Arc::clone(&self.buffer); let is_recording_data = Arc::clone(&self.is_recording); let is_recording_error = Arc::clone(&self.is_recording); - let speech_threshold = self.config.speech_threshold; - let hang_sec = self.config.hang_sec; - let auto_silence = self.config.auto_silence; - // sample_rate is already set above to native_sample_rate - - let silent_frames = Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let silent_frames_clone = Arc::clone(&silent_frames); let on_data = self.on_data.take(); let stream = device @@ -357,30 +303,7 @@ impl Recorder { } } - // Use Silero VAD for speech detection (with automatic resampling to 16kHz) - let speech_prob = vad::speech_probability(data, sample_rate); - let is_silence = speech_prob < speech_threshold; - - // Only check silence if still recording (avoid spam after stop) - if auto_silence && is_recording_data.load(Ordering::SeqCst) { - // Check for silence using VAD - if is_silence { - silent_frames_clone.fetch_add(data.len(), Ordering::SeqCst); - } else { - silent_frames_clone.store(0, Ordering::SeqCst); - } - - // Check if silence duration exceeds hang time - let current_silent = silent_frames_clone.load(Ordering::SeqCst); - let silent_duration = current_silent as f32 / sample_rate as f32; - if silent_duration > hang_sec { - info!( - "Silence detected for > {:.2}s. Stopping collection.", - hang_sec - ); - is_recording_data.store(false, Ordering::SeqCst); - } - } + let _ = is_recording_data.load(Ordering::SeqCst); }, move |err| { error!("Audio stream error: {}", err); @@ -399,35 +322,6 @@ impl Recorder { self.stream = Some(stream); self.device = Some(device); - // Spawn monitoring task - let is_recording_clone = Arc::clone(&self.is_recording); - let stop_tx_clone = self.stop_tx.clone(); - let on_vad_stop_clone = self.on_vad_stop.clone(); - tokio::spawn(async move { - loop { - tokio::select! { - _ = stop_rx.recv() => { - debug!("Stop signal received"); - break; - } - _ = tokio::time::sleep(tokio::time::Duration::from_millis(100)) => { - if !is_recording_clone.load(Ordering::SeqCst) { - debug!("Recording stopped by silence detection"); - // Invoke VAD callback if set - if let Some(ref callback) = on_vad_stop_clone { - info!("VAD triggered - invoking callback"); - callback(); - } - if let Some(tx) = stop_tx_clone.as_ref() { - let _ = tx.send(()).await; - } - break; - } - } - } - } - }); - Ok(()) } @@ -439,6 +333,17 @@ impl Recorder { /// Returns the absolute path to the saved .wav file, or None if no audio /// was recorded or an error occurred. pub async fn stop(&mut self) -> Result> { + self.stop_internal(true).await + } + + /// Stops the audio recording without saving a WAV file. + /// + /// Returns None if no audio was recorded or an error occurred. + pub async fn stop_without_saving(&mut self) -> Result> { + self.stop_internal(false).await + } + + async fn stop_internal(&mut self, save_wav: bool) -> Result> { if !self.is_recording.load(Ordering::SeqCst) && self.stream.is_none() { warn!("Stop called but no active stream"); self.last_duration = 0.0; @@ -447,11 +352,6 @@ impl Recorder { info!("Stopping recording..."); - // Signal stop - if let Some(tx) = self.stop_tx.take() { - let _ = tx.send(()).await; - } - // Stop stream if let Some(stream) = self.stream.take() { drop(stream); // Dropping the stream stops it @@ -462,17 +362,14 @@ impl Recorder { self.is_recording.store(false, Ordering::SeqCst); // Get buffer data - let wav_data = { - let buf = self.buffer.lock().unwrap_or_else(|e| e.into_inner()); - if buf.is_empty() { - warn!("No audio data captured"); - self.last_duration = 0.0; - return Ok(None); - } - buf.clone() - }; + let mut buf = self.buffer.lock().unwrap_or_else(|e| e.into_inner()); + if buf.is_empty() { + warn!("No audio data captured"); + self.last_duration = 0.0; + return Ok(None); + } - let num_frames = wav_data.len(); + let num_frames = buf.len(); self.last_duration = num_frames as f32 / self.actual_sample_rate as f32; self.diagnostics.frames = num_frames; self.diagnostics.bytes = num_frames * std::mem::size_of::(); @@ -483,6 +380,13 @@ impl Recorder { num_frames, self.last_duration, self.actual_sample_rate ); + if !save_wav { + buf.clear(); + return Ok(None); + } + + let wav_data = buf.clone(); + // Create temp file let temp_path = std::env::temp_dir().join(format!( "codescribe_recording_{}.wav", @@ -502,10 +406,7 @@ impl Recorder { info!("Audio successfully saved to WAV file"); // Clear buffer - self.buffer - .lock() - .unwrap_or_else(|e| e.into_inner()) - .clear(); + buf.clear(); Ok(Some(temp_path)) } @@ -513,7 +414,10 @@ impl Recorder { impl Default for Recorder { fn default() -> Self { - Self::new().expect("Failed to create default recorder") + Self::new().unwrap_or_else(|e| { + error!("Recorder::default() failed: {e}"); + panic!("Cannot create default Recorder: {e}"); + }) } } @@ -530,7 +434,7 @@ impl Drop for Recorder { // --- helper functions --- -// Note: RMS-based silence detection replaced with WebRTC VAD (see vad module) +// Note: RMS-based silence detection replaced with Silero VAD neural network (see vad module) /// Write audio samples to a WAV file. fn write_wav_file(path: &PathBuf, samples: &[i16], sample_rate: u32, channels: u16) -> Result<()> { @@ -559,7 +463,7 @@ fn write_wav_file(path: &PathBuf, samples: &[i16], sample_rate: u32, channels: u mod tests { use super::*; - // Note: RMS tests removed - now using WebRTC VAD (see vad module tests) + // Note: RMS tests removed - now using Silero VAD neural network (see vad module tests) #[test] fn test_recorder_config_default() { diff --git a/core/audio/streaming_recorder.rs b/core/audio/streaming_recorder.rs index 8d1d56bf..c06035b6 100644 --- a/core/audio/streaming_recorder.rs +++ b/core/audio/streaming_recorder.rs @@ -1,32 +1,27 @@ use crate::audio::recorder::{Recorder, RecorderConfig}; -use crate::stream_postprocess::StreamPostProcessor; -use crate::stt::whisper; -use crate::stt::whisper::append_with_overlap_dedup; -use crate::stt::whisper::singleton::engine as get_engine; -use crate::vad; -use anyhow::{Context, Result, anyhow}; -use std::collections::VecDeque; +use crate::pipeline::stream_postprocess::StreamPostProcessor; +use crate::pipeline::streaming::{ + buffered_transcription_worker, env_bool_default, stream_log_path, transcription_worker, +}; +use anyhow::{Context, Result}; use std::sync::Arc; -use std::{fs::OpenOptions, io::Write, path::Path}; use tokio::sync::{Mutex, mpsc}; use tokio::task::JoinHandle; -use tokio::time::{Duration, Instant}; -use tracing::{debug, error, info}; +use tracing::{debug, info}; -const DEFAULT_CHUNK_DURATION_SEC: f32 = 15.0; -const DEFAULT_OVERLAP_RATIO: f32 = 0.25; // 25% overlap for context -// VAD config now centralized in core/vad/config.rs (CODESCRIBE_VAD_* env vars) -const DEFAULT_BUFFER_DELAY_MS: u64 = 3000; -const DEFAULT_TYPING_CPS: f32 = 30.0; +// Re-export public API that was moved to pipeline::streaming +#[allow(deprecated)] +pub use crate::pipeline::streaming::StreamDeltaCallback; +pub use crate::pipeline::streaming::transcribe_streaming_samples; -pub type StreamDeltaCallback = Arc; +use crate::pipeline::contracts::DeltaSink; pub struct StreamingRecorder { pub recorder: Recorder, transcript_buffer: Arc>, transcription_handle: Option>, sample_rate: u32, - delta_callback: Option, + delta_callback: Option>, } impl StreamingRecorder { @@ -56,25 +51,31 @@ impl StreamingRecorder { }) } - pub fn set_delta_callback(&mut self, callback: Option) { + pub fn set_delta_callback(&mut self, callback: Option>) { self.delta_callback = callback; } pub async fn start(&mut self, language: Option) -> Result<()> { + let use_buffered_stream = env_bool_default("CODESCRIBE_BUFFERED_STREAM", true); + self.start_with_buffered(language, use_buffered_stream) + .await + } + + pub async fn start_with_buffered( + &mut self, + language: Option, + use_buffered_stream: bool, + ) -> Result<()> { // Clear previous transcript *self.transcript_buffer.lock().await = String::new(); // Create channel for audio chunks - // Buffer size: enough to hold a few seconds if worker is slow let (tx, rx) = mpsc::channel::>(500); // Setup callback to send audio data - // Note: try_send to avoid blocking audio thread self.recorder.set_callback(Box::new(move |data| { if let Err(_e) = tx.try_send(data.to_vec()) { // If channel is full, we drop audio (better than blocking) - // But we should log occasionally? - // For now just ignore or print to stderr if needed, but avoid spamming logs } })); @@ -82,7 +83,6 @@ impl StreamingRecorder { self.recorder.start().await?; // Update sample rate to the one used by the input stream. - // This is critical: we must pass the correct `sample_rate` to Whisper so it can resample. let actual_sample_rate = self.recorder.actual_sample_rate(); if actual_sample_rate != self.sample_rate { info!( @@ -96,9 +96,8 @@ impl StreamingRecorder { // Start transcription worker (after we know the real sample rate) let transcript_buffer = self.transcript_buffer.clone(); - let stream_log_path = stream_log_path(); + let log_path = stream_log_path(); let delta_callback = self.delta_callback.clone(); - let use_buffered_stream = env_bool_default("CODESCRIBE_BUFFERED_STREAM", true); self.transcription_handle = Some(tokio::spawn(async move { if use_buffered_stream { buffered_transcription_worker( @@ -107,11 +106,13 @@ impl StreamingRecorder { actual_sample_rate, language, delta_callback, - stream_log_path, + log_path, ) .await; } else { - let postprocessor = StreamPostProcessor::new(); + // Always-on contract: every transcript passes through postprocessor + // (lexicon + cleanup + semantic gate) regardless of buffered mode. + let postprocessor = Some(StreamPostProcessor::new()); transcription_worker( rx, transcript_buffer, @@ -119,7 +120,7 @@ impl StreamingRecorder { language, postprocessor, delta_callback, - stream_log_path, + log_path, ) .await; } @@ -144,646 +145,23 @@ impl StreamingRecorder { let transcript = self.transcript_buffer.lock().await.clone(); Ok((transcript, audio_path)) } -} - -async fn transcription_worker( - mut chunk_receiver: mpsc::Receiver>, - transcript_buffer: Arc>, - sample_rate: u32, - language: Option, - mut postprocessor: StreamPostProcessor, - delta_callback: Option, - stream_log_path: Option, -) { - info!("Transcription worker started"); - - let mut pending_samples: Vec = Vec::new(); - let chunk_duration_sec = stream_chunk_duration_sec(); - let overlap_sec = stream_overlap_sec(chunk_duration_sec); - let chunk_limit = (sample_rate as f32 * chunk_duration_sec) as usize; - let overlap_size = (sample_rate as f32 * overlap_sec) as usize; - - // We keep track of how many samples we've processed to know when to overlap - // Actually, we just keep the last samples in pending_samples? - // No, pending_samples grows. When it hits limit, we transcribe. - // Then we keep the tail as the new pending_samples. - - while let Some(mut data) = chunk_receiver.recv().await { - pending_samples.append(&mut data); - - if pending_samples.len() >= chunk_limit { - process_chunk( - &pending_samples, - &transcript_buffer, - sample_rate, - language.as_deref(), - &mut postprocessor, - delta_callback.as_ref(), - stream_log_path.as_deref(), - ) - .await; - - // Keep overlap for next chunk - if pending_samples.len() > overlap_size { - let start_idx = pending_samples.len() - overlap_size; - pending_samples = pending_samples[start_idx..].to_vec(); - } else { - // Should not happen if chunk_limit > overlap_size - pending_samples.clear(); - } - } - } - - // Process remaining samples (final chunk) - if !pending_samples.is_empty() { - debug!("Processing final chunk ({} samples)", pending_samples.len()); - process_chunk( - &pending_samples, - &transcript_buffer, - sample_rate, - language.as_deref(), - &mut postprocessor, - delta_callback.as_ref(), - stream_log_path.as_deref(), - ) - .await; - } - - info!("Transcription worker finished"); -} - -// VADConfig removed - now using centralized vad::VadConfig from core/vad/config.rs - -struct VADSegmenter { - pending_samples: Vec, - sample_rate: u32, - /// Speech probability threshold (0.0-1.0) - speech_threshold: f32, - silence_duration_sec: f32, - max_utterance_sec: f32, - pre_roll_samples: usize, - silence_frames: usize, - is_in_speech: bool, -} - -impl VADSegmenter { - fn new(sample_rate: u32) -> Self { - Self::with_config(sample_rate, vad::VadConfig::default()) - } - - fn with_config(sample_rate: u32, config: vad::VadConfig) -> Self { - let pre_roll_samples = (config.pre_roll_sec * sample_rate as f32).round().max(1.0) as usize; - - // Ensure VAD is initialized with the passed config (not default!) - // Note: if VAD already initialized, this is a no-op (early exit) - if let Err(e) = vad::init_with_config(&vad::default_model_path(), config.clone()) { - tracing::warn!( - "VAD init failed in VADSegmenter: {} - silence detection disabled, \ - will rely on max_utterance_sec ({:.1}s) for flush", - e, - config.max_utterance_sec - ); - } - - Self { - pending_samples: Vec::new(), - sample_rate, - speech_threshold: config.threshold, - silence_duration_sec: config.max_silence_duration_sec, - max_utterance_sec: config.max_utterance_sec, - pre_roll_samples, - silence_frames: 0, - is_in_speech: false, - } - } - - fn feed(&mut self, audio: &[f32]) -> Option> { - if audio.is_empty() { - return None; - } - - self.pending_samples.extend_from_slice(audio); - - // Use Silero VAD for speech detection (with automatic resampling to 16kHz) - let speech_prob = vad::speech_probability(audio, self.sample_rate); - let is_silence = speech_prob < self.speech_threshold; - - if is_silence { - if self.is_in_speech { - self.silence_frames = self.silence_frames.saturating_add(audio.len()); - let silence_duration = self.silence_frames as f32 / self.sample_rate as f32; - if silence_duration >= self.silence_duration_sec { - // Cut before silence — don't feed trailing silence to Whisper - let utterance_end = self - .pending_samples - .len() - .saturating_sub(self.silence_frames); - return self.split_at(utterance_end); - } - } else if self.pending_samples.len() > self.pre_roll_samples { - // Trim leading silence to prevent unbounded buffer growth - let start_idx = self.pending_samples.len() - self.pre_roll_samples; - self.pending_samples = self.pending_samples[start_idx..].to_vec(); - } - } else { - self.is_in_speech = true; - self.silence_frames = 0; - } - - let max_samples = (self.max_utterance_sec * self.sample_rate as f32) as usize; - if self.pending_samples.len() >= max_samples { - return self.split_at(self.pending_samples.len()); - } - - None - } - - fn flush(&mut self) -> Option> { - if self.pending_samples.is_empty() { - return None; - } - let utterance = std::mem::take(&mut self.pending_samples); - self.silence_frames = 0; - self.is_in_speech = false; - Some(utterance) - } - - /// Split utterance at given position. Keeps pre-roll from speech end. - fn split_at(&mut self, utterance_end: usize) -> Option> { - if utterance_end == 0 { - self.pending_samples.clear(); - self.silence_frames = 0; - self.is_in_speech = false; - return None; - } - - let utterance = self.pending_samples[..utterance_end].to_vec(); - let pre_roll_start = utterance_end.saturating_sub(self.pre_roll_samples); - self.pending_samples = self.pending_samples[pre_roll_start..utterance_end].to_vec(); - self.silence_frames = 0; - self.is_in_speech = false; - Some(utterance) - } -} - -struct TranscriptionPipeline { - language: Option, - postprocessor: StreamPostProcessor, -} - -impl TranscriptionPipeline { - fn new(language: Option) -> Self { - Self { - language, - postprocessor: StreamPostProcessor::new(), - } - } - - fn postprocess(&mut self, text: &str) -> Option { - self.postprocessor.process_utterance(text) - } -} - -struct BufferedEmitter { - queue: VecDeque, - initial_delay_ms: u64, - typing_speed_cps: f32, - first_output_at: Option, - current_segment: Option, - current_index: usize, - current_len: usize, - delta_callback: Option, - transcript_buffer: Arc>, - stream_log_path: Option, - finished: bool, - has_output: bool, -} - -impl BufferedEmitter { - fn new( - transcript_buffer: Arc>, - delta_callback: Option, - stream_log_path: Option, - ) -> Self { - Self { - queue: VecDeque::new(), - initial_delay_ms: env_u64("CODESCRIBE_BUFFER_DELAY_MS", DEFAULT_BUFFER_DELAY_MS), - typing_speed_cps: env_f32("CODESCRIBE_TYPING_CPS", DEFAULT_TYPING_CPS).max(5.0), - first_output_at: None, - current_segment: None, - current_index: 0, - current_len: 0, - delta_callback, - transcript_buffer, - stream_log_path, - finished: false, - has_output: false, - } - } - - fn push_segment(&mut self, text: String) { - if text.trim().is_empty() { - return; - } - let mut segment = text; - if !segment.starts_with(char::is_whitespace) - && (self.has_output || self.current_segment.is_some() || !self.queue.is_empty()) - { - segment = format!(" {}", segment); - } - self.queue.push_back(segment); - if self.first_output_at.is_none() { - self.first_output_at = Some(Instant::now()); - } - } - - async fn tick(&mut self) -> bool { - if self.finished && self.queue.is_empty() && self.current_segment.is_none() { - return true; - } - - if self.is_buffering() { - return false; - } - - if self.current_segment.is_none() { - self.current_segment = self.queue.pop_front(); - self.current_index = 0; - self.current_len = self - .current_segment - .as_ref() - .map(|segment| segment.chars().count()) - .unwrap_or(0); - } - - if let Some(segment) = self.current_segment.as_ref() - && let Some(ch) = segment.chars().nth(self.current_index) - { - let delta = ch.to_string(); - self.current_index += 1; - self.has_output = true; - - if self.current_index >= self.current_len { - self.current_segment = None; - self.current_index = 0; - self.current_len = 0; - } - - { - let mut buffer = self.transcript_buffer.lock().await; - buffer.push_str(&delta); - } - - if let Some(callback) = &self.delta_callback { - callback(&delta); - } - - if let Some(path) = self.stream_log_path.as_deref() { - let _ = append_to_stream_log(path, &delta); - } - } - - self.finished && self.queue.is_empty() && self.current_segment.is_none() - } - - fn is_buffering(&self) -> bool { - let Some(start) = self.first_output_at else { - return true; - }; - start.elapsed() < Duration::from_millis(self.initial_delay_ms) - } - - fn finish(&mut self) { - self.finished = true; - } -} - -async fn emitter_tick_loop(emitter: Arc>) { - let interval = { - let guard = emitter.lock().await; - Duration::from_secs_f32(1.0 / guard.typing_speed_cps) - }; - let mut ticker = tokio::time::interval(interval); - - loop { - ticker.tick().await; - let should_stop = { - let mut guard = emitter.lock().await; - guard.tick().await - }; - if should_stop { - break; - } - } -} - -async fn buffered_transcription_worker( - mut chunk_receiver: mpsc::Receiver>, - transcript_buffer: Arc>, - sample_rate: u32, - language: Option, - delta_callback: Option, - stream_log_path: Option, -) { - info!("Buffered transcription worker started"); - - let mut segmenter = VADSegmenter::new(sample_rate); - let mut pipeline = TranscriptionPipeline::new(language); - let emitter = Arc::new(Mutex::new(BufferedEmitter::new( - transcript_buffer.clone(), - delta_callback, - stream_log_path, - ))); - - let emitter_handle = tokio::spawn(emitter_tick_loop(emitter.clone())); - - while let Some(data) = chunk_receiver.recv().await { - if let Some(utterance) = segmenter.feed(&data) - && let Err(e) = handle_utterance(utterance, sample_rate, &mut pipeline, &emitter).await - { - error!("Buffered transcription failed: {}", e); - } - } - - if let Some(utterance) = segmenter.flush() - && let Err(e) = handle_utterance(utterance, sample_rate, &mut pipeline, &emitter).await - { - error!("Final buffered transcription failed: {}", e); - } - - { - let mut guard = emitter.lock().await; - guard.finish(); - } - - if let Err(e) = emitter_handle.await { - error!("Buffered emitter task failed: {}", e); - } - - info!("Buffered transcription worker finished"); -} - -async fn handle_utterance( - utterance: Vec, - sample_rate: u32, - pipeline: &mut TranscriptionPipeline, - emitter: &Arc>, -) -> Result<()> { - if utterance.is_empty() { - return Ok(()); - } - - let language = pipeline.language.clone(); - let raw_text = tokio::task::spawn_blocking(move || { - whisper::transcribe(&utterance, sample_rate, language.as_deref()) - }) - .await??; - - if let Some(cleaned) = pipeline.postprocess(&raw_text) { - let mut guard = emitter.lock().await; - guard.push_segment(cleaned); - } - - Ok(()) -} - -async fn process_chunk( - samples: &[f32], - transcript_buffer: &Arc>, - sample_rate: u32, - language: Option<&str>, - postprocessor: &mut StreamPostProcessor, - delta_callback: Option<&StreamDeltaCallback>, - stream_log_path: Option<&Path>, -) { - if samples.is_empty() { - return; - } - - let samples_owned = samples.to_vec(); - let lang_owned = language.map(String::from); - - // Run in blocking task - let result = tokio::task::spawn_blocking(move || { - let engine_mutex = match get_engine() { - Ok(m) => m, - Err(e) => return Err(anyhow!("Engine error: {}", e)), - }; - - let mut engine_guard = match engine_mutex.lock() { - Ok(g) => g, - Err(e) => return Err(anyhow!("Lock error: {}", e)), - }; - - // If sample_rate is not 16k, engine handles resampling? - // transcribe_samples_16k expects 16k. - // But our Recorder is configured for 16k (SAMPLE_RATE constant). - // However, Recorder might use native rate. - // Recorder::start() sets actual_sample_rate. - // If actual_sample_rate != 16k, we need to resample. - // Current implementation passes raw samples. - // transcribe_samples_16k assumes 16k. - // transcribe_with_language handles resampling. - - // Wait, engine.transcribe_samples_16k is specific. - // engine.transcribe_with_language(audio, sample_rate, language) handles everything. - // Let's use that one to be safe, or check if we need 16k. - - // The plan says "transcribe_samples_16k() - transcribes raw f32, zero I/O". - // If we use transcribe_with_language, it calls transcribe_long_with_language -> detect_language -> ... - // transcribe_samples_16k is lower level. - - // If sample_rate is 16000, we can use transcribe_samples_16k directly? - // Yes, but we should be robust. - // Let's use transcribe_with_language which handles resampling if needed. - // It's safer. - - engine_guard.transcribe_with_language(&samples_owned, sample_rate, lang_owned.as_deref()) - }) - .await; - - match result { - Ok(Ok(text)) => { - if !text.trim().is_empty() { - debug!("Chunk transcribed: '{}'", text.trim()); - if let Some(cleaned) = postprocessor.process(&text) { - let mut buffer = transcript_buffer.lock().await; - let before_len = buffer.len(); - append_with_overlap_dedup(&mut buffer, &cleaned); - if let Some(delta) = buffer.get(before_len..) - && !delta.trim().is_empty() - { - if let Some(callback) = delta_callback { - callback(delta); - } - - // Log to file if enabled - if let Some(path) = stream_log_path { - let _ = append_to_stream_log(path, delta); - } - } - } else { - debug!("Stream postprocessor dropped chunk"); - } - } - } - Ok(Err(e)) => { - error!("Chunk transcription failed: {}", e); - } - Err(e) => { - error!("Transcription task join error: {}", e); - } - } -} -fn stream_log_path() -> Option { - if let Ok(path) = std::env::var("CODESCRIBE_STREAM_LOG_PATH") { - let trimmed = path.trim(); - if !trimmed.is_empty() { - return Some(std::path::PathBuf::from(trimmed)); - } - } - - if env_bool("CODESCRIBE_STREAM_LOG") { - let root = crate::config::Config::config_dir(); - return Some(root.join("stream.log")); - } - - None -} - -fn append_to_stream_log(path: &Path, text: &str) -> std::io::Result<()> { - if let Some(parent) = path.parent() { - std::fs::create_dir_all(parent)?; - } - - let mut file = OpenOptions::new().create(true).append(true).open(path)?; - writeln!(file, "{}", text.trim_end())?; - Ok(()) -} + pub async fn stop_without_saving(&mut self) -> Result { + info!("Stopping streaming recorder (no WAV)..."); -fn env_bool(key: &str) -> bool { - std::env::var(key) - .ok() - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) - .unwrap_or(false) -} - -fn env_bool_default(key: &str, default: bool) -> bool { - std::env::var(key) - .ok() - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) - .unwrap_or(default) -} - -fn env_f32(key: &str, default: f32) -> f32 { - std::env::var(key) - .ok() - .and_then(|v| v.parse::().ok()) - .unwrap_or(default) -} - -fn env_u64(key: &str, default: u64) -> u64 { - std::env::var(key) - .ok() - .and_then(|v| v.parse::().ok()) - .unwrap_or(default) -} - -fn stream_chunk_duration_sec() -> f32 { - env_f32("CODESCRIBE_STREAM_CHUNK_SEC", DEFAULT_CHUNK_DURATION_SEC).clamp(0.5, 30.0) -} - -fn stream_overlap_sec(chunk_duration_sec: f32) -> f32 { - let ratio = env_f32("CODESCRIBE_STREAM_OVERLAP_RATIO", DEFAULT_OVERLAP_RATIO).clamp(0.05, 0.8); - (chunk_duration_sec * ratio).min(chunk_duration_sec * 0.8) -} - -pub fn transcribe_streaming_samples( - samples: &[f32], - sample_rate: u32, - language: Option<&str>, - mut postprocessor: Option<&mut StreamPostProcessor>, -) -> Result { - if samples.is_empty() { - return Ok(String::new()); - } - - let chunk_duration_sec = stream_chunk_duration_sec(); - let overlap_sec = stream_overlap_sec(chunk_duration_sec); - let chunk_limit = (sample_rate as f32 * chunk_duration_sec) as usize; - let overlap_size = (sample_rate as f32 * overlap_sec) as usize; - let step = chunk_limit.saturating_sub(overlap_size).max(1); - - let total_audio_sec = samples.len() as f32 / sample_rate as f32; - let stride_sec = chunk_duration_sec - overlap_sec; - let n_chunks = - ((samples.len().saturating_sub(chunk_limit)) as f32 / step as f32).ceil() as usize + 1; - let processing_factor = chunk_duration_sec / stride_sec; - let effective_audio_sec = n_chunks as f32 * chunk_duration_sec; - - info!( - "[STREAM_DIAG] chunk={:.1}s overlap={:.1}s stride={:.1}s | audio={:.1}s chunks={} factor={:.2}x effective={:.1}s", - chunk_duration_sec, - overlap_sec, - stride_sec, - total_audio_sec, - n_chunks, - processing_factor, - effective_audio_sec - ); - - let engine_mutex = get_engine()?; - let mut engine = engine_mutex - .lock() - .map_err(|e| anyhow!("Lock error: {}", e))?; - - let mut out = String::new(); - let mut offset = 0usize; - let mut chunks_processed = 0usize; - let t_start = std::time::Instant::now(); - - while offset < samples.len() { - let end = (offset + chunk_limit).min(samples.len()); - let chunk = &samples[offset..end]; - let chunk_sec = chunk.len() as f32 / sample_rate as f32; - let t_chunk = std::time::Instant::now(); - let text = engine.transcribe_with_language(chunk, sample_rate, language)?; - let chunk_ms = t_chunk.elapsed().as_millis(); - chunks_processed += 1; - - debug!( - "[STREAM_CHUNK] #{} offset={:.1}s len={:.1}s transcribe={}ms words={}", - chunks_processed, - offset as f32 / sample_rate as f32, - chunk_sec, - chunk_ms, - text.split_whitespace().count() - ); + // 1. Stop recording without writing a WAV file + let _ = self.recorder.stop_without_saving().await?; - if let Some(processor) = postprocessor.as_deref_mut() { - if let Some(cleaned) = processor.process(&text) { - append_with_overlap_dedup(&mut out, &cleaned); - } - } else { - append_with_overlap_dedup(&mut out, &text); + // 2. Wait for worker to finish processing remaining chunks + if let Some(handle) = self.transcription_handle.take() { + debug!("Waiting for transcription worker to finish..."); + handle.await.context("Transcription worker failed")?; } - if end == samples.len() { - break; - } - offset = offset.saturating_add(step); + // 3. Return collected transcript + let transcript = self.transcript_buffer.lock().await.clone(); + Ok(transcript) } - - let total_ms = t_start.elapsed().as_millis(); - info!( - "[STREAM_DONE] chunks_processed={} total_ms={} out_words={}", - chunks_processed, - total_ms, - out.split_whitespace().count() - ); - - Ok(out) } // Note: calculate_rms_db removed - now using vad::speech_probability for voice detection @@ -791,51 +169,23 @@ pub fn transcribe_streaming_samples( #[cfg(test)] mod tests { use super::*; + use crate::audio::chunker::{SpeechEvent, SpeechSession}; use crate::audio::load_audio_file; use crate::stt::whisper; + use crate::vad; use serial_test::serial; use std::fs; use std::path::{Path, PathBuf}; + use tokio::time::Duration; #[test] - #[ignore] // Requires Silero VAD model (run with: cargo test -- --ignored) - fn test_vad_segmenter_flush_on_silence() { - // Initialize VAD - skip if model unavailable - let model_path = vad::default_model_path(); - if !model_path.exists() { - eprintln!( - "Skipping: Silero VAD model not found at {}", - model_path.display() - ); + #[ignore] // Manual: requires microphone + Silero model (set CODESCRIBE_E2E_MIC=1) + fn test_vad_gate_live_chunk_sizes() { + if !env_bool("CODESCRIBE_E2E_MIC") { + eprintln!("Skipping mic gate test (set CODESCRIBE_E2E_MIC=1 to enable)"); return; } - vad::init(&model_path).expect("Failed to init VAD"); - - let config = vad::VadConfig { - threshold: 0.5, // Probability threshold for speech detection - min_speech_duration_sec: 0.1, - max_silence_duration_sec: 0.3, - max_utterance_sec: 2.0, - pre_roll_sec: 0.1, // 100ms - }; - let mut segmenter = VADSegmenter::with_config(1000, config); - - // Feed speech (high amplitude triggers VAD) - let speech = vec![0.8; 400]; - assert!(segmenter.feed(&speech).is_none()); - - // Feed silence (VAD should detect no speech) - let silence = vec![0.0; 300]; - let utterance = segmenter - .feed(&silence) - .expect("Expected utterance after silence"); - assert!(!utterance.is_empty()); - } - #[test] - #[ignore] // Requires Silero VAD model (run with: cargo test -- --ignored) - fn test_vad_segmenter_flush_on_max_duration() { - // Initialize VAD - skip if model unavailable let model_path = vad::default_model_path(); if !model_path.exists() { eprintln!( @@ -844,21 +194,61 @@ mod tests { ); return; } - vad::init(&model_path).expect("Failed to init VAD"); - - let config = vad::VadConfig { - threshold: 0.5, // Probability threshold for speech detection - min_speech_duration_sec: 0.1, - max_silence_duration_sec: 1.0, - max_utterance_sec: 0.5, - pre_roll_sec: 0.1, // 100ms - }; - let mut segmenter = VADSegmenter::with_config(1000, config); - let speech = vec![0.8; 600]; - let utterance = segmenter - .feed(&speech) - .expect("Expected utterance after max duration"); - assert!(!utterance.is_empty()); + + let record_sec = env_f32("CODESCRIBE_E2E_MIC_SEC", 6.0).max(2.0); + println!("Speak now for ~{:.1}s...", record_sec); + + let mut recorder = Recorder::new().expect("Failed to create recorder"); + let rt = tokio::runtime::Runtime::new().expect("Failed to create tokio runtime"); + let wav_path = rt + .block_on(async { + recorder.start().await.expect("Failed to start recorder"); + tokio::time::sleep(Duration::from_secs_f32(record_sec)).await; + recorder.stop().await.expect("Failed to stop recorder") + }) + .expect("No WAV produced"); + + let (samples, sample_rate) = + load_audio_file(&wav_path).expect("Failed to load recorded audio"); + + let mut resampler = vad::Resampler::new(sample_rate); + let samples_16k = resampler.resample(&samples); + let chunk_sec = 4.0f32; + let chunk_limit = (vad::VAD_SAMPLE_RATE as f32 * chunk_sec) as usize; + + let cases = [ + ("lt", chunk_limit / 2), + ("eq", chunk_limit), + ("gt", chunk_limit * 2), + ]; + + for (label, block_len) in cases { + let mut session = SpeechSession::new_stream(vad::VAD_SAMPLE_RATE, chunk_sec, 0.0); + let mut chunk_events = 0usize; + let mut idx = 0usize; + while idx < samples_16k.len() { + let end = (idx + block_len).min(samples_16k.len()); + let slice = &samples_16k[idx..end]; + for event in session.feed(slice, vad::VAD_SAMPLE_RATE) { + if matches!(event, SpeechEvent::Chunk(_)) { + chunk_events += 1; + } + } + idx = end; + } + if let Some(SpeechEvent::Chunk(_)) = session.flush() { + chunk_events += 1; + } + + assert!( + chunk_events > 0, + "Expected at least one chunk for case {} (block_len={})", + label, + block_len + ); + } + + let _ = fs::remove_file(&wav_path); } #[test] @@ -904,7 +294,7 @@ mod tests { let raw = transcribe_streaming_samples(&samples, sample_rate, language.as_deref(), None) .expect("Raw streaming transcription failed"); - let mut postprocessor = StreamPostProcessor::new(); + let mut postprocessor = crate::pipeline::stream_postprocess::StreamPostProcessor::new(); let post = transcribe_streaming_samples( &samples, sample_rate, @@ -1090,6 +480,244 @@ mod tests { dist as f32 / denom } + #[test] + fn test_vad_index_sync_no_drift() { + let input_sr = 48000u32; + let callback_size = 1024usize; + let num_callbacks = 100usize; + + let mut session = SpeechSession::new_stream(input_sr, 15.0, 0.0); + + if session.gate_mode() != crate::audio::chunker::VadGateMode::Supervisor { + eprintln!("Skipping: gate mode is not Supervisor"); + return; + } + + let freq = 440.0f32; + let mut phase = 0.0f32; + let phase_inc = 2.0 * std::f32::consts::PI * freq / input_sr as f32; + + for _ in 0..num_callbacks { + let mut buf = Vec::with_capacity(callback_size); + for _ in 0..callback_size { + buf.push(phase.sin() * 0.5); + phase += phase_inc; + } + let _ = session.feed(&buf, input_sr); + } + + let total_raw = num_callbacks * callback_size; + assert_eq!( + session.raw_cursor(), + total_raw, + "raw_cursor should equal total input samples" + ); + + if let Some(vad_sample) = session.vad_current_sample() { + let mapped = session.vad_to_raw_index_pub(vad_sample); + let raw_cur = session.raw_cursor(); + let drift = mapped.abs_diff(raw_cur); + let vad_chunk = 512usize; + let vad_sr = 16000.0f32; + let tolerance = ((vad_chunk as f32 * input_sr as f32) / vad_sr) as usize; + assert!( + drift <= tolerance, + "VAD index drift too large: mapped={} raw_cursor={} drift={} tolerance={}", + mapped, + raw_cur, + drift, + tolerance + ); + } + + let vad_chunk_size = 512usize; + assert!( + session.vad_resample_buf_len() < vad_chunk_size, + "Residual buffer should be < CHUNK_SIZE, got {}", + session.vad_resample_buf_len() + ); + } + + /// Run VAD v5 on real WAV files and report segmentation quality. + #[test] + fn test_vad_supervisor_segments_real_audio() { + let corpus_dir = + std::path::PathBuf::from(shellexpand::tilde("~/.codescribe/transcriptions").as_ref()); + if !corpus_dir.exists() { + eprintln!("Skipping: no transcriptions dir"); + return; + } + let model_path = vad::default_model_path(); + if !model_path.exists() { + eprintln!("Skipping: no Silero model"); + return; + } + + let edge_cases = [ + "192322_nie-zmienia-to_raw.wav", + "133135_no-dobra-teraz_raw.wav", + "182340_klaudiusz-zacznijmy-od_raw.wav", + "001615_dziekuje---dziekuje_raw.wav", + "184818_dzien-dobry-chcialem_raw.wav", + ]; + + let mut wavs: Vec = Vec::new(); + if let Ok(dirs) = fs::read_dir(&corpus_dir) { + for dir_entry in dirs.flatten() { + if !dir_entry.path().is_dir() { + continue; + } + for case in &edge_cases { + let candidate = dir_entry.path().join(case); + if candidate.exists() { + wavs.push(candidate); + } + } + } + } + if wavs.is_empty() { + let mut dirs: Vec<_> = fs::read_dir(&corpus_dir) + .unwrap() + .flatten() + .filter(|e| e.path().is_dir()) + .collect(); + dirs.sort_by_key(|e| e.file_name()); + dirs.reverse(); + for dir in dirs.iter().take(2) { + if let Ok(entries) = fs::read_dir(dir.path()) { + for entry in entries.flatten() { + let p = entry.path(); + if p.extension().and_then(|s| s.to_str()) == Some("wav") { + wavs.push(p); + if wavs.len() >= 5 { + break; + } + } + } + } + } + } + + println!("\n╭─── VAD v5 Segmentation Test ───────────────────────╮"); + let mut all_pass = true; + + for wav_path in &wavs { + let fname = wav_path + .file_name() + .unwrap_or_default() + .to_string_lossy() + .to_string(); + let (samples, sample_rate) = match load_audio_file(wav_path) { + Ok(v) => v, + Err(e) => { + println!("│ SKIP {} — {}", fname, e); + continue; + } + }; + let audio_sec = samples.len() as f32 / sample_rate as f32; + + let vad_config = vad::VadConfig { + threshold: 0.50, + min_speech_duration_sec: 0.05, + max_silence_duration_sec: 0.20, + max_utterance_sec: 300.0, + pre_roll_sec: 0.064, + }; + let mut silero = vad::SileroVad::new(&model_path, vad_config).expect("load Silero"); + let mut resampler = vad::Resampler::new(sample_rate); + let samples_16k = resampler.resample(&samples); + + let mut above = 0usize; + let mut total = 0usize; + for chunk in samples_16k.chunks(vad::CHUNK_SIZE) { + if chunk.len() < vad::CHUNK_SIZE { + break; + } + total += 1; + if silero.predict(chunk).unwrap_or(0.0) >= 0.5 { + above += 1; + } + } + + let callback_size = 1024usize; + let mut session = SpeechSession::new_utterance(sample_rate); + let mut events = Vec::new(); + let mut offset = 0usize; + while offset < samples.len() { + let end = (offset + callback_size).min(samples.len()); + for event in session.feed(&samples[offset..end], sample_rate) { + events.push(event); + } + offset = end; + } + if let Some(event) = session.flush() { + events.push(event); + } + + let n_segments = events.len(); + let speech_samples: usize = events + .iter() + .map(|e| match e { + SpeechEvent::Utterance(s) | SpeechEvent::Chunk(s) => s.len(), + }) + .sum(); + let speech_sec = speech_samples as f32 / sample_rate as f32; + let silence_cut = audio_sec - speech_sec; + let cut_pct = if audio_sec > 0.0 { + silence_cut / audio_sec * 100.0 + } else { + 0.0 + }; + + let raw_txt = wav_path.to_string_lossy().replace("_raw.wav", "_raw.txt"); + let old_len = fs::read_to_string(&raw_txt).map(|s| s.len()).unwrap_or(0); + + println!("│"); + println!("│ 📁 {}", fname); + println!( + "│ Audio: {:.1}s | VAD speech: {:.0}% ({}/{} frames)", + audio_sec, + if total > 0 { + above as f32 / total as f32 * 100.0 + } else { + 0.0 + }, + above, + total, + ); + println!( + "│ Segments: {} | Speech: {:.1}s | Silence cut: {:.1}s ({:.0}%)", + n_segments, speech_sec, silence_cut, cut_pct, + ); + println!("│ Old transcript: {} chars", old_len,); + + let old_text = fs::read_to_string(&raw_txt).unwrap_or_default(); + let halluc_count = old_text.matches("Thank you").count() + + old_text.matches("Dziękuję.").count() + + old_text.matches(".com/").count(); + if halluc_count > 2 { + println!( + "│ ⚠ Old transcript had {} hallucination markers (Thank you/Dziękuję./.com/)", + halluc_count, + ); + println!( + "│ ✅ VAD v5 would cut {:.1}s silence → these tails eliminated", + silence_cut, + ); + } + + if above == 0 && audio_sec > 1.0 { + println!("│ ❌ VAD detected NO speech — possible model issue"); + all_pass = false; + } + } + + println!("│"); + println!("╰────────────────────────────────────────────────────╯\n"); + + assert!(all_pass, "Some files had zero speech detection"); + } + fn levenshtein(a: &[T], b: &[T]) -> usize { let mut prev: Vec = (0..=b.len()).collect(); let mut cur = vec![0usize; b.len() + 1]; diff --git a/core/build.rs b/core/build.rs index 21822622..c311df3a 100644 --- a/core/build.rs +++ b/core/build.rs @@ -3,9 +3,9 @@ //! Exports embedded model data and configuration. //! Generates embedded_model_data.rs and embedded_tts_data.rs in OUT_DIR for release builds. //! -//! Release builds REQUIRE the Whisper model by default. -//! Set CODESCRIBE_NO_EMBED=1 to build without embedding (for dev/CI). -//! TTS model embedding is optional via CODESCRIBE_EMBED_TTS=1. +//! Release builds EMBED Whisper model by default for zero-dependency distribution. +//! Opt-out with CODESCRIBE_NO_EMBED=1 to skip embedding. +//! Additional models (TTS, E5) require opt-in via CODESCRIBE_EMBED_TTS / CODESCRIBE_EMBED_E5. //! //! Created by M&K (c)2026 VetCoders @@ -15,9 +15,17 @@ use std::path::{Path, PathBuf}; /// Default Whisper model to embed const DEFAULT_MODEL_NAME: &str = "whisper-large-v3-turbo-mlx-q8"; +const DEFAULT_WHISPER_REPO: &str = "LibraxisAI/whisper-large-v3-turbo-mlx-q8"; /// Default TTS model to embed const DEFAULT_TTS_MODEL_NAME: &str = "csm-1b"; +const DEFAULT_TTS_REPO: &str = "sesame/csm-1b"; +const DEFAULT_MIMI_REPO: &str = "kyutai/mimi"; + +/// Default E5 embedder model to embed (base = ~1.1GB, good balance) +/// Override with CODESCRIBE_EMBEDDER_REPO for e5-large (~2.3GB) or e5-small (~470MB) +const DEFAULT_E5_MODEL_NAME: &str = "e5-base"; +const DEFAULT_E5_REPO: &str = "intfloat/multilingual-e5-base"; fn main() { println!("cargo:rerun-if-changed=Cargo.toml"); @@ -25,6 +33,8 @@ fn main() { println!("cargo:rerun-if-env-changed=CODESCRIBE_NO_EMBED"); println!("cargo:rerun-if-env-changed=CODESCRIBE_EMBED_TTS"); println!("cargo:rerun-if-env-changed=CODESCRIBE_TTS_PATH"); + println!("cargo:rerun-if-env-changed=CODESCRIBE_EMBED_E5"); + println!("cargo:rerun-if-env-changed=CODESCRIBE_EMBEDDER_REPO"); let profile = env::var("PROFILE").unwrap_or_else(|_| "debug".to_string()); let is_release = profile == "release"; @@ -41,23 +51,43 @@ fn main() { let _ = fs::write(&repo_path_file, &manifest_dir); } + let out_dir = env::var("OUT_DIR").unwrap(); + let dest_path = Path::new(&out_dir).join("embedded_model_data.rs"); let embed_model = env::var("CODESCRIBE_EMBED_MODEL") .ok() - .filter(|value| !value.trim().is_empty()) + .map(|v| v.trim().to_string()) + .filter(|value| !value.is_empty()) .unwrap_or_else(|| DEFAULT_MODEL_NAME.to_string()); - let model_path = resolve_embed_model_path(&manifest_dir, &embed_model); - println!("cargo:rerun-if-changed={}", model_path.display()); - let out_dir = env::var("OUT_DIR").unwrap(); - let dest_path = Path::new(&out_dir).join("embedded_model_data.rs"); - let model_exists = model_path.join("tokenizer.json").exists(); + let model_path = + resolve_whisper_embed_model_path(&manifest_dir, &embed_model, DEFAULT_WHISPER_REPO); + let weights_path = if model_path.join("weights.safetensors").exists() { + model_path.join("weights.safetensors") + } else { + model_path.join("model.safetensors") + }; + let model_exists = model_path.join("config.json").exists() + && model_path.join("tokenizer.json").exists() + && model_path.join("mel_filters.npz").exists() + && weights_path.exists(); + if model_exists { + println!("cargo:rerun-if-changed={}", model_path.display()); + } // TTS model embedding (optional, via CODESCRIBE_EMBED_TTS=1) - let embed_tts = env::var("CODESCRIBE_EMBED_TTS").is_ok(); - let tts_model_path = resolve_embed_model_path(&manifest_dir, DEFAULT_TTS_MODEL_NAME); + let embed_tts = env::var("CODESCRIBE_EMBED_TTS").is_ok() && !no_embed; + let tts_model_path = + resolve_tts_embed_model_path(&manifest_dir, DEFAULT_TTS_MODEL_NAME, DEFAULT_TTS_REPO); let tts_dest_path = Path::new(&out_dir).join("embedded_tts_data.rs"); let tts_model_exists = tts_model_path.join("config.json").exists(); + let mimi_path_from_cache = + find_hf_snapshot(DEFAULT_MIMI_REPO).map(|p| p.join("model.safetensors")); + let mimi_weights_path = if tts_model_path.join("mimi.safetensors").exists() { + tts_model_path.join("mimi.safetensors") + } else { + mimi_path_from_cache.unwrap_or_else(|| tts_model_path.join("mimi.safetensors")) + }; - if embed_tts && tts_model_exists { + if embed_tts && tts_model_exists && mimi_weights_path.exists() { println!( "cargo:warning=Embedding TTS model from: {}", tts_model_path.display() @@ -74,19 +104,98 @@ fn main() { tts_model_path.join("config.json").display(), tts_model_path.join("tokenizer.json").display(), tts_model_path.join("model.safetensors").display(), - tts_model_path.join("mimi.safetensors").display(), + mimi_weights_path.display(), ); fs::write(&tts_dest_path, tts_content).expect("Failed to write embedded_tts_data.rs"); println!("cargo:rustc-cfg=embed_tts"); - } else if embed_tts && !tts_model_exists { + } else if embed_tts && (!tts_model_exists || !mimi_weights_path.exists()) { println!( "cargo:warning=CODESCRIBE_EMBED_TTS set but TTS model not found at: {}", tts_model_path.display() ); - println!("cargo:warning=Download with: ./scripts/download-csm.sh"); + println!( + "cargo:warning=Download with: hf download {}", + DEFAULT_TTS_REPO + ); + println!( + "cargo:warning=Download Mimi with: hf download {}", + DEFAULT_MIMI_REPO + ); + } + + // E5 embedder embedding (opt-in only, runtime loading from HF cache is default) + // Enable with CODESCRIBE_EMBED_E5=1 (warning: large models may cause dyld issues) + let embed_e5 = env_flag("CODESCRIBE_EMBED_E5", false) && !no_embed; + let e5_repo = env::var("CODESCRIBE_EMBEDDER_REPO") + .ok() + .map(|v| v.trim().to_string()) + .filter(|v| !v.is_empty()) + .unwrap_or_else(|| DEFAULT_E5_REPO.to_string()); + let e5_model_path = + resolve_e5_embed_model_path(&manifest_dir, DEFAULT_E5_MODEL_NAME, &e5_repo); + let e5_dest_path = Path::new(&out_dir).join("embedded_e5_data.rs"); + let e5_model_exists = e5_model_path.join("config.json").exists() + && e5_model_path.join("tokenizer.json").exists() + && e5_model_path.join("model.safetensors").exists(); + + if embed_e5 && e5_model_exists { + println!( + "cargo:warning=Embedding E5 model from: {}", + e5_model_path.display() + ); + let e5_content = format!( + r#" + pub static CONFIG: &[u8] = include_bytes!(r"{}"); + pub static TOKENIZER: &[u8] = include_bytes!(r"{}"); + pub static WEIGHTS: &[u8] = include_bytes!(r"{}"); + "#, + e5_model_path.join("config.json").display(), + e5_model_path.join("tokenizer.json").display(), + e5_model_path.join("model.safetensors").display(), + ); + fs::write(&e5_dest_path, e5_content).expect("Failed to write embedded_e5_data.rs"); + println!("cargo:rustc-cfg=embed_e5"); + } else if embed_e5 && !e5_model_exists { + println!( + "cargo:warning=E5 model not found at: {}", + e5_model_path.display() + ); + println!("cargo:warning=Download with: hf download {}", e5_repo); + } else if !embed_e5 { + println!("cargo:warning=E5 embedding disabled (set CODESCRIBE_EMBED_E5=1 to embed)"); } - if is_release && model_exists { + // Silero VAD embedding (small, default in release when available) + let silero_path = codescribe_dir.join("models").join("silero_vad.onnx"); + let silero_dest_path = Path::new(&out_dir).join("embedded_vad_data.rs"); + let silero_exists = silero_path.exists(); + if is_release && silero_exists && !no_embed { + println!( + "cargo:warning=Embedding Silero VAD model from: {}", + silero_path.display() + ); + let silero_content = format!( + r#" + pub static MODEL: &[u8] = include_bytes!(r"{}"); + "#, + silero_path.display(), + ); + fs::write(&silero_dest_path, silero_content) + .expect("Failed to write embedded_vad_data.rs"); + println!("cargo:rustc-cfg=embed_vad"); + } else if is_release && !silero_exists { + println!( + "cargo:warning=Silero VAD model not found at: {}", + silero_path.display() + ); + println!("cargo:warning=Download with: scripts/download-silero.sh"); + } + + // Release builds embed Whisper by default (zero-dependency distribution) + // Skip with CODESCRIBE_NO_EMBED=1 or if model not found + let should_embed_whisper = is_release && model_exists && !no_embed; + + if should_embed_whisper { // Release + model found → embed it println!( "cargo:warning=Embedding model from: {}", @@ -102,7 +211,7 @@ fn main() { model_path.join("config.json").display(), model_path.join("tokenizer.json").display(), model_path.join("mel_filters.npz").display(), - model_path.join("weights.safetensors").display() + weights_path.display() ); fs::write(&dest_path, content).expect("Failed to write embedded_model_data.rs"); println!("cargo:rustc-cfg=embed_model"); @@ -110,28 +219,24 @@ fn main() { "cargo:rustc-env=CODESCRIBE_MODEL_DIR={}", model_path.display() ); - } else if is_release && !model_exists && !no_embed { - // Release + no model + no opt-out → HARD FAIL - eprintln!(); - eprintln!("═══════════════════════════════════════════════════════════════"); - eprintln!(" ERROR: Whisper model not found for embedding!"); - eprintln!("═══════════════════════════════════════════════════════════════"); - eprintln!(" Expected: {}", model_path.display()); - eprintln!(); - eprintln!(" Solutions:"); - eprintln!(" 1. Download model: make download-model"); - eprintln!(" 2. Skip embedding: CODESCRIBE_NO_EMBED=1 cargo build --release"); - eprintln!(); - eprintln!(" Note: Without embedded model, set CODESCRIBE_MODEL_PATH at runtime."); - eprintln!("═══════════════════════════════════════════════════════════════"); - eprintln!(); - std::process::exit(1); + } else if is_release && no_embed { + // Explicit opt-out + println!("cargo:warning=CODESCRIBE_NO_EMBED set - skipping Whisper embedding"); + println!( + "cargo:warning=Binary will require CODESCRIBE_MODEL_PATH or HF cache at runtime" + ); + println!("cargo:rustc-env=CODESCRIBE_MODEL_DIR="); + } else if is_release && !model_exists { + // Release but model not found + println!("cargo:warning=Whisper model not found - cannot embed"); + println!( + "cargo:warning=Download with: hf download {}", + DEFAULT_WHISPER_REPO + ); + println!("cargo:warning=Or set CODESCRIBE_MODEL_PATH at runtime"); + println!("cargo:rustc-env=CODESCRIBE_MODEL_DIR="); } else { - // Debug build OR explicit no-embed → skip embedding - if is_release && no_embed { - println!("cargo:warning=CODESCRIBE_NO_EMBED set - skipping model embedding"); - println!("cargo:warning=Binary will require CODESCRIBE_MODEL_PATH at runtime"); - } + // Debug build → skip embedding (use runtime loading) println!("cargo:rustc-env=CODESCRIBE_MODEL_DIR="); } } @@ -149,3 +254,144 @@ fn resolve_embed_model_path(manifest_dir: &str, embed_model: &str) -> PathBuf { Path::new(manifest_dir).join("models").join(embed_model) } + +fn resolve_whisper_embed_model_path( + manifest_dir: &str, + embed_model: &str, + default_repo: &str, +) -> PathBuf { + if embed_model.contains('/') { + if let Some(snapshot) = find_hf_snapshot(embed_model) { + return snapshot; + } + } else if embed_model == DEFAULT_MODEL_NAME + && let Some(snapshot) = find_hf_snapshot(default_repo) + { + return snapshot; + } + resolve_embed_model_path(manifest_dir, embed_model) +} + +fn resolve_tts_embed_model_path( + manifest_dir: &str, + embed_model: &str, + default_repo: &str, +) -> PathBuf { + if embed_model.contains('/') { + if let Some(snapshot) = find_hf_snapshot(embed_model) { + return snapshot; + } + } else if embed_model == DEFAULT_TTS_MODEL_NAME + && let Some(snapshot) = find_hf_snapshot(default_repo) + { + return snapshot; + } + resolve_embed_model_path(manifest_dir, embed_model) +} + +fn resolve_e5_embed_model_path(manifest_dir: &str, embed_model: &str, repo: &str) -> PathBuf { + if let Some(snapshot) = find_hf_snapshot(repo) { + return snapshot; + } + resolve_embed_model_path(manifest_dir, embed_model) +} + +fn hf_cache_bases() -> Vec { + let mut out = Vec::new(); + if let Ok(path) = env::var("CODESCRIBE_HF_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HUGGINGFACE_HUB_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HF_HUB_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HF_HOME") { + out.push(PathBuf::from(path).join("hub")); + } + if let Some(home) = dirs::home_dir().map(|h| h.join(".cache").join("huggingface").join("hub")) { + out.push(home); + } + if let Some(home) = dirs::home_dir().map(|h| h.join(".codescribe").join("embeddings")) { + out.push(home.clone()); + out.push(home.join("hub")); + } + out.sort(); + out.dedup(); + out +} + +fn find_hf_snapshot(repo: &str) -> Option { + for base in hf_cache_bases() { + if let Some(snapshot) = find_hf_snapshot_in_base(&base, repo) { + return Some(snapshot); + } + } + None +} + +fn find_hf_snapshot_in_base(base: &PathBuf, repo: &str) -> Option { + let repo_dir = base.join(format!("models--{}", repo.replace('/', "--"))); + let snapshots_dir = repo_dir.join("snapshots"); + + let snapshots_dir = if snapshots_dir.exists() { + snapshots_dir + } else { + let target = repo.to_ascii_lowercase(); + let mut matched: Option = None; + if let Ok(entries) = fs::read_dir(base) { + for entry in entries.flatten() { + let name = entry.file_name(); + let name = name.to_string_lossy(); + if !name.starts_with("models--") { + continue; + } + let repo_id = name + .strip_prefix("models--") + .unwrap_or("") + .replace("--", "/"); + if repo_id.to_ascii_lowercase() == target { + matched = Some(entry.path().join("snapshots")); + break; + } + } + } + matched? + }; + + let entries = fs::read_dir(&snapshots_dir).ok()?; + + let mut best: Option<(std::time::SystemTime, PathBuf)> = None; + + for entry in entries.flatten() { + let path = entry.path(); + if !path.is_dir() { + continue; + } + let modified = entry + .metadata() + .and_then(|m| m.modified()) + .unwrap_or(std::time::SystemTime::UNIX_EPOCH); + match &best { + Some((best_time, _)) if *best_time >= modified => {} + _ => best = Some((modified, path)), + } + } + + best.map(|(_, p)| p) +} + +fn env_flag(name: &str, default: bool) -> bool { + match env::var(name) { + Ok(value) => { + let trimmed = value.trim(); + if trimmed.is_empty() { + return default; + } + let v = trimmed.to_ascii_lowercase(); + !(v == "0" || v == "false" || v == "off" || v == "no") + } + Err(_) => default, + } +} diff --git a/core/config/default_env.txt b/core/config/default_env.txt index 472fd0e5..bb287a2b 100644 --- a/core/config/default_env.txt +++ b/core/config/default_env.txt @@ -37,17 +37,21 @@ CODESCRIBE_STREAM_CHUNK_SEC=3.0 CODESCRIBE_STREAM_OVERLAP_RATIO=0.2 CODESCRIBE_BUFFER_DELAY_MS=1800 CODESCRIBE_TYPING_CPS=35 +# Number of words emitted per tick (buffered mode) +CODESCRIBE_EMIT_WORDS_MAX=3 # VAD config (centralized in core/vad/config.rs) -CODESCRIBE_VAD_THRESHOLD=0.5 -CODESCRIBE_VAD_MIN_SPEECH_SEC=0.1 -CODESCRIBE_VAD_MAX_SILENCE_SEC=1.2 -CODESCRIBE_VAD_MAX_UTTERANCE_SEC=60 -CODESCRIBE_VAD_PRE_ROLL_SEC=0.3 +# CODESCRIBE_VAD_THRESHOLD=0.35 # Speech probability 0.0-1.0 (lower=more sensitive) +# CODESCRIBE_VAD_SILENCE_SEC=2.5 # Silence duration before utterance flush +# CODESCRIBE_VAD_MIN_SPEECH_SEC=0.1 # Min speech before triggering +# CODESCRIBE_VAD_MAX_UTTERANCE_SEC=60 # Force flush after this duration +# CODESCRIBE_VAD_PRE_ROLL_SEC=0.5 # Buffer before speech onset CODESCRIBE_STREAM_SIMILARITY=0.90 CODESCRIBE_STREAM_NOVELTY=0.20 +# Embedder model override (HuggingFace cache repo id) +# CODESCRIBE_EMBEDDER_REPO=intfloat/multilingual-e5-small # ➡️ fewer cuts, more natural phrases, stable overlay. -——— +# --------------------------------------------------------------------------- ## ✅ B) "Almost live" (streaming without VAD) @@ -79,7 +83,6 @@ RESTORE_CLIPBOARD=1 RESTORE_CLIPBOARD_DELAY_MS=1000 # === Audio === -AUTO_SILENCE=0 BEEP_ON_START=1 SOUND_NAME=Tink SOUND_VOLUME=0.25 diff --git a/core/conversation/config.rs b/core/conversation/config.rs index 1af37a7a..1a101356 100644 --- a/core/conversation/config.rs +++ b/core/conversation/config.rs @@ -4,6 +4,17 @@ use std::path::PathBuf; +use crate::hf_cache; + +/// HuggingFace repos for Moshi models +const MOSHIKO_REPO: &str = "kyutai/moshiko-candle-q8"; +const MOSHIKA_REPO: &str = "kyutai/moshika-candle-q8"; + +/// Required files for Moshi +const MOSHI_MODEL_FILE: &str = "model.q8.gguf"; +const MOSHI_MIMI_FILE: &str = "tokenizer-e351c8d8-checkpoint125.safetensors"; +const MOSHI_TOKENIZER_FILE: &str = "tokenizer_spm_32k_3.model"; + /// Configuration for the Moshi conversation engine #[derive(Debug, Clone)] pub struct MoshiConfig { @@ -48,8 +59,11 @@ impl Default for MoshiConfig { Self { // Moshiko LM weights (.gguf quantized) model_path: models_dir.join("moshiko-q8").join("model.q8.gguf"), - // Mimi codec (shared with CSM TTS) - mimi_path: models_dir.join("csm-1b").join("mimi.safetensors"), + // Mimi codec for moshi crate (NOT the same as candle-transformers mimi!) + // This is "tokenizer-e351c8d8-checkpoint125.safetensors" from moshiko repo + mimi_path: models_dir + .join("moshiko-q8") + .join("tokenizer-e351c8d8-checkpoint125.safetensors"), // SentencePiece tokenizer tokenizer_path: models_dir .join("moshiko-q8") @@ -70,6 +84,36 @@ impl MoshiConfig { Self::default() } + /// Create config from HuggingFace cache for Moshiko (male voice) + /// + /// Looks for models in ~/.cache/huggingface/hub/ + pub fn moshiko_from_hf_cache() -> Option { + let snapshot = hf_cache::find_snapshot(MOSHIKO_REPO, &[MOSHI_MODEL_FILE, MOSHI_MIMI_FILE])?; + + Some(Self { + model_path: snapshot.join(MOSHI_MODEL_FILE), + mimi_path: snapshot.join(MOSHI_MIMI_FILE), + tokenizer_path: snapshot.join(MOSHI_TOKENIZER_FILE), + voice: "moshiko".to_string(), + ..Self::default() + }) + } + + /// Create config from HuggingFace cache for Moshika (female voice) + /// + /// Looks for models in ~/.cache/huggingface/hub/ + pub fn moshika_from_hf_cache() -> Option { + let snapshot = hf_cache::find_snapshot(MOSHIKA_REPO, &[MOSHI_MODEL_FILE, MOSHI_MIMI_FILE])?; + + Some(Self { + model_path: snapshot.join(MOSHI_MODEL_FILE), + mimi_path: snapshot.join(MOSHI_MIMI_FILE), + tokenizer_path: snapshot.join(MOSHI_TOKENIZER_FILE), + voice: "moshika".to_string(), + ..Self::default() + }) + } + /// Create config for Moshika (female voice) pub fn moshika() -> Self { // All models in ~/.codescribe/models/ (unified path) @@ -81,6 +125,10 @@ impl MoshiConfig { Self { model_path: models_dir.join("moshika-q8").join("model.q8.gguf"), + // Mimi codec (same weights shared between moshiko/moshika) + mimi_path: models_dir + .join("moshika-q8") + .join("tokenizer-e351c8d8-checkpoint125.safetensors"), tokenizer_path: models_dir .join("moshika-q8") .join("tokenizer_spm_32k_3.model"), diff --git a/core/demux/mod.rs b/core/demux/mod.rs new file mode 100644 index 00000000..419d91b0 --- /dev/null +++ b/core/demux/mod.rs @@ -0,0 +1,3 @@ +pub mod parser; + +pub use parser::{DemuxEvent, StreamingTagParser, TagKind}; diff --git a/core/demux/parser.rs b/core/demux/parser.rs new file mode 100644 index 00000000..8ed2cfd0 --- /dev/null +++ b/core/demux/parser.rs @@ -0,0 +1,488 @@ +//! Streaming tag parser for demuxed LLM output. +//! +//! MVP scope: +//! - Tags: ... , ... +//! - Flat (no nesting) +//! - Partial chunks (SSE) supported + +/// Maximum buffer size before forced flush (防DoS - prevents unbounded memory growth) +const MAX_BUFFER_SIZE: usize = 64 * 1024; // 64KB + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TagKind { + Speak, + Tool, + Unknown, +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum DemuxEvent { + Speak(String), + Tool { name: String, args: String }, + Text(String), + Partial(TagKind), +} + +#[derive(Debug, Clone)] +pub struct StreamingTagParser { + buffer: String, + state: ParserState, + speak_min_chars: usize, + speak_max_chars: usize, +} + +#[derive(Debug, Clone)] +enum ParserState { + Text, + TagOpen, + TagContent { + kind: TagKind, + tag: String, + open_tag: String, + tool_name: Option, + last_emit: usize, + content: String, + }, +} + +#[derive(Debug)] +struct TagInfo { + kind: TagKind, + tag: String, + tool_name: Option, +} + +impl StreamingTagParser { + pub fn new() -> Self { + Self { + buffer: String::new(), + state: ParserState::Text, + speak_min_chars: 40, + speak_max_chars: 160, + } + } + + pub fn with_speak_chunking(min_chars: usize, max_chars: usize) -> Self { + Self { + buffer: String::new(), + state: ParserState::Text, + speak_min_chars: min_chars.max(1), + speak_max_chars: max_chars.max(1), + } + } + + /// Feed a new chunk and return parsed events. + pub fn feed(&mut self, chunk: &str) -> Vec { + self.buffer.push_str(chunk); + let mut events = Vec::new(); + + // DoS protection: force flush if buffer exceeds limit + if self.buffer.len() > MAX_BUFFER_SIZE { + tracing::warn!( + "Demux buffer exceeded {}KB, forcing flush", + MAX_BUFFER_SIZE / 1024 + ); + events.extend(self.flush()); + return events; + } + let speak_min = self.speak_min_chars; + let speak_max = self.speak_max_chars; + + loop { + match &mut self.state { + ParserState::Text => { + if let Some(pos) = self.buffer.find('<') { + if pos > 0 { + events.push(DemuxEvent::Text(self.buffer[..pos].to_string())); + } + self.buffer = self.buffer[pos..].to_string(); + self.state = ParserState::TagOpen; + continue; + } + + if !self.buffer.is_empty() { + events.push(DemuxEvent::Text(std::mem::take(&mut self.buffer))); + } + break; + } + ParserState::TagOpen => { + if let Some(end) = self.buffer.find('>') { + let tag_literal = self.buffer[..=end].to_string(); + self.buffer = self.buffer[end + 1..].to_string(); + + if let Some(info) = parse_open_tag(&tag_literal) + && info.kind != TagKind::Unknown + { + self.state = ParserState::TagContent { + kind: info.kind, + tag: info.tag, + open_tag: tag_literal, + tool_name: info.tool_name, + last_emit: 0, + content: String::new(), + }; + continue; + } + + // Unknown or malformed tag: treat as text. + events.push(DemuxEvent::Text(tag_literal)); + self.state = ParserState::Text; + continue; + } + + // Partial tag (no closing '>'). + let kind = partial_kind(&self.buffer); + events.push(DemuxEvent::Partial(kind)); + break; + } + ParserState::TagContent { + kind, + tag, + open_tag, + tool_name, + last_emit, + content, + } => { + let close_tag = format!("", tag); + if let Some(pos) = self.buffer.find(&close_tag) { + content.push_str(&self.buffer[..pos]); + self.buffer = self.buffer[pos + close_tag.len()..].to_string(); + + match kind { + TagKind::Speak => { + emit_speak_chunks( + speak_min, + speak_max, + content, + last_emit, + &mut events, + ); + let remainder = content.get(*last_emit..).unwrap_or(""); + let remainder = remainder.trim_start(); + if !remainder.is_empty() { + events.push(DemuxEvent::Speak(remainder.to_string())); + } + content.clear(); + *last_emit = 0; + } + TagKind::Tool => { + events.push(DemuxEvent::Tool { + name: tool_name.clone().unwrap_or_default(), + args: std::mem::take(content), + }); + } + TagKind::Unknown => { + events.push(DemuxEvent::Text(format!( + "{}{}{}", + open_tag, + std::mem::take(content), + close_tag + ))); + } + } + + self.state = ParserState::Text; + continue; + } + + // Tag not closed yet: buffer content. + content.push_str(&self.buffer); + self.buffer.clear(); + if *kind == TagKind::Speak { + emit_speak_chunks(speak_min, speak_max, content, last_emit, &mut events); + } + events.push(DemuxEvent::Partial(kind.clone())); + break; + } + } + } + + events + } + + /// Flush any buffered content (EOF/timeout). + pub fn flush(&mut self) -> Vec { + let mut events = Vec::new(); + let speak_min = self.speak_min_chars; + let speak_max = self.speak_max_chars; + + match &mut self.state { + ParserState::Text => { + if !self.buffer.is_empty() { + events.push(DemuxEvent::Text(std::mem::take(&mut self.buffer))); + } + } + ParserState::TagOpen => { + if !self.buffer.is_empty() { + events.push(DemuxEvent::Text(std::mem::take(&mut self.buffer))); + } + self.state = ParserState::Text; + } + ParserState::TagContent { + kind, + open_tag, + content, + last_emit, + .. + } => { + let trailing = std::mem::take(&mut self.buffer); + content.push_str(&trailing); + + match kind { + TagKind::Speak => { + emit_speak_chunks(speak_min, speak_max, content, last_emit, &mut events); + let remainder = content.get(*last_emit..).unwrap_or(""); + let remainder = remainder.trim_start(); + if !remainder.is_empty() { + events.push(DemuxEvent::Speak(remainder.to_string())); + } + content.clear(); + *last_emit = 0; + } + TagKind::Tool => { + // Incomplete tool tag: do not execute. Preserve as text. + let args = std::mem::take(content); + events.push(DemuxEvent::Text(format!("{}{}", open_tag, args))); + } + TagKind::Unknown => { + events.push(DemuxEvent::Text(format!( + "{}{}", + open_tag, + std::mem::take(content) + ))); + } + } + self.state = ParserState::Text; + } + } + + events + } +} + +impl Default for StreamingTagParser { + fn default() -> Self { + Self::new() + } +} + +fn emit_speak_chunks( + min_chars: usize, + max_chars: usize, + content: &str, + last_emit: &mut usize, + events: &mut Vec, +) { + if min_chars == 0 || max_chars == 0 { + return; + } + loop { + let boundary = find_speak_boundary(content, *last_emit, min_chars, max_chars); + let Some(boundary) = boundary else { + break; + }; + + if boundary <= *last_emit { + break; + } + + let chunk = content.get(*last_emit..boundary).unwrap_or("").trim_start(); + if !chunk.is_empty() { + events.push(DemuxEvent::Speak(chunk.to_string())); + } + *last_emit = boundary; + } +} + +fn parse_open_tag(tag: &str) -> Option { + if !tag.starts_with('<') || !tag.ends_with('>') { + return None; + } + let inner = &tag[1..tag.len() - 1]; + let inner = inner.trim(); + if inner.starts_with('/') { + return None; + } + + let mut parts = inner.splitn(2, char::is_whitespace); + let name = parts.next()?.trim(); + if name.is_empty() { + return None; + } + let attrs = parts.next().unwrap_or("").trim(); + + let kind = match name { + "speak" => TagKind::Speak, + "tool" => TagKind::Tool, + _ => TagKind::Unknown, + }; + + let tool_name = if kind == TagKind::Tool { + find_attr_value(attrs, "name") + } else { + None + }; + + Some(TagInfo { + kind, + tag: name.to_string(), + tool_name, + }) +} + +fn partial_kind(buf: &str) -> TagKind { + let s = buf.trim_start_matches('<').trim_start_matches('/'); + if "speak".starts_with(s) || s.starts_with("speak") { + return TagKind::Speak; + } + if "tool".starts_with(s) || s.starts_with("tool") { + return TagKind::Tool; + } + TagKind::Unknown +} + +fn find_attr_value(attrs: &str, key: &str) -> Option { + let bytes = attrs.as_bytes(); + let mut i = 0; + while i < bytes.len() { + while i < bytes.len() && bytes[i].is_ascii_whitespace() { + i += 1; + } + let start = i; + while i < bytes.len() && !bytes[i].is_ascii_whitespace() && bytes[i] != b'=' { + i += 1; + } + if start == i { + break; + } + let attr_key = &attrs[start..i]; + while i < bytes.len() && bytes[i].is_ascii_whitespace() { + i += 1; + } + if i >= bytes.len() || bytes[i] != b'=' { + continue; + } + i += 1; + while i < bytes.len() && bytes[i].is_ascii_whitespace() { + i += 1; + } + if i >= bytes.len() { + break; + } + let quote = bytes[i]; + if quote != b'"' && quote != b'\'' { + continue; + } + i += 1; + let val_start = i; + while i < bytes.len() && bytes[i] != quote { + i += 1; + } + if i >= bytes.len() { + break; + } + let val = &attrs[val_start..i]; + i += 1; + if attr_key == key { + return Some(val.to_string()); + } + } + None +} + +fn find_speak_boundary( + content: &str, + start: usize, + min_chars: usize, + max_chars: usize, +) -> Option { + if start >= content.len() { + return None; + } + let mut count = 0usize; + let mut last_punct: Option = None; + let mut last_space: Option = None; + let mut idx_at_max: Option = None; + + for (i, ch) in content[start..].char_indices() { + count += 1; + let byte_idx = start + i; + if matches!(ch, '.' | '!' | '?' | ';' | ':' | '\n') && count >= min_chars { + last_punct = Some(byte_idx + ch.len_utf8()); + } + if ch.is_whitespace() { + last_space = Some(byte_idx + ch.len_utf8()); + } + if count >= max_chars { + idx_at_max = Some(byte_idx + ch.len_utf8()); + break; + } + } + + if let Some(punct) = last_punct { + return Some(punct); + } + if let Some(max_idx) = idx_at_max { + if let Some(space) = last_space.filter(|s| *s > start) { + return Some(space); + } + return Some(max_idx); + } + None +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_speak_single_chunk() { + let mut p = StreamingTagParser::new(); + let ev = p.feed("hi"); + assert_eq!(ev, vec![DemuxEvent::Speak("hi".to_string())]); + } + + #[test] + fn parses_tool_with_name() { + let mut p = StreamingTagParser::new(); + let ev = p.feed(r#"{"q":1}"#); + assert_eq!( + ev, + vec![DemuxEvent::Tool { + name: "weather".to_string(), + args: r#"{"q":1}"#.to_string() + }] + ); + } + + #[test] + fn parses_text_and_tags() { + let mut p = StreamingTagParser::new(); + let ev = p.feed("hello hi world"); + assert_eq!( + ev, + vec![ + DemuxEvent::Text("hello ".to_string()), + DemuxEvent::Speak("hi".to_string()), + DemuxEvent::Text(" world".to_string()) + ] + ); + } + + #[test] + fn handles_partial_tag() { + let mut p = StreamingTagParser::new(); + let ev1 = p.feed("hi"); + assert_eq!(ev2, vec![DemuxEvent::Speak("hi".to_string())]); + } + + #[test] + fn flushes_open_speak() { + let mut p = StreamingTagParser::new(); + let _ = p.feed("hello"); + let ev = p.flush(); + assert_eq!(ev, vec![DemuxEvent::Speak("hello".to_string())]); + } +} diff --git a/core/embedder/embedded.rs b/core/embedder/embedded.rs new file mode 100644 index 00000000..c92fb3f1 --- /dev/null +++ b/core/embedder/embedded.rs @@ -0,0 +1,78 @@ +//! Embedded E5 embedder model - direct include via generated code +//! +//! Release builds: Model files included directly in binary (~1GB) +//! Debug builds: Empty slices, use CODESCRIBE_EMBEDDER_PATH +//! +//! Created by M&K (c)2026 VetCoders + +#[cfg(embed_e5)] +mod data { + include!(concat!(env!("OUT_DIR"), "/embedded_e5_data.rs")); +} + +#[cfg(not(embed_e5))] +mod data { + pub static CONFIG: &[u8] = &[]; + pub static TOKENIZER: &[u8] = &[]; + pub static WEIGHTS: &[u8] = &[]; +} + +/// Check if embedded E5 model is available +/// +/// Note: We only check weights_size, not cfg!(embed_e5). +/// The cfg!() macro can return false in workspace builds even when +/// the data was correctly embedded via #[cfg(embed_e5)]. +/// If weights exist (len > 0), the model is available. +pub fn is_embedded_available() -> bool { + let weights_size = data::WEIGHTS.len(); + tracing::debug!("[E5] Embedded check: weights_size={}", weights_size); + weights_size > 0 +} + +/// Get embedded model data if available +pub fn get_embedded_data() -> Option { + if !is_embedded_available() { + return None; + } + Some(EmbeddedE5 { + config: data::CONFIG, + tokenizer: data::TOKENIZER, + weights: data::WEIGHTS, + }) +} + +/// Embedded E5 model data - static byte slices from binary +pub struct EmbeddedE5 { + /// E5 model configuration (config.json) + pub config: &'static [u8], + /// Tokenizer (tokenizer.json) + pub tokenizer: &'static [u8], + /// Model weights (model.safetensors) + pub weights: &'static [u8], +} + +impl EmbeddedE5 { + /// Total size in bytes + pub fn total_size(&self) -> usize { + self.config.len() + self.tokenizer.len() + self.weights.len() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_embedded_availability() { + let available = is_embedded_available(); + println!("Embedded E5 available: {}", available); + + if available { + let model = get_embedded_data().unwrap(); + println!( + "E5 model size: {:.1} MB", + model.total_size() as f64 / 1_000_000.0 + ); + } + } +} diff --git a/core/embedder/engine.rs b/core/embedder/engine.rs index 9da82943..56357c51 100644 --- a/core/embedder/engine.rs +++ b/core/embedder/engine.rs @@ -1,65 +1,75 @@ -//! Embedder Engine - fastembed wrapper for text embeddings. +//! Embedder Engine - offline E5 embeddings via Candle BERT. //! -//! Provides text embeddings using fastembed with E5 models. -//! Supports query and passage prefixes for optimal retrieval performance. +//! Provides text embeddings using a local/embedded multilingual-e5-large model. +//! No runtime downloads; model must be embedded or present on disk. //! //! Created by M&K (c)2026 VetCoders -use anyhow::{Context, Result}; -use fastembed::{EmbeddingModel, InitOptions, TextEmbedding}; -use tracing::info; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, anyhow}; +use candle_core::{DType, Device, Tensor}; +use candle_nn::VarBuilder; +use candle_transformers::models::bert::{BertModel, Config as BertConfig}; +use tokenizers::{PaddingParams, PaddingStrategy, Tokenizer, TruncationParams}; +use tracing::{debug, info}; + +use super::embedded; +use crate::{hf_cache, safe_path}; + +const DEFAULT_MAX_LENGTH: usize = 512; +const DEFAULT_E5_REPO: &str = "intfloat/multilingual-e5-large"; +const ENV_EMBEDDER_REPO: &str = "CODESCRIBE_EMBEDDER_REPO"; /// Configuration for the embedder #[derive(Debug, Clone)] pub struct EmbedderConfig { - /// Model to use (default: multilingual-e5-large) - pub model: EmbeddingModel, - /// Whether to show download progress - pub show_download_progress: bool, - /// Cache directory for models - pub cache_dir: Option, + /// Optional explicit model path + pub model_path: Option, + /// Override max token length (default from model config) + pub max_length: Option, + /// Prefer embedded model if available + pub use_embedded: bool, } impl Default for EmbedderConfig { fn default() -> Self { Self { - model: EmbeddingModel::MultilingualE5Large, - show_download_progress: true, - cache_dir: None, + model_path: None, + max_length: None, + use_embedded: true, } } } impl EmbedderConfig { - /// Create config for a specific model - pub fn with_model(model: EmbeddingModel) -> Self { + /// Create config with explicit model path + pub fn with_model_path(path: PathBuf) -> Self { Self { - model, + model_path: Some(path), ..Default::default() } } - /// Use smaller model for faster inference - pub fn small() -> Self { - Self { - model: EmbeddingModel::MultilingualE5Small, - ..Default::default() - } + /// Override max token length + pub fn with_max_length(mut self, max_length: usize) -> Self { + self.max_length = Some(max_length); + self } - /// Use base model for balanced performance - pub fn base() -> Self { - Self { - model: EmbeddingModel::MultilingualE5Base, - ..Default::default() - } + /// Disable embedded model usage + pub fn disable_embedded(mut self) -> Self { + self.use_embedded = false; + self } } -/// Text embedding engine using fastembed +/// Text embedding engine using Candle BERT (E5) pub struct EmbedderEngine { - model: TextEmbedding, - config: EmbedderConfig, + model: BertModel, + tokenizer: Tokenizer, + config: BertConfig, + device: Device, } impl EmbedderEngine { @@ -69,78 +79,161 @@ impl EmbedderEngine { } /// Create with custom configuration - pub fn with_config(config: EmbedderConfig) -> Result { - info!("Initializing embedder with model: {:?}", config.model); - - let mut init_options = InitOptions::new(config.model.clone()); - - if config.show_download_progress { - init_options = init_options.with_show_download_progress(true); + pub fn with_config(mut config: EmbedderConfig) -> Result { + let device = Device::new_metal(0).unwrap_or(Device::Cpu); + debug!("Embedder using device: {:?}", device); + + // Explicit overrides disable embedded usage. + let repo_override = std::env::var(ENV_EMBEDDER_REPO) + .ok() + .map(|v| v.trim().to_string()) + .filter(|v| !v.is_empty()); + let path_override = std::env::var("CODESCRIBE_EMBEDDER_PATH") + .ok() + .map(|v| v.trim().to_string()) + .filter(|v| !v.is_empty()); + if config.model_path.is_some() || repo_override.is_some() || path_override.is_some() { + config.use_embedded = false; } - if let Some(ref cache_dir) = config.cache_dir { - init_options = init_options.with_cache_dir(cache_dir.into()); + if config.use_embedded + && let Some(embedded) = embedded::get_embedded_data() + { + return Self::from_embedded(&embedded, device, config.max_length); } - let model = TextEmbedding::try_new(init_options) - .context("Failed to initialize text embedding model")?; + let model_path = resolve_model_path(config.model_path.as_ref(), repo_override.as_deref())?; + Self::from_path(&model_path, device, config.max_length) + } + + fn from_embedded( + embedded: &embedded::EmbeddedE5, + device: Device, + max_length: Option, + ) -> Result { + let config: BertConfig = serde_json::from_slice(embedded.config) + .context("Failed to parse embedded E5 config")?; + let tokenizer = Tokenizer::from_bytes(embedded.tokenizer) + .map_err(|e| anyhow!("Failed to load embedded tokenizer: {}", e))?; + + let tokenizer = prepare_tokenizer(tokenizer, &config, max_length)?; + + let dtype = device.bf16_default_to_f32(); + let tensors = candle_core::safetensors::load_buffer(embedded.weights, &Device::Cpu) + .context("Failed to deserialize embedded E5 weights")?; + let tensors = move_tensors_to_device(tensors, &device, dtype)?; + let vb = VarBuilder::from_tensors(tensors, dtype, &device); + let model = BertModel::load(vb, &config).context("Failed to load E5 model")?; + + info!( + "Embedder initialized from embedded model (device: {:?}, dim={})", + device, config.hidden_size + ); + + Ok(Self { + model, + tokenizer, + config, + device, + }) + } - info!("Embedder initialized successfully"); + fn from_path(model_path: &Path, device: Device, max_length: Option) -> Result { + let config_path = model_path.join("config.json"); + let tokenizer_path = model_path.join("tokenizer.json"); + let weights_path = model_path.join("model.safetensors"); + if !weights_path.exists() { + let onnx_path = model_path.join("model_optimized.onnx"); + return Err(anyhow!( + "Unsupported embedder format. Expected model.safetensors at {} or ONNX at {}", + weights_path.display(), + onnx_path.display() + )); + } - Ok(Self { model, config }) + let config_str = safe_path::safe_read_to_string(&config_path) + .with_context(|| format!("Failed to read {}", config_path.display()))?; + let config: BertConfig = + serde_json::from_str(&config_str).context("Failed to parse E5 config.json")?; + + let tokenizer_str = safe_path::safe_read_to_string(&tokenizer_path) + .with_context(|| format!("Failed to read {}", tokenizer_path.display()))?; + let tokenizer: Tokenizer = tokenizer_str + .parse() + .map_err(|e| anyhow!("Failed to load tokenizer: {}", e))?; + let tokenizer = prepare_tokenizer(tokenizer, &config, max_length)?; + + let dtype = device.bf16_default_to_f32(); + let vb = unsafe { + VarBuilder::from_mmaped_safetensors(&[&weights_path], dtype, &device) + .context("Failed to load E5 weights")? + }; + let model = BertModel::load(vb, &config).context("Failed to load E5 model")?; + + info!( + "Embedder initialized from path: {} (device: {:?}, dim={})", + model_path.display(), + device, + config.hidden_size + ); + + Ok(Self { + model, + tokenizer, + config, + device, + }) } - /// Embed a single text + /// Embed a single text (query) /// - /// For queries (search), the text is automatically prefixed with "query: " - /// For passages (documents), use `embed_passage` instead + /// For queries (search), the text is prefixed with "query: " pub fn embed(&mut self, text: &str) -> Result> { - let query = format!("query: {}", text); - let embeddings = self - .model - .embed(vec![query], None) - .context("Failed to generate embedding")?; - - embeddings - .into_iter() + let vecs = self.embed_batch(&[text])?; + vecs.into_iter() .next() - .ok_or_else(|| anyhow::anyhow!("No embedding generated")) + .ok_or_else(|| anyhow!("No embedding generated")) } /// Embed a passage (document) for indexing /// /// Passages are prefixed with "passage: " for optimal retrieval pub fn embed_passage(&mut self, text: &str) -> Result> { - let passage = format!("passage: {}", text); - let embeddings = self - .model - .embed(vec![passage], None) - .context("Failed to generate passage embedding")?; - - embeddings - .into_iter() + let vecs = self.embed_passages(&[text])?; + vecs.into_iter() .next() - .ok_or_else(|| anyhow::anyhow!("No embedding generated")) + .ok_or_else(|| anyhow!("No embedding generated")) } /// Embed multiple texts at once (queries) pub fn embed_batch(&mut self, texts: &[&str]) -> Result>> { - let queries: Vec = texts.iter().map(|t| format!("query: {}", t)).collect(); - let string_refs: Vec<&str> = queries.iter().map(|s| s.as_str()).collect(); - - self.model - .embed(string_refs, None) - .context("Failed to generate batch embeddings") + let inputs: Vec = texts.iter().map(|t| format!("query: {}", t)).collect(); + self.embed_internal(&inputs) } /// Embed multiple passages at once (documents) pub fn embed_passages(&mut self, texts: &[&str]) -> Result>> { - let passages: Vec = texts.iter().map(|t| format!("passage: {}", t)).collect(); - let string_refs: Vec<&str> = passages.iter().map(|s| s.as_str()).collect(); + let inputs: Vec = texts.iter().map(|t| format!("passage: {}", t)).collect(); + self.embed_internal(&inputs) + } - self.model - .embed(string_refs, None) - .context("Failed to generate passage embeddings") + fn embed_internal(&mut self, inputs: &[String]) -> Result>> { + let (input_ids, token_type_ids, attention_mask) = encode_batch( + &self.tokenizer, + inputs, + self.config.pad_token_id as u32, + self.device.clone(), + )?; + + let outputs = self + .model + .forward(&input_ids, &token_type_ids, Some(&attention_mask))?; + + let pooled = mean_pool(&outputs, &attention_mask)?; + let normalized = l2_normalize(&pooled)?; + normalized + .to_vec2::() + .context("Failed to convert embeddings to Vec") } /// Calculate cosine similarity between two embeddings @@ -162,19 +255,179 @@ impl EmbedderEngine { /// Get embedding dimension pub fn dimension(&self) -> usize { - // E5 models output 1024-dim for large, 768 for base, 384 for small - match self.config.model { - EmbeddingModel::MultilingualE5Large => 1024, - EmbeddingModel::MultilingualE5Base => 768, - EmbeddingModel::MultilingualE5Small => 384, - _ => 1024, // Default assumption + self.config.hidden_size + } + + /// Get the device being used + pub fn device(&self) -> &Device { + &self.device + } +} + +fn resolve_model_path(explicit: Option<&PathBuf>, repo_override: Option<&str>) -> Result { + if let Some(path) = explicit { + return Ok(path.clone()); + } + + if let Ok(path) = std::env::var("CODESCRIBE_EMBEDDER_PATH") { + let p = PathBuf::from(path); + if model_files_present(&p) { + return Ok(p); } } - /// Get the model being used - pub fn model(&self) -> &EmbeddingModel { - &self.config.model + if let Some(repo) = repo_override { + if let Some(snapshot) = hf_cache::find_snapshot_with_any( + repo, + &["config.json", "tokenizer.json"], + &["model.safetensors", "model_optimized.onnx"], + ) { + return Ok(snapshot); + } + } else if let Some(snapshot) = hf_cache::find_snapshot_with_any( + DEFAULT_E5_REPO, + &["config.json", "tokenizer.json"], + &["model.safetensors", "model_optimized.onnx"], + ) { + return Ok(snapshot); + } + + Err(anyhow!( + "E5 model not found. Run: hf download {} (uses cache) or set CODESCRIBE_EMBEDDER_PATH / {}", + repo_override.unwrap_or(DEFAULT_E5_REPO), + ENV_EMBEDDER_REPO + )) +} + +fn model_files_present(path: &Path) -> bool { + let has_config = path.join("config.json").exists() && path.join("tokenizer.json").exists(); + let has_weights = path.join("model.safetensors").exists(); + let has_onnx = path.join("model_optimized.onnx").exists(); + has_config && (has_weights || has_onnx) +} + +fn prepare_tokenizer( + tokenizer: Tokenizer, + config: &BertConfig, + max_length_override: Option, +) -> Result { + let max_len = max_length_override + .unwrap_or(config.max_position_embeddings) + .min(DEFAULT_MAX_LENGTH); + + let pad_id = config.pad_token_id as u32; + let pad_token = tokenizer + .id_to_token(pad_id) + .unwrap_or_else(|| "[PAD]".to_string()); + + let mut tokenizer = tokenizer; + tokenizer.with_padding(Some(PaddingParams { + strategy: PaddingStrategy::BatchLongest, + pad_id, + pad_token, + ..Default::default() + })); + tokenizer + .with_truncation(Some(TruncationParams { + max_length: max_len, + ..Default::default() + })) + .map_err(anyhow::Error::msg)?; + + Ok(tokenizer) +} + +fn encode_batch( + tokenizer: &Tokenizer, + inputs: &[String], + pad_id: u32, + device: Device, +) -> Result<(Tensor, Tensor, Tensor)> { + let encodings = tokenizer + .encode_batch(inputs.to_vec(), true) + .map_err(|e| anyhow!("Tokenization failed: {}", e))?; + + let max_len = encodings.iter().map(|e| e.len()).max().unwrap_or(0); + + let mut input_ids = Vec::with_capacity(encodings.len() * max_len); + let mut token_type_ids = Vec::with_capacity(encodings.len() * max_len); + let mut attention_mask = Vec::with_capacity(encodings.len() * max_len); + + for enc in encodings { + let ids = enc.get_ids(); + let types = enc.get_type_ids(); + let mask = enc.get_attention_mask(); + + let mut ids_vec = ids.to_vec(); + let mut type_vec = if types.is_empty() { + vec![0u32; ids.len()] + } else { + types.to_vec() + }; + let mut mask_vec = mask.to_vec(); + + pad_to(&mut ids_vec, max_len, pad_id); + pad_to(&mut type_vec, max_len, 0); + pad_to(&mut mask_vec, max_len, 0); + + input_ids.extend_from_slice(&ids_vec); + token_type_ids.extend_from_slice(&type_vec); + attention_mask.extend_from_slice(&mask_vec); + } + + let batch = inputs.len(); + let input_ids = Tensor::from_vec(input_ids, (batch, max_len), &device)?.to_dtype(DType::I64)?; + let token_type_ids = + Tensor::from_vec(token_type_ids, (batch, max_len), &device)?.to_dtype(DType::I64)?; + let attention_mask = + Tensor::from_vec(attention_mask, (batch, max_len), &device)?.to_dtype(DType::F32)?; + + Ok((input_ids, token_type_ids, attention_mask)) +} + +fn pad_to(vec: &mut Vec, target_len: usize, pad: u32) { + if vec.len() < target_len { + vec.extend(std::iter::repeat_n(pad, target_len - vec.len())); + } +} + +fn mean_pool(hidden: &Tensor, mask: &Tensor) -> Result { + // hidden: [batch, seq, hidden], mask: [batch, seq] + let mask = mask.unsqueeze(2)?; // [batch, seq, 1] + let masked = hidden.broadcast_mul(&mask)?; + let sum = masked.sum(1)?; // [batch, hidden] + let counts = mask.sum(1)?; // [batch, 1] + let eps = Tensor::from_vec(vec![1e-9f32], (1,), hidden.device())?; + let counts = counts.broadcast_add(&eps)?; + Ok(sum.broadcast_div(&counts)?) +} + +fn l2_normalize(t: &Tensor) -> Result { + let squared = t.sqr()?; + let sum = squared.sum(1)?.unsqueeze(1)?; + let norm = sum.sqrt()?; + let eps = Tensor::from_vec(vec![1e-9f32], (1,), t.device())?; + let norm = norm.broadcast_add(&eps)?; + Ok(t.broadcast_div(&norm)?) +} + +fn move_tensors_to_device( + tensors: std::collections::HashMap, + device: &Device, + dtype: DType, +) -> Result> { + let mut result = std::collections::HashMap::with_capacity(tensors.len()); + + for (name, tensor) in tensors { + let mut t = tensor; + if t.dtype() != dtype { + t = t.to_dtype(dtype)?; + } + t = t.to_device(device)?; + result.insert(name, t); } + + Ok(result) } #[cfg(test)] @@ -196,20 +449,4 @@ mod tests { let sim = EmbedderEngine::similarity(&a, &b); assert!(sim.abs() < 0.001); } - - #[test] - fn test_similarity_opposite() { - let a = vec![1.0, 0.0, 0.0]; - let b = vec![-1.0, 0.0, 0.0]; - let sim = EmbedderEngine::similarity(&a, &b); - assert!((sim + 1.0).abs() < 0.001); - } - - #[test] - fn test_similarity_empty() { - let a: Vec = vec![]; - let b: Vec = vec![]; - let sim = EmbedderEngine::similarity(&a, &b); - assert_eq!(sim, 0.0); - } } diff --git a/core/embedder/mod.rs b/core/embedder/mod.rs index 6dda3bdb..d917468f 100644 --- a/core/embedder/mod.rs +++ b/core/embedder/mod.rs @@ -1,14 +1,15 @@ -//! Text Embedder module - semantic embeddings using E5 via fastembed. +//! Text Embedder module - semantic embeddings using E5 (offline). //! //! Provides semantic text embeddings for RAG, similarity search, and context matching. -//! Uses fastembed with the multilingual-e5-large model for high-quality embeddings. +//! Uses a local/embedded multilingual-e5-large model (no runtime downloads by default). +//! Override with `CODESCRIBE_EMBEDDER_REPO=intfloat/multilingual-e5-small` (HF cache). //! //! ## Quick Start //! //! ```ignore //! use codescribe_core::embedder; //! -//! // Initialize embedder (downloads model on first use) +//! // Initialize embedder (embedded or local model) //! embedder::init()?; //! //! // Embed text @@ -23,6 +24,7 @@ //! //! Created by M&K (c)2026 VetCoders +pub mod embedded; pub mod engine; pub mod singleton; @@ -32,5 +34,5 @@ pub use singleton::{embed, embed_batch, init, is_initialized, similarity}; /// Default embedding dimension for E5 models pub const EMBEDDING_DIM: usize = 1024; -/// Model identifier for HuggingFace -pub const DEFAULT_MODEL: &str = "intfloat/multilingual-e5-large"; +/// Default model directory name (for local/embedded builds) +pub const DEFAULT_MODEL: &str = "e5-large"; diff --git a/core/hf_cache.rs b/core/hf_cache.rs new file mode 100644 index 00000000..50e3bd0e --- /dev/null +++ b/core/hf_cache.rs @@ -0,0 +1,128 @@ +//! HuggingFace cache utilities. +//! +//! Resolves local snapshot paths for repos downloaded via `hf download`. +//! This avoids hardcoded model directories and uses HF cache directly. +//! +//! Created by M&K (c)2026 VetCoders + +use std::env; +use std::fs; +use std::path::PathBuf; +use std::time::SystemTime; + +use directories::BaseDirs; + +pub fn find_snapshot(repo: &str, required: &[&str]) -> Option { + find_snapshot_with_any(repo, required, &[]) +} + +pub fn find_snapshot_with_any( + repo: &str, + required_all: &[&str], + required_any: &[&str], +) -> Option { + for base in cache_bases() { + if let Some(snapshot) = find_snapshot_in_base(&base, repo, required_all, required_any) { + return Some(snapshot); + } + } + None +} + +fn cache_bases() -> Vec { + let mut out = Vec::new(); + if let Ok(path) = env::var("CODESCRIBE_HF_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HUGGINGFACE_HUB_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HF_HUB_CACHE") { + out.push(PathBuf::from(path)); + } + if let Ok(path) = env::var("HF_HOME") { + out.push(PathBuf::from(path).join("hub")); + } + if let Some(dirs) = BaseDirs::new() { + out.push( + dirs.home_dir() + .join(".cache") + .join("huggingface") + .join("hub"), + ); + // Support local cache used by CodeScribe tools (hf download into ~/.codescribe/embeddings) + out.push(dirs.home_dir().join(".codescribe").join("embeddings")); + out.push( + dirs.home_dir() + .join(".codescribe") + .join("embeddings") + .join("hub"), + ); + } + out.sort(); + out.dedup(); + out +} + +fn find_snapshot_in_base( + base: &PathBuf, + repo: &str, + required_all: &[&str], + required_any: &[&str], +) -> Option { + let repo_dir = base.join(format!("models--{}", repo.replace('/', "--"))); + let snapshots_dir = repo_dir.join("snapshots"); + + let snapshots_dir = if snapshots_dir.exists() { + snapshots_dir + } else { + // Case-insensitive repo match fallback (HF cache uses original casing) + let target = repo.to_ascii_lowercase(); + let mut matched: Option = None; + if let Ok(entries) = fs::read_dir(base) { + for entry in entries.flatten() { + let name = entry.file_name(); + let name = name.to_string_lossy(); + if !name.starts_with("models--") { + continue; + } + let repo_id = name + .strip_prefix("models--") + .unwrap_or("") + .replace("--", "/"); + if repo_id.to_ascii_lowercase() == target { + matched = Some(entry.path().join("snapshots")); + break; + } + } + } + matched? + }; + + let entries = fs::read_dir(&snapshots_dir).ok()?; + + let mut best: Option<(SystemTime, PathBuf)> = None; + + for entry in entries.flatten() { + let path = entry.path(); + if !path.is_dir() { + continue; + } + if !required_all.iter().all(|f| path.join(f).exists()) { + continue; + } + if !required_any.is_empty() && !required_any.iter().any(|f| path.join(f).exists()) { + continue; + } + let modified = entry + .metadata() + .and_then(|m| m.modified()) + .unwrap_or(SystemTime::UNIX_EPOCH); + match &best { + Some((best_time, _)) if *best_time >= modified => {} + _ => best = Some((modified, path)), + } + } + + best.map(|(_, p)| p) +} diff --git a/core/lib.rs b/core/lib.rs index bcc74079..0c24f84c 100644 --- a/core/lib.rs +++ b/core/lib.rs @@ -16,8 +16,8 @@ //! //! - **whisper** - Embedded Whisper model (~900MB in binary), zero I/O //! - **tts** - Embedded CSM-1B model (~1GB in binary), text-to-speech -//! - **vad** - Voice activity detection using WebRTC VAD -//! - **embedder** - Text embeddings using E5 model via fastembed +//! - **vad** - Voice activity detection using Silero VAD neural network +//! - **embedder** - Text embeddings using E5 model (offline) //! - **audio** - Recording and audio loading //! - **config** - User configuration //! - **ai_formatting** - Post-processing with LLMs @@ -31,7 +31,9 @@ pub mod audio; pub mod config; pub mod conversation; +pub mod demux; pub mod embedder; +mod hf_cache; pub mod ipc; pub mod llm; pub mod pipeline; @@ -86,7 +88,7 @@ pub mod vad_api { // Public API - Embedder (text embeddings) // ═══════════════════════════════════════════════════════════ -/// Text embeddings using E5 model via fastembed +/// Text embeddings using E5 model (offline) pub mod embedder_api { pub use crate::embedder::{ DEFAULT_MODEL, EMBEDDING_DIM, EmbedderConfig, EmbedderEngine, embed, embed_batch, init, @@ -123,6 +125,7 @@ pub use config::{get_assistive_prompt_path, get_formatting_prompt_path, reset_to // ═══════════════════════════════════════════════════════════ pub use llm::{ai_formatting, client}; +pub use pipeline::contracts; pub use pipeline::stream_postprocess; pub use quality::{quality_loop, quality_report}; pub use util::{safe_path, status}; diff --git a/core/pipeline/contracts.rs b/core/pipeline/contracts.rs new file mode 100644 index 00000000..b4a054e2 --- /dev/null +++ b/core/pipeline/contracts.rs @@ -0,0 +1,334 @@ +//! Pipeline contracts — shared data types for the transcription pipeline. +//! +//! These types define the boundaries between pipeline stages: +//! AudioChunk → SpeechUtterance → RawTranscript → PostprocessResult → TranscriptDelta → DeltaSink +//! +//! Created by M&K (c)2026 VetCoders + +use serde::{Deserialize, Serialize}; + +// ═══════════════════════════════════════════════════════════ +// Audio stage +// ═══════════════════════════════════════════════════════════ + +/// A chunk of raw audio samples from the recorder. +#[derive(Debug, Clone)] +pub struct AudioChunk { + pub samples: Vec, + pub sample_rate: u32, + /// Start time relative to recording session (seconds). + pub start_ts: f32, + /// End time relative to recording session (seconds). + pub end_ts: f32, +} + +/// A complete speech utterance (after VAD gating / silence detection). +#[derive(Debug, Clone)] +pub struct SpeechUtterance { + pub samples: Vec, + pub sample_rate: u32, + pub start_ts: f32, + pub end_ts: f32, +} + +impl SpeechUtterance { + /// Duration in seconds. + pub fn duration(&self) -> f32 { + self.end_ts - self.start_ts + } +} + +// ═══════════════════════════════════════════════════════════ +// STT stage +// ═══════════════════════════════════════════════════════════ + +/// Raw output from a speech-to-text engine (Whisper or future providers). +#[derive(Debug, Clone, Default)] +pub struct RawTranscript { + /// The transcribed text (untouched by postprocessing). + pub text: String, + /// Per-segment breakdown, if the engine provides it. + pub segments: Vec, +} + +/// A single segment from the STT engine (optional granularity). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct TranscriptSegment { + pub text: String, + pub start_ts: f32, + pub end_ts: f32, +} + +// ═══════════════════════════════════════════════════════════ +// Postprocess stage +// ═══════════════════════════════════════════════════════════ + +/// Result of postprocessing a raw transcript (lexicon + semantic gate + cleanup). +#[derive(Debug, Clone)] +pub struct PostprocessResult { + /// Cleaned text ready for user-facing output. + pub text: String, + /// Whether the semantic gate dropped this chunk (text will be empty). + pub dropped: bool, +} + +// ═══════════════════════════════════════════════════════════ +// Delta / presentation stage +// ═══════════════════════════════════════════════════════════ + +/// Backspace character used in delta encoding. +pub const BACKSPACE: char = '\u{0008}'; + +/// An incremental update to the transcript buffer. +/// +/// The `delta` string may contain `\u{0008}` (backspace) characters +/// that instruct the consumer to delete preceding characters before +/// appending the rest. This allows corrections without full-buffer resend. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct TranscriptDelta { + pub delta: String, +} + +impl TranscriptDelta { + /// Wrap a raw delta string (may contain backspace chars) without modification. + pub fn from_raw(delta: impl Into) -> Self { + Self { + delta: delta.into(), + } + } + + /// Create a simple append-only delta (no backspaces). + pub fn append(text: impl Into) -> Self { + Self { delta: text.into() } + } + + /// Create a delta that replaces the last `delete_count` characters, then appends `new_text`. + pub fn replace(delete_count: usize, new_text: &str) -> Self { + let mut delta = String::with_capacity(delete_count + new_text.len()); + for _ in 0..delete_count { + delta.push(BACKSPACE); + } + delta.push_str(new_text); + Self { delta } + } + + /// Build a minimal delta from "before" and "after" full texts. + /// + /// Finds the common prefix, emits backspaces for the removed tail of `before`, + /// then appends the new tail of `after`. Returns `None` if texts are identical. + pub fn from_diff(before: &str, after: &str) -> Option { + if before == after { + return None; + } + + let common_prefix_len = before + .chars() + .zip(after.chars()) + .take_while(|(a, b)| a == b) + .count(); + + let delete_count = before.chars().count() - common_prefix_len; + let new_tail: String = after.chars().skip(common_prefix_len).collect(); + + Some(Self::replace(delete_count, &new_tail)) + } + + /// Apply this delta to a mutable string buffer (the inverse of `from_diff`). + pub fn apply(&self, target: &mut String) { + for ch in self.delta.chars() { + if ch == BACKSPACE { + target.pop(); + } else { + target.push(ch); + } + } + } + + /// Returns `true` if this delta contains only backspace characters (pure deletion). + pub fn is_delete_only(&self) -> bool { + !self.delta.is_empty() && self.delta.chars().all(|c| c == BACKSPACE) + } + + /// Returns `true` if this delta contains no backspaces (pure append). + pub fn is_append_only(&self) -> bool { + !self.delta.is_empty() && self.delta.chars().all(|c| c != BACKSPACE) + } +} + +impl std::fmt::Display for TranscriptDelta { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.delta) + } +} + +// ═══════════════════════════════════════════════════════════ +// Traits (adapter boundaries) +// ═══════════════════════════════════════════════════════════ + +/// Adapter for speech-to-text engines. +/// +/// Implementations: `LocalWhisperEngine` (current), future cloud STT providers. +pub trait TranscriptionAdapter: Send + Sync { + fn transcribe( + &self, + utterance: &SpeechUtterance, + language: Option<&str>, + ) -> anyhow::Result; +} + +/// Post-processor for raw transcripts. +/// +/// Implementations: `StreamPostProcessor` (semantic gate), `LexiconPostProcessor`. +pub trait PostProcessor: Send { + fn process(&mut self, raw: &RawTranscript) -> Option; +} + +/// Sink for transcript deltas (UI, IPC, clipboard, etc). +/// +/// This decouples the streaming pipeline from presentation concerns. +pub trait DeltaSink: Send + Sync { + fn apply(&self, delta: &TranscriptDelta); +} + +// ═══════════════════════════════════════════════════════════ +// Tests +// ═══════════════════════════════════════════════════════════ + +#[cfg(test)] +mod tests { + use super::*; + + // ── TranscriptDelta roundtrip ── + + #[test] + fn delta_append_roundtrip() { + let mut buf = String::from("Hello"); + let delta = TranscriptDelta::append(" world"); + delta.apply(&mut buf); + assert_eq!(buf, "Hello world"); + } + + #[test] + fn delta_replace_roundtrip() { + let mut buf = String::from("Hello worl"); + // Fix typo: delete "worl" (4 chars), append "world!" + let delta = TranscriptDelta::replace(4, "world!"); + delta.apply(&mut buf); + assert_eq!(buf, "Hello world!"); + } + + #[test] + fn delta_from_diff_roundtrip() { + let before = "Cześć, jestem lekarzem"; + let after = "Cześć, jestem weterynarzem"; + + let delta = TranscriptDelta::from_diff(before, after).unwrap(); + let mut buf = before.to_string(); + delta.apply(&mut buf); + assert_eq!(buf, after); + } + + #[test] + fn delta_from_diff_identical_returns_none() { + assert!(TranscriptDelta::from_diff("same", "same").is_none()); + } + + #[test] + fn delta_from_diff_complete_replacement() { + let before = "abc"; + let after = "xyz"; + let delta = TranscriptDelta::from_diff(before, after).unwrap(); + let mut buf = before.to_string(); + delta.apply(&mut buf); + assert_eq!(buf, after); + } + + #[test] + fn delta_from_diff_empty_to_text() { + let delta = TranscriptDelta::from_diff("", "hello").unwrap(); + let mut buf = String::new(); + delta.apply(&mut buf); + assert_eq!(buf, "hello"); + assert!(delta.is_append_only()); + } + + #[test] + fn delta_from_diff_text_to_empty() { + let delta = TranscriptDelta::from_diff("hello", "").unwrap(); + let mut buf = String::from("hello"); + delta.apply(&mut buf); + assert_eq!(buf, ""); + assert!(delta.is_delete_only()); + } + + #[test] + fn delta_from_diff_unicode_polish() { + let before = "Zółty pies"; + let after = "Żółty pies"; + let delta = TranscriptDelta::from_diff(before, after).unwrap(); + let mut buf = before.to_string(); + delta.apply(&mut buf); + assert_eq!(buf, after); + } + + #[test] + fn delta_backspace_sequence() { + // Simulate Whisper correcting "transkryp" → "transkrypcja" + let mut buf = String::from("transkryp"); + let delta = TranscriptDelta::append("cja"); + delta.apply(&mut buf); + assert_eq!(buf, "transkrypcja"); + } + + #[test] + fn delta_multi_step_simulation() { + // Simulate streaming: chunk1 → chunk2 (with correction) → chunk3 + let mut buf = String::new(); + + // Chunk 1: "Witaj " + TranscriptDelta::append("Witaj ").apply(&mut buf); + assert_eq!(buf, "Witaj "); + + // Chunk 2: Whisper corrects to "Witaj, " (backspace space, add ", ") + TranscriptDelta::replace(1, ", ").apply(&mut buf); + assert_eq!(buf, "Witaj, "); + + // Chunk 3: append "świecie!" + TranscriptDelta::append("świecie!").apply(&mut buf); + assert_eq!(buf, "Witaj, świecie!"); + } + + // ── SpeechUtterance ── + + #[test] + fn utterance_duration() { + let u = SpeechUtterance { + samples: vec![0.0; 16000], + sample_rate: 16000, + start_ts: 1.5, + end_ts: 2.5, + }; + assert!((u.duration() - 1.0).abs() < f32::EPSILON); + } + + // ── RawTranscript ── + + #[test] + fn raw_transcript_default_has_no_segments() { + let rt = RawTranscript::default(); + assert!(rt.text.is_empty()); + assert!(rt.segments.is_empty()); + } + + // ── PostprocessResult ── + + #[test] + fn postprocess_result_dropped() { + let r = PostprocessResult { + text: String::new(), + dropped: true, + }; + assert!(r.dropped); + assert!(r.text.is_empty()); + } +} diff --git a/core/pipeline/dedup.rs b/core/pipeline/dedup.rs new file mode 100644 index 00000000..7f411f3f --- /dev/null +++ b/core/pipeline/dedup.rs @@ -0,0 +1,241 @@ +//! Unified deduplication for the transcription pipeline. +//! +//! Two granularities: +//! - **Chunk overlap** (`dedup_chunk_overlap`): word-level exact+fuzzy dedup at chunk boundaries +//! (ported from `engine::append_with_overlap_dedup`) +//! - **Suffix overlap** (`strip_suffix_overlap`): character-level suffix/prefix strip between utterances +//! (ported from `TranscriptionPipeline::strip_overlap`) +//! +//! # Note: batch vs live dedup +//! +//! The **live streaming** path (`pipeline::streaming`) uses these functions. +//! The **batch/file** path (`engine::transcribe_long_streaming`) still uses +//! `engine::append_with_overlap_dedup` — an identical algorithm kept local to +//! the engine module. This is intentional: the batch path is self-contained +//! and does not route through the pipeline. + +// ── helpers ────────────────────────────────────────────── + +fn normalize_token_for_overlap(token: &str) -> String { + let mut out = String::new(); + for ch in token.chars() { + if ch.is_alphanumeric() { + out.extend(ch.to_lowercase()); + } + } + if out.is_empty() { + token.to_lowercase() + } else { + out + } +} + +/// Word-level edit distance for short sequences (used by fuzzy overlap). +fn word_edit_distance(a: &[String], b: &[String]) -> usize { + let m = a.len(); + let n = b.len(); + let mut prev: Vec = (0..=n).collect(); + let mut cur = vec![0usize; n + 1]; + + for i in 1..=m { + cur[0] = i; + for j in 1..=n { + let cost = if a[i - 1] == b[j - 1] { 0 } else { 1 }; + cur[j] = (prev[j] + 1).min(cur[j - 1] + 1).min(prev[j - 1] + cost); + } + prev.clone_from(&cur); + } + prev[n] +} + +// ── public API ─────────────────────────────────────────── + +/// Append `segment` to `out`, deduplicating overlapping word sequences at the boundary. +/// +/// Two-pass approach: +/// 1. Exact match (fast path) — suffix of `out` == prefix of `segment` +/// 2. Fuzzy match (fallback) — allows up to k/3 word-level edits in overlap region. +/// Catches cases where Whisper produces slightly different text for the same audio. +pub fn dedup_chunk_overlap(out: &mut String, segment: &str) { + let seg = segment.trim(); + if seg.is_empty() { + return; + } + + if out.trim().is_empty() { + out.push_str(seg); + return; + } + + let out_trim = out.trim_end(); + let out_words: Vec<&str> = out_trim.split_whitespace().collect(); + let seg_words: Vec<&str> = seg.split_whitespace().collect(); + if out_words.is_empty() || seg_words.is_empty() { + if !out.ends_with(' ') { + out.push(' '); + } + out.push_str(seg); + return; + } + + let out_norm: Vec = out_words + .iter() + .map(|word| normalize_token_for_overlap(word)) + .collect(); + let seg_norm: Vec = seg_words + .iter() + .map(|word| normalize_token_for_overlap(word)) + .collect(); + + let max_overlap = out_words.len().min(seg_words.len()).min(30); + let mut overlap = 0usize; + + // Pass 1: exact match (fast path) + for k in (1..=max_overlap).rev() { + if out_norm[out_norm.len() - k..] == seg_norm[..k] { + overlap = k; + break; + } + } + + // Pass 2: fuzzy match — allow up to k/3 word edits (min 1) + if overlap == 0 { + for k in (3..=max_overlap).rev() { + let tail = &out_norm[out_norm.len() - k..]; + let head = &seg_norm[..k]; + let max_errors = (k / 3).max(1); + let dist = word_edit_distance(tail, head); + if dist <= max_errors { + overlap = k; + tracing::debug!( + "[FUZZY_DEDUP] matched k={} dist={} max_err={} tail={:?} head={:?}", + k, + dist, + max_errors, + &tail[..tail.len().min(5)], + &head[..head.len().min(5)] + ); + break; + } + } + } + + if !out.ends_with(' ') { + out.push(' '); + } + + if overlap >= seg_words.len() { + return; + } + if overlap > 0 { + out.push_str(&seg_words[overlap..].join(" ")); + } else { + out.push_str(seg); + } +} + +/// Strip overlapping prefix from `new_text` that matches a suffix of `last_suffix`. +/// +/// Character-level, case-insensitive. Uses char boundaries to avoid +/// panics on multi-byte UTF-8 (Polish diacritics, emoji, etc.). +pub fn strip_suffix_overlap(last_suffix: &str, new_text: &str) -> String { + if last_suffix.is_empty() { + return new_text.to_string(); + } + + // Collect valid byte offsets from char boundaries (longest first). + let suffix_bounds: Vec = last_suffix.char_indices().map(|(i, _)| i).collect(); + let text_bounds: Vec = { + let mut v: Vec = new_text.char_indices().map(|(i, _)| i).collect(); + v.push(new_text.len()); // include final boundary + v + }; + + // Try overlap lengths from longest to shortest (min 3 bytes). + for &suffix_start in &suffix_bounds { + let suffix_tail = &last_suffix[suffix_start..]; + let tail_len = suffix_tail.len(); + if tail_len < 3 { + break; + } + // Find the matching char boundary in new_text for this byte length. + if text_bounds.binary_search(&tail_len).is_ok() + && suffix_tail.eq_ignore_ascii_case(&new_text[..tail_len]) + { + let stripped = new_text[tail_len..].trim_start(); + if !stripped.is_empty() { + return stripped.to_string(); + } + return String::new(); + } + } + new_text.to_string() +} + +#[cfg(test)] +mod tests { + use super::*; + + // ── chunk dedup ────────────────────────────────────── + + #[test] + fn test_chunk_dedup_exact() { + let mut out = "Hello world this is".to_string(); + dedup_chunk_overlap(&mut out, "this is a test"); + assert_eq!(out, "Hello world this is a test"); + } + + #[test] + fn test_chunk_dedup_fuzzy() { + // 1-word edit in a 3-word overlap region → should still dedup + let mut out = "one two three four".to_string(); + dedup_chunk_overlap(&mut out, "three foor five six"); + // "four" vs "foor" = 1 edit in k=2 region... but fuzzy needs k>=3 + // Let's use a bigger overlap: "two three four" vs "two three foor" + let mut out2 = "one two three four".to_string(); + dedup_chunk_overlap(&mut out2, "two three foor five six"); + // k=3 overlap: ["two","three","four"] vs ["two","three","foor"] → dist=1, max_err=1 → match + assert_eq!(out2, "one two three four five six"); + } + + #[test] + fn test_chunk_dedup_no_overlap() { + let mut out = "Hello world".to_string(); + dedup_chunk_overlap(&mut out, "completely different"); + assert_eq!(out, "Hello world completely different"); + } + + // ── suffix overlap ─────────────────────────────────── + + #[test] + fn test_suffix_overlap_basic() { + let result = strip_suffix_overlap("Hello world.", "world. And more."); + assert_eq!(result, "And more."); + } + + #[test] + fn test_suffix_overlap_no_match() { + let result = strip_suffix_overlap("Hello world.", "Something else."); + assert_eq!(result, "Something else."); + } + + #[test] + fn test_suffix_overlap_empty() { + let result = strip_suffix_overlap("", "Hello world."); + assert_eq!(result, "Hello world."); + } + + #[test] + fn test_suffix_overlap_polish_diacritics() { + // "ż" is 2 bytes in UTF-8 — old code would panic slicing mid-char + let result = strip_suffix_overlap("weterynarzem.", "weterynarzem. Dziękuję."); + assert_eq!(result, "Dziękuję."); + } + + #[test] + fn test_suffix_overlap_emoji() { + // 🐕 is 4 bytes — stress-test char boundary logic + let result = strip_suffix_overlap("pies 🐕.", "🐕. Koniec."); + assert_eq!(result, "Koniec."); + } +} diff --git a/core/pipeline/mod.rs b/core/pipeline/mod.rs index 74bcbc87..d55fff58 100644 --- a/core/pipeline/mod.rs +++ b/core/pipeline/mod.rs @@ -1 +1,7 @@ +pub mod contracts; +pub mod dedup; +pub mod sinks; pub mod stream_postprocess; +pub mod streaming; +#[cfg(test)] +mod tests; diff --git a/core/pipeline/sinks.rs b/core/pipeline/sinks.rs new file mode 100644 index 00000000..d4c86cf9 --- /dev/null +++ b/core/pipeline/sinks.rs @@ -0,0 +1,102 @@ +/// Concrete `DeltaSink` adapters for pipeline consumers. +/// +/// - `CallbackSink`: backward-compat bridge wrapping `Arc` +/// - `CollectorSink`: test helper that collects all deltas +use std::sync::{Arc, Mutex}; + +use crate::pipeline::contracts::{DeltaSink, TranscriptDelta}; + +/// Backward-compatible adapter: wraps a plain `Fn(&str)` closure into `DeltaSink`. +pub struct CallbackSink { + callback: Arc, +} + +impl CallbackSink { + pub fn new(callback: Arc) -> Self { + Self { callback } + } +} + +impl DeltaSink for CallbackSink { + fn apply(&self, delta: &TranscriptDelta) { + (self.callback)(&delta.delta); + } +} + +/// Convenience constructor: wrap a plain `Fn(&str)` closure into `Arc`. +/// +/// This is the recommended entry point for external consumers who just want +/// to receive transcript deltas as `&str` without importing `CallbackSink` directly. +pub fn from_callback(f: F) -> Arc +where + F: Fn(&str) + Send + Sync + 'static, +{ + Arc::new(CallbackSink::new(Arc::new(f))) +} + +/// Test helper: collects all delta strings for assertions. +pub struct CollectorSink { + collected: Mutex>, +} + +impl Default for CollectorSink { + fn default() -> Self { + Self { + collected: Mutex::new(Vec::new()), + } + } +} + +impl CollectorSink { + pub fn new() -> Self { + Self::default() + } + + pub fn collected(&self) -> Vec { + self.collected.lock().unwrap().clone() + } +} + +impl DeltaSink for CollectorSink { + fn apply(&self, delta: &TranscriptDelta) { + self.collected.lock().unwrap().push(delta.delta.clone()); + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + + #[test] + fn test_callback_sink_forwards() { + let received = Arc::new(Mutex::new(String::new())); + let r = received.clone(); + let sink = CallbackSink::new(Arc::new(move |s: &str| { + *r.lock().unwrap() = s.to_string(); + })); + sink.apply(&TranscriptDelta::from_raw("hello")); + assert_eq!(*received.lock().unwrap(), "hello"); + } + + #[test] + fn test_from_callback_convenience() { + let received = Arc::new(Mutex::new(Vec::new())); + let r = received.clone(); + let sink = super::from_callback(move |s: &str| { + r.lock().unwrap().push(s.to_string()); + }); + sink.apply(&TranscriptDelta::from_raw("alpha")); + sink.apply(&TranscriptDelta::from_raw("beta")); + assert_eq!(*received.lock().unwrap(), vec!["alpha", "beta"]); + } + + #[test] + fn test_collector_sink_collects() { + let sink = CollectorSink::new(); + sink.apply(&TranscriptDelta::from_raw("one")); + sink.apply(&TranscriptDelta::from_raw("two")); + sink.apply(&TranscriptDelta::from_raw("three")); + assert_eq!(sink.collected(), vec!["one", "two", "three"]); + } +} diff --git a/core/pipeline/stream_postprocess.rs b/core/pipeline/stream_postprocess.rs index 866f8481..56ffed37 100644 --- a/core/pipeline/stream_postprocess.rs +++ b/core/pipeline/stream_postprocess.rs @@ -1,11 +1,8 @@ use std::collections::HashSet; use std::fs; use std::path::PathBuf; -use std::sync::{Mutex, OnceLock}; use std::time::SystemTime; -use anyhow::Result; -use fastembed::{EmbeddingModel, TextEmbedding, TextInitOptions}; use regex::Regex; use serde::{Deserialize, Serialize}; use tracing::{debug, info, warn}; @@ -24,8 +21,6 @@ const DEFAULT_NOVELTY_THRESHOLD: f32 = 0.12; const MAX_EMBED_CHARS: usize = 512; const MAX_DROPS_IN_ROW: u8 = 2; -static EMBEDDER: OnceLock>> = OnceLock::new(); - #[derive(Debug, Deserialize)] struct LexiconEntry { term: String, @@ -243,31 +238,19 @@ impl SemanticGate { return None; } - let embedder = EMBEDDER.get_or_init(|| Mutex::new(None)); - let mut guard = embedder.lock().ok()?; - - if guard.is_none() { - match init_embedder() { - Ok(model) => { - *guard = Some(model); - } - Err(e) => { - warn!("Failed to initialize ParaphraseMLMiniLM embedder: {}", e); - return None; - } + let input = truncate_for_embedding(text); + match crate::embedder::embed(&input) { + Ok(vec) => Some(vec), + Err(e) => { + warn!("Failed to embed text for semantic gate: {}", e); + None } } - - let model = guard.as_mut()?; - let input = truncate_for_embedding(text); - let embeddings = model.embed(vec![input.as_str()], None).ok()?; - embeddings.into_iter().next() } } #[derive(Debug)] pub struct StreamPostProcessor { - lexicon: Lexicon, gate: SemanticGate, stats: StreamPostProcessStats, } @@ -287,7 +270,6 @@ pub struct StreamPostProcessStats { impl StreamPostProcessor { pub fn new() -> Self { Self { - lexicon: Lexicon::from_builtin(), gate: SemanticGate::new(), stats: StreamPostProcessStats { embeddings_enabled: embeddings_enabled(), @@ -296,12 +278,13 @@ impl StreamPostProcessor { } } - /// Process a streaming chunk — applies lexicon, cleanup, and semantic gate. + /// Process a streaming chunk — applies cleanup + semantic gate only. + /// Lexicon is handled in buffered mode. pub fn process(&mut self, text: &str) -> Option { self.process_internal(text, true) } - /// Process a complete utterance — applies lexicon and cleanup, no semantic gate. + /// Process a complete utterance — cleanup only, no semantic gate. /// Use this for VAD-segmented utterances where each segment is naturally distinct. pub fn process_utterance(&mut self, text: &str) -> Option { self.process_internal(text, false) @@ -309,24 +292,16 @@ impl StreamPostProcessor { fn process_internal(&mut self, text: &str, apply_gate: bool) -> Option { self.stats.input_chunks += 1; - self.lexicon.maybe_reload(); - if text.trim().is_empty() { self.stats.dropped_chunks += 1; return None; } - let mut cleaned = self.lexicon.apply(text); - if cleaned != text { - self.stats.lexicon_rewrites += 1; - } - - let cleaned_after_cleanup = cleanup_artifacts(&cleaned); - if cleaned_after_cleanup != cleaned { + let cleaned_after_cleanup = cleanup_artifacts(text); + if cleaned_after_cleanup != text { self.stats.repetition_cleanups += 1; } - cleaned = cleaned_after_cleanup; - cleaned = normalize_whitespace(&cleaned); + let cleaned = normalize_whitespace(&cleaned_after_cleanup); if cleaned.trim().is_empty() { self.stats.dropped_chunks += 1; @@ -354,12 +329,44 @@ impl StreamPostProcessor { } } +pub struct LexiconPostProcessor { + lexicon: Lexicon, +} + +impl LexiconPostProcessor { + pub fn new() -> Self { + Self { + lexicon: Lexicon::from_builtin(), + } + } + + pub fn process(&mut self, text: &str) -> Option { + if text.trim().is_empty() { + return None; + } + self.lexicon.maybe_reload(); + let cleaned = self.lexicon.apply(text); + let cleaned_after_cleanup = cleanup_artifacts(&cleaned); + let cleaned_after_cleanup = normalize_whitespace(&cleaned_after_cleanup); + if cleaned_after_cleanup.trim().is_empty() { + return None; + } + Some(cleaned_after_cleanup) + } +} + impl Default for StreamPostProcessor { fn default() -> Self { Self::new() } } +impl Default for LexiconPostProcessor { + fn default() -> Self { + Self::new() + } +} + fn build_word_regex(input: &str) -> Option { let mut escaped = regex::escape(input); escaped = escaped.replace("\\ ", "\\s+"); @@ -377,7 +384,10 @@ fn env_f32(key: &str, default: f32) -> f32 { fn env_bool(key: &str) -> bool { std::env::var(key) .ok() - .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .map(|v| { + let v = v.trim(); + v == "1" || v.eq_ignore_ascii_case("true") + }) .unwrap_or(false) } @@ -449,7 +459,7 @@ fn cleanup_artifacts(text: &str) -> String { text.to_string() } -fn normalize_whitespace(text: &str) -> String { +pub(crate) fn normalize_whitespace(text: &str) -> String { text.split_whitespace().collect::>().join(" ") } @@ -468,23 +478,6 @@ fn is_suspicious(text: &str) -> bool { ratio < 0.5 || crate::ai_formatting::has_repetition_loop(text) } -fn init_embedder() -> Result { - let cache_dir = embedding_cache_dir(); - std::fs::create_dir_all(&cache_dir)?; - - info!("Initializing ParaphraseMLMiniLM embedder (this can take a while on first run)"); - let options = TextInitOptions::new(EmbeddingModel::ParaphraseMLMiniLML12V2Q) - .with_max_length(256) - .with_cache_dir(cache_dir) - .with_show_download_progress(true); - - TextEmbedding::try_new(options) -} - -fn embedding_cache_dir() -> PathBuf { - Config::config_dir().join("embeddings") -} - fn truncate_for_embedding(text: &str) -> String { if text.len() <= MAX_EMBED_CHARS { return text.to_string(); @@ -499,7 +492,7 @@ mod tests { #[test] fn test_lexicon_rewrite() { - let mut processor = StreamPostProcessor::new(); + let mut processor = LexiconPostProcessor::new(); let input = "Uzywam doker do kontenerow i mam api key do github."; let output = processor.process(input).expect("expected output"); assert!( @@ -640,28 +633,18 @@ mod tests { } #[test] - fn test_postprocessor_always_applies_lexicon_contract() { - // Contract: every call to process() applies lexicon rewrites - // regardless of semantic gate state or chunk history - let mut processor = StreamPostProcessor::new(); + fn test_lexicon_applied_in_buffered_processor() { + let mut processor = LexiconPostProcessor::new(); - // First call — lexicon should rewrite known terms let out1 = processor .process("Uzywam doker do kontenerow") .expect("non-empty"); - assert!( - out1.contains("Docker"), - "First call should apply lexicon: {out1}" - ); + assert!(out1.contains("Docker"), "Expected lexicon rewrite: {out1}"); - // Second call with different text — still applies lexicon let out2 = processor .process("Mam git hub repository z kodem") .expect("non-empty"); - assert!( - out2.contains("GitHub"), - "Second call should apply lexicon: {out2}" - ); + assert!(out2.contains("GitHub"), "Expected lexicon rewrite: {out2}"); } #[test] diff --git a/core/pipeline/streaming.rs b/core/pipeline/streaming.rs new file mode 100644 index 00000000..19b8b919 --- /dev/null +++ b/core/pipeline/streaming.rs @@ -0,0 +1,926 @@ +//! Streaming transcription pipeline — orchestration, buffered emission, and policy. +//! +//! Extracted from `audio::streaming_recorder` to decouple pipeline logic +//! (hallucination filtering, overlap dedup, re-transcription, buffered "typing" +//! emission) from the audio capture layer. +//! +//! Created by M&K (c)2026 VetCoders + +use crate::audio::chunker::{SpeechEvent, SpeechSession}; +use crate::pipeline::dedup::{dedup_chunk_overlap, strip_suffix_overlap}; +use crate::pipeline::stream_postprocess::{LexiconPostProcessor, StreamPostProcessor}; +use crate::stt::whisper; +use crate::stt::whisper::singleton::engine as get_engine; +use anyhow::{Result, anyhow}; +use chrono::SecondsFormat; +use lazy_static::lazy_static; +use regex::Regex; +use std::collections::VecDeque; +use std::sync::Arc; +use std::{fs::OpenOptions, io::Write, path::Path}; +use tokio::sync::{Mutex, mpsc}; +use tokio::time::{Duration, Instant}; +use tracing::{debug, error, info, warn}; + +// ── Constants ──────────────────────────────────────────────────────────────── + +const DEFAULT_CHUNK_DURATION_SEC: f32 = 15.0; +const DEFAULT_OVERLAP_RATIO: f32 = 0.25; // 25% overlap for context +const DEFAULT_BUFFER_DELAY_MS: u64 = 3000; +const DEFAULT_TYPING_CPS: f32 = 30.0; +const DEFAULT_EMIT_WORDS_MAX: usize = 3; + +lazy_static! { + static ref TOKEN_RE: Regex = Regex::new(r"\s+|\S+\s*").expect("token regex"); +} + +// ── Public type alias ──────────────────────────────────────────────────────── + +use crate::pipeline::contracts::{DeltaSink, TranscriptDelta}; + +/// Legacy alias — now backed by `DeltaSink` trait instead of bare `Fn(&str)`. +/// Consumers should migrate to `Arc` directly. +#[deprecated(note = "Use Arc directly")] +pub type StreamDeltaCallback = Arc; + +// ── Hallucination filter ───────────────────────────────────────────────────── + +const WHISPER_HALLUCINATIONS: &[&str] = &[ + "thank you", + "thanks for watching", + "thanks for listening", + "dziękuję za uwagę", + "do zobaczenia", + "subscribe", + "like and subscribe", + ".com", + "codescribe", + "www.", +]; + +const SHORT_SPEECH_WHITELIST: &[&str] = &[ + "tak", "nie", "co?", "co", "dobra", "dobrze", "ok", "okej", "no", "no?", "mhm", "aha", "jasne", + "pewnie", "super", "hej", "halo", "cześć", "siema", "dzięki", "proszę", +]; + +pub(crate) fn is_hallucination(text: &str) -> bool { + let lower = text.trim().to_lowercase(); + if SHORT_SPEECH_WHITELIST.iter().any(|w| lower == *w) { + return false; + } + if WHISPER_HALLUCINATIONS.iter().any(|h| lower == *h) { + return true; + } + if lower.len() < 30 + && WHISPER_HALLUCINATIONS.iter().any(|h| lower.contains(h)) + && lower.split_whitespace().count() <= 4 + { + return true; + } + false +} + +// ── TranscriptionPipeline ──────────────────────────────────────────────────── + +pub(crate) struct TranscriptionPipeline { + pub(crate) language: Option, + pub(crate) postprocessor: LexiconPostProcessor, + pub(crate) last_suffix: String, + pub(crate) hallucination_drops: u64, + pub(crate) overlap_strips: u64, +} + +impl TranscriptionPipeline { + pub(crate) fn new(language: Option) -> Self { + Self { + language, + postprocessor: LexiconPostProcessor::new(), + last_suffix: String::new(), + hallucination_drops: 0, + overlap_strips: 0, + } + } + + pub(crate) fn strip_overlap(&mut self, text: &str) -> String { + strip_suffix_overlap(&self.last_suffix, text) + } + + pub(crate) fn postprocess(&mut self, text: &str) -> Option { + if is_hallucination(text) { + self.hallucination_drops += 1; + return None; + } + + let stripped = self.strip_overlap(text); + if stripped.is_empty() { + self.overlap_strips += 1; + return None; + } + + let processed = self.postprocessor.process(&stripped)?; + + let suffix_len = 50; + let start = processed.len().saturating_sub(suffix_len); + self.last_suffix = processed[start..].to_string(); + + Some(processed) + } +} + +// ── BufferedEmitter ────────────────────────────────────────────────────────── + +pub(crate) struct BufferedEmitter { + queue: VecDeque, + initial_delay_ms: u64, + typing_speed_cps: f32, + emit_words_max: usize, + first_output_at: Option, + current_segment: Option, + current_tokens: Vec, + current_token_index: usize, + delta_callback: Option>, + transcript_buffer: Arc>, + stream_log_path: Option, + finished: bool, + has_output: bool, + emitted_text: String, + correction_pending: Option, + last_correction_at: Option, + pub(crate) corrections_applied: u64, +} + +impl BufferedEmitter { + pub(crate) fn new( + transcript_buffer: Arc>, + delta_callback: Option>, + stream_log_path: Option, + ) -> Self { + Self { + queue: VecDeque::new(), + initial_delay_ms: env_u64("CODESCRIBE_BUFFER_DELAY_MS", DEFAULT_BUFFER_DELAY_MS), + typing_speed_cps: env_f32("CODESCRIBE_TYPING_CPS", DEFAULT_TYPING_CPS).max(5.0), + emit_words_max: env_usize("CODESCRIBE_EMIT_WORDS_MAX", DEFAULT_EMIT_WORDS_MAX) + .clamp(1, 10), + first_output_at: None, + current_segment: None, + current_tokens: Vec::new(), + current_token_index: 0, + delta_callback, + transcript_buffer, + stream_log_path, + finished: false, + has_output: false, + emitted_text: String::new(), + correction_pending: None, + last_correction_at: None, + corrections_applied: 0, + } + } + + pub(crate) fn push_correction(&mut self, corrected: String) { + if self.emitted_text.is_empty() { + return; + } + // Guard: reject corrections that would rewrite most of the text. + // Common-prefix must cover >= 70% of the shorter string. + let prefix_len = self + .emitted_text + .chars() + .zip(corrected.chars()) + .take_while(|(a, b)| a == b) + .count(); + let min_len = self + .emitted_text + .chars() + .count() + .min(corrected.chars().count()); + if min_len > 0 && (prefix_len as f64 / min_len as f64) < 0.70 { + debug!( + "Correction rejected: common prefix {}/{} ({:.0}%) < 70%", + prefix_len, + min_len, + prefix_len as f64 / min_len as f64 * 100.0, + ); + return; + } + self.correction_pending = Some(corrected); + } + + pub(crate) fn push_segment(&mut self, text: String) { + if text.trim().is_empty() { + return; + } + let mut segment = text; + if !segment.starts_with(char::is_whitespace) + && (self.has_output || self.current_segment.is_some() || !self.queue.is_empty()) + { + segment = format!(" {}", segment); + } + self.queue.push_back(segment); + if self.first_output_at.is_none() { + self.first_output_at = Some(Instant::now()); + } + } + + pub(crate) async fn tick(&mut self) -> bool { + if self.finished && self.queue.is_empty() && self.current_segment.is_none() { + return true; + } + + if self.is_buffering() { + return false; + } + + const CORRECTION_COOLDOWN_MS: u64 = 500; + if let Some(ref _corrected) = self.correction_pending { + let can_correct = self + .last_correction_at + .map(|t| t.elapsed() >= Duration::from_millis(CORRECTION_COOLDOWN_MS)) + .unwrap_or(true); + if can_correct + && let Some(corrected) = self.correction_pending.take() + && let Some(delta) = build_redacted_delta(&self.emitted_text, &corrected) + { + apply_delta_to_string(&mut self.emitted_text, &delta); + { + let mut buffer = self.transcript_buffer.lock().await; + *buffer = self.emitted_text.clone(); + } + if let Some(sink) = &self.delta_callback { + sink.apply(&TranscriptDelta::from_raw(&delta)); + } + if let Some(path) = self.stream_log_path.as_deref() { + let _ = append_to_stream_log(path, &delta); + } + self.last_correction_at = Some(Instant::now()); + self.corrections_applied += 1; + return false; + } + } + + if self.current_segment.is_none() { + self.current_segment = self.queue.pop_front(); + self.current_tokens = self + .current_segment + .as_deref() + .map(tokenize_for_emit) + .unwrap_or_default(); + self.current_token_index = 0; + } + + if let Some(delta) = self.next_emit_chunk() { + self.has_output = true; + self.emitted_text.push_str(&delta); + { + let mut buffer = self.transcript_buffer.lock().await; + apply_delta_to_string(&mut buffer, &delta); + } + + if let Some(sink) = &self.delta_callback { + sink.apply(&TranscriptDelta::from_raw(&delta)); + } + + if let Some(path) = self.stream_log_path.as_deref() { + let _ = append_to_stream_log(path, &delta); + } + } + + self.finished && self.queue.is_empty() && self.current_segment.is_none() + } + + fn is_buffering(&self) -> bool { + let Some(start) = self.first_output_at else { + return true; + }; + start.elapsed() < Duration::from_millis(self.initial_delay_ms) + } + + pub(crate) fn finish(&mut self) { + self.finished = true; + } + + fn next_emit_chunk(&mut self) -> Option { + let _ = self.current_segment.as_ref()?; + if self.current_token_index >= self.current_tokens.len() { + self.current_segment = None; + self.current_tokens.clear(); + self.current_token_index = 0; + return None; + } + + let mut chunk = String::new(); + let mut words = 0usize; + + while self.current_token_index < self.current_tokens.len() { + let token = &self.current_tokens[self.current_token_index]; + chunk.push_str(token); + if token.chars().any(|c| !c.is_whitespace()) { + words += 1; + } + self.current_token_index += 1; + + if words >= self.emit_words_max { + if self.current_token_index < self.current_tokens.len() { + let next = &self.current_tokens[self.current_token_index]; + if next.chars().all(|c| c.is_whitespace()) { + chunk.push_str(next); + self.current_token_index += 1; + } + } + break; + } + } + + if self.current_token_index >= self.current_tokens.len() { + self.current_segment = None; + self.current_tokens.clear(); + self.current_token_index = 0; + } + + if chunk.is_empty() { None } else { Some(chunk) } + } +} + +// ── Emitter tick loop ──────────────────────────────────────────────────────── + +pub(crate) async fn emitter_tick_loop(emitter: Arc>) { + let interval = { + let guard = emitter.lock().await; + Duration::from_secs_f32(1.0 / guard.typing_speed_cps) + }; + let mut ticker = tokio::time::interval(interval); + + loop { + ticker.tick().await; + let should_stop = { + let mut guard = emitter.lock().await; + guard.tick().await + }; + if should_stop { + break; + } + } +} + +// ── Worker functions ───────────────────────────────────────────────────────── + +pub(crate) async fn transcription_worker( + mut chunk_receiver: mpsc::Receiver>, + transcript_buffer: Arc>, + sample_rate: u32, + language: Option, + mut postprocessor: Option, + delta_callback: Option>, + stream_log_path: Option, +) { + info!("Transcription worker started"); + + let chunk_duration_sec = stream_chunk_duration_sec(); + let overlap_sec = stream_overlap_sec(chunk_duration_sec); + let mut session = SpeechSession::new_stream(sample_rate, chunk_duration_sec, overlap_sec); + + while let Some(data) = chunk_receiver.recv().await { + for event in session.feed(&data, sample_rate) { + if let SpeechEvent::Chunk(samples) = event { + process_chunk( + &samples, + &transcript_buffer, + session.output_sample_rate(), + language.as_deref(), + postprocessor.as_mut(), + delta_callback.as_ref(), + stream_log_path.as_deref(), + ) + .await; + } + } + } + + if let Some(SpeechEvent::Chunk(samples)) = session.flush() { + debug!("Processing final chunk ({} samples)", samples.len()); + process_chunk( + &samples, + &transcript_buffer, + session.output_sample_rate(), + language.as_deref(), + postprocessor.as_mut(), + delta_callback.as_ref(), + stream_log_path.as_deref(), + ) + .await; + } + + info!("Transcription worker finished"); +} + +pub(crate) async fn buffered_transcription_worker( + mut chunk_receiver: mpsc::Receiver>, + transcript_buffer: Arc>, + sample_rate: u32, + language: Option, + delta_callback: Option>, + stream_log_path: Option, +) { + info!("Buffered transcription worker started"); + + let mut session = SpeechSession::new_utterance(sample_rate); + let mut pipeline = TranscriptionPipeline::new(language); + let emitter = Arc::new(Mutex::new(BufferedEmitter::new( + transcript_buffer.clone(), + delta_callback, + stream_log_path, + ))); + + let emitter_handle = tokio::spawn(emitter_tick_loop(emitter.clone())); + + let mut correction_audio_buf: Vec = Vec::new(); + let mut utterance_count: usize = 0; + let mut suffix_snapshot = String::new(); + + while let Some(data) = chunk_receiver.recv().await { + for event in session.feed(&data, sample_rate) { + if let SpeechEvent::Utterance(utterance) = event { + if utterance_count == 0 && correction_audio_buf.is_empty() { + suffix_snapshot = pipeline.last_suffix.clone(); + } + + let audio_copy = utterance.clone(); + let result = handle_utterance( + utterance, + session.output_sample_rate(), + &mut pipeline, + &emitter, + ) + .await; + + if let Err(e) = result { + error!("Buffered transcription failed: {}", e); + continue; + } + + correction_audio_buf.extend_from_slice(&audio_copy); + utterance_count += 1; + + let audio_duration_s = correction_audio_buf.len() as f32 / sample_rate as f32; + if utterance_count >= 3 || audio_duration_s >= 10.0 { + let audio = std::mem::take(&mut correction_audio_buf); + let lang = pipeline.language.clone(); + + let current_suffix = pipeline.last_suffix.clone(); + pipeline.last_suffix = suffix_snapshot.clone(); + + let re_text = tokio::task::spawn_blocking(move || { + whisper::transcribe(&audio, sample_rate, lang.as_deref()) + }) + .await; + + match re_text { + Ok(Ok(raw)) => { + if let Some(cleaned) = pipeline.postprocess(&raw) { + let mut guard = emitter.lock().await; + guard.push_correction(cleaned); + // postprocess() already updated last_suffix to match + // the re-transcribed text — no restore needed. + } else { + // Re-transcription was empty/filtered — restore suffix + // so the next utterance deduplicates against the Phase-1 draft. + pipeline.last_suffix = current_suffix; + } + } + _ => { + warn!("Re-transcription failed; keeping Phase 1 draft"); + pipeline.last_suffix = current_suffix; + } + } + + utterance_count = 0; + } + } + } + } + + if let Some(SpeechEvent::Utterance(utterance)) = session.flush() + && let Err(e) = handle_utterance( + utterance, + session.output_sample_rate(), + &mut pipeline, + &emitter, + ) + .await + { + error!("Final buffered transcription failed: {}", e); + } + + { + let mut guard = emitter.lock().await; + guard.finish(); + info!( + "Stream Session Stats: Hallucinations dropped: {}, Overlaps stripped: {}, Corrections applied: {}", + pipeline.hallucination_drops, pipeline.overlap_strips, guard.corrections_applied + ); + } + + if let Err(e) = emitter_handle.await { + error!("Buffered emitter task failed: {}", e); + } + + info!("Buffered transcription worker finished"); +} + +async fn handle_utterance( + utterance: Vec, + sample_rate: u32, + pipeline: &mut TranscriptionPipeline, + emitter: &Arc>, +) -> Result> { + if utterance.is_empty() { + return Ok(None); + } + + let language = pipeline.language.clone(); + let raw_text = tokio::task::spawn_blocking(move || { + whisper::transcribe(&utterance, sample_rate, language.as_deref()) + }) + .await??; + + if let Some(cleaned) = pipeline.postprocess(&raw_text) { + let mut guard = emitter.lock().await; + guard.push_segment(cleaned.clone()); + return Ok(Some(cleaned)); + } + + Ok(None) +} + +async fn process_chunk( + samples: &[f32], + transcript_buffer: &Arc>, + sample_rate: u32, + language: Option<&str>, + mut postprocessor: Option<&mut StreamPostProcessor>, + delta_callback: Option<&Arc>, + stream_log_path: Option<&Path>, +) { + if samples.is_empty() { + return; + } + + let samples_owned = samples.to_vec(); + let lang_owned = language.map(String::from); + + let result = tokio::task::spawn_blocking(move || { + let engine_mutex = match get_engine() { + Ok(m) => m, + Err(e) => return Err(anyhow!("Engine error: {}", e)), + }; + + let mut engine_guard = match engine_mutex.lock() { + Ok(g) => g, + Err(e) => return Err(anyhow!("Lock error: {}", e)), + }; + + engine_guard.transcribe_with_language(&samples_owned, sample_rate, lang_owned.as_deref()) + }) + .await; + + match result { + Ok(Ok(text)) => { + if !text.trim().is_empty() { + debug!("Chunk transcribed: '{}'", text.trim()); + let cleaned = if let Some(processor) = postprocessor.as_mut() { + processor.process(&text) + } else { + let cleaned = crate::pipeline::stream_postprocess::normalize_whitespace(&text); + if cleaned.trim().is_empty() { + None + } else { + Some(cleaned) + } + }; + + if let Some(cleaned) = cleaned { + let mut buffer = transcript_buffer.lock().await; + let before = buffer.clone(); + dedup_chunk_overlap(&mut buffer, &cleaned); + if let Some(delta) = build_redacted_delta(&before, &buffer) { + let has_effect = + delta.chars().any(|c| c == '\u{0008}' || !c.is_whitespace()); + if has_effect { + if let Some(sink) = delta_callback { + sink.apply(&TranscriptDelta::from_raw(&delta)); + } + if let Some(path) = stream_log_path { + let _ = append_to_stream_log(path, &delta); + } + } + } + } else { + debug!("Stream postprocessor dropped chunk"); + } + } + } + Ok(Err(e)) => { + error!("Chunk transcription failed: {}", e); + } + Err(e) => { + error!("Transcription task join error: {}", e); + } + } +} + +// ── Public: batch streaming transcription ──────────────────────────────────── + +pub fn transcribe_streaming_samples( + samples: &[f32], + sample_rate: u32, + language: Option<&str>, + mut postprocessor: Option<&mut StreamPostProcessor>, +) -> Result { + if samples.is_empty() { + return Ok(String::new()); + } + + let chunk_duration_sec = stream_chunk_duration_sec(); + let overlap_sec = stream_overlap_sec(chunk_duration_sec); + let chunk_limit = (sample_rate as f32 * chunk_duration_sec) as usize; + let overlap_size = (sample_rate as f32 * overlap_sec) as usize; + let step = chunk_limit.saturating_sub(overlap_size).max(1); + + let total_audio_sec = samples.len() as f32 / sample_rate as f32; + let stride_sec = chunk_duration_sec - overlap_sec; + let n_chunks = + ((samples.len().saturating_sub(chunk_limit)) as f32 / step as f32).ceil() as usize + 1; + let processing_factor = chunk_duration_sec / stride_sec; + let effective_audio_sec = n_chunks as f32 * chunk_duration_sec; + + info!( + "[STREAM_DIAG] chunk={:.1}s overlap={:.1}s stride={:.1}s | audio={:.1}s chunks={} factor={:.2}x effective={:.1}s", + chunk_duration_sec, + overlap_sec, + stride_sec, + total_audio_sec, + n_chunks, + processing_factor, + effective_audio_sec + ); + + let engine_mutex = get_engine()?; + let mut engine = engine_mutex + .lock() + .map_err(|e| anyhow!("Lock error: {}", e))?; + + let mut out = String::new(); + let mut offset = 0usize; + let mut chunks_processed = 0usize; + let t_start = std::time::Instant::now(); + + while offset < samples.len() { + let end = (offset + chunk_limit).min(samples.len()); + let chunk = &samples[offset..end]; + let chunk_sec = chunk.len() as f32 / sample_rate as f32; + let t_chunk = std::time::Instant::now(); + let text = engine.transcribe_with_language(chunk, sample_rate, language)?; + let chunk_ms = t_chunk.elapsed().as_millis(); + chunks_processed += 1; + + debug!( + "[STREAM_CHUNK] #{} offset={:.1}s len={:.1}s transcribe={}ms words={}", + chunks_processed, + offset as f32 / sample_rate as f32, + chunk_sec, + chunk_ms, + text.split_whitespace().count() + ); + + if let Some(processor) = postprocessor.as_mut() { + if let Some(cleaned) = processor.process(&text) { + dedup_chunk_overlap(&mut out, &cleaned); + } + } else { + dedup_chunk_overlap(&mut out, &text); + } + + if end == samples.len() { + break; + } + offset = offset.saturating_add(step); + } + + let total_ms = t_start.elapsed().as_millis(); + info!( + "[STREAM_DONE] chunks_processed={} total_ms={} out_words={}", + chunks_processed, + total_ms, + out.split_whitespace().count() + ); + + Ok(out) +} + +// ── Delta helpers ──────────────────────────────────────────────────────────── + +pub(crate) fn build_redacted_delta(before: &str, after: &str) -> Option { + if before == after { + return None; + } + + let mut prefix_len = 0usize; + for (a, b) in before.chars().zip(after.chars()) { + if a == b { + prefix_len += a.len_utf8(); + } else { + break; + } + } + + let removed = before[prefix_len..].chars().count(); + let mut delta = String::new(); + for _ in 0..removed { + delta.push('\u{0008}'); + } + delta.push_str(&after[prefix_len..]); + Some(delta) +} + +pub(crate) fn apply_delta_to_string(target: &mut String, delta: &str) { + crate::pipeline::contracts::TranscriptDelta::from_raw(delta).apply(target); +} + +fn tokenize_for_emit(text: &str) -> Vec { + let mut tokens = Vec::new(); + for m in TOKEN_RE.find_iter(text) { + tokens.push(m.as_str().to_string()); + } + if tokens.is_empty() && !text.is_empty() { + tokens.push(text.to_string()); + } + tokens +} + +// ── Logging ────────────────────────────────────────────────────────────────── + +pub(crate) fn stream_log_path() -> Option { + if let Ok(path) = std::env::var("CODESCRIBE_STREAM_LOG_PATH") { + let trimmed = path.trim(); + if !trimmed.is_empty() { + return Some(std::path::PathBuf::from(trimmed)); + } + } + + if env_bool("CODESCRIBE_STREAM_LOG") { + let root = crate::config::Config::config_dir(); + return Some(root.join("stream.log")); + } + + None +} + +fn append_to_stream_log(path: &Path, text: &str) -> std::io::Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + + let mut file = OpenOptions::new().create(true).append(true).open(path)?; + let ts = chrono::Utc::now().to_rfc3339_opts(SecondsFormat::Millis, true); + let mut payload = text.replace('\n', "\\n").replace('\r', "\\r"); + payload = payload.replace('\u{0008}', "\\b"); + writeln!(file, "[{}] {}", ts, payload)?; + Ok(()) +} + +// ── Env helpers ────────────────────────────────────────────────────────────── + +pub(crate) fn env_bool(key: &str) -> bool { + std::env::var(key) + .ok() + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(false) +} + +pub(crate) fn env_bool_default(key: &str, default: bool) -> bool { + std::env::var(key) + .ok() + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(default) +} + +fn env_f32(key: &str, default: f32) -> f32 { + std::env::var(key) + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(default) +} + +fn env_u64(key: &str, default: u64) -> u64 { + std::env::var(key) + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(default) +} + +fn env_usize(key: &str, default: usize) -> usize { + std::env::var(key) + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(default) +} + +pub(crate) fn stream_chunk_duration_sec() -> f32 { + env_f32("CODESCRIBE_STREAM_CHUNK_SEC", DEFAULT_CHUNK_DURATION_SEC).clamp(0.5, 30.0) +} + +pub(crate) fn stream_overlap_sec(chunk_duration_sec: f32) -> f32 { + let ratio = env_f32("CODESCRIBE_STREAM_OVERLAP_RATIO", DEFAULT_OVERLAP_RATIO).clamp(0.05, 0.8); + (chunk_duration_sec * ratio).min(chunk_duration_sec * 0.8) +} + +// ── Tests ──────────────────────────────────────────────────────────────────── + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_postprocess_components() { + // Hallucination + assert!(is_hallucination("Thank you")); + assert!(is_hallucination(" Dziękuję za uwagę ")); + assert!(!is_hallucination("Tak")); // Whitelisted + assert!(!is_hallucination("This is a normal sentence.")); + + // Overlap + let mut pipeline = TranscriptionPipeline::new(None); + pipeline.last_suffix = "Alice has a cat.".to_string(); + + let res = pipeline.strip_overlap("Alice has a cat. And a dog."); + assert_eq!(res, "And a dog."); + + pipeline.last_suffix = "going to the park".to_string(); + let res = pipeline.strip_overlap("park tomorrow."); + assert_eq!(res, "tomorrow."); + + let res = pipeline.strip_overlap("Hello world"); + assert_eq!(res, "Hello world"); + } + + #[test] + fn test_suffix_preserved_when_postprocess_filters() { + // Simulates the re-transcription scenario: if postprocess returns None + // (e.g. hallucination), last_suffix must stay at the pre-snapshot value. + let mut pipeline = TranscriptionPipeline::new(None); + pipeline.last_suffix = "original suffix".to_string(); + + // "Thank you" is a hallucination — postprocess returns None + let result = pipeline.postprocess("Thank you"); + assert!(result.is_none()); + // last_suffix unchanged (strip_overlap was never reached) + assert_eq!(pipeline.last_suffix, "original suffix"); + } + + #[test] + fn test_suffix_updated_after_successful_postprocess() { + let mut pipeline = TranscriptionPipeline::new(None); + pipeline.last_suffix = "old tail".to_string(); + + let result = pipeline.postprocess("This is a brand new sentence."); + assert!(result.is_some()); + // last_suffix should now reflect the new text's suffix + assert_ne!(pipeline.last_suffix, "old tail"); + assert!(pipeline.last_suffix.contains("sentence")); + } + + #[test] + fn test_correction_guard_rejects_low_prefix() { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap(); + rt.block_on(async { + let buf = Arc::new(Mutex::new(String::new())); + let mut emitter = BufferedEmitter::new(buf, None, None); + emitter.emitted_text = "Hello world, this is a test.".to_string(); + + // Completely different text — should be rejected (<70% prefix) + emitter.push_correction("Goodbye universe, nothing alike.".to_string()); + assert!(emitter.correction_pending.is_none()); + + // Similar text with minor tail fix — should be accepted (>70% prefix) + emitter.push_correction("Hello world, this is a test!".to_string()); + assert!(emitter.correction_pending.is_some()); + }); + } + + #[test] + fn test_correction_delta() { + let before = "This is a dratf."; + let after = "This is a draft."; + let delta = build_redacted_delta(before, after).expect("should produce delta"); + + assert!(delta.contains("\u{0008}\u{0008}\u{0008}")); + assert!(delta.ends_with("ft.")); + + let mut target = before.to_string(); + apply_delta_to_string(&mut target, &delta); + assert_eq!(target, after); + } +} diff --git a/core/pipeline/tests.rs b/core/pipeline/tests.rs new file mode 100644 index 00000000..31d3de48 --- /dev/null +++ b/core/pipeline/tests.rs @@ -0,0 +1,139 @@ +/// Integration tests for the unified pipeline components. +/// +/// These exercise dedup + sinks together, simulating multi-chunk and +/// multi-utterance scenarios without needing a real Whisper engine. +#[cfg(test)] +mod pipeline_integration { + use crate::pipeline::contracts::DeltaSink; + use crate::pipeline::contracts::TranscriptDelta; + use crate::pipeline::dedup::{dedup_chunk_overlap, strip_suffix_overlap}; + use crate::pipeline::sinks::{CallbackSink, CollectorSink}; + use std::sync::Arc; + + // ── Test 1: Pipeline roundtrip, no dedup needed ────────────── + + #[test] + fn test_pipeline_roundtrip_no_dedup() { + let collector = Arc::new(CollectorSink::new()); + let sink: Arc = collector.clone(); + + // Simulate: single chunk → delta → sink + let mut buffer = String::new(); + let chunk = "Hello world."; + dedup_chunk_overlap(&mut buffer, chunk); + + let delta = TranscriptDelta::from_raw(&buffer); + sink.apply(&delta); + + assert_eq!(buffer, "Hello world."); + assert_eq!(collector.collected(), vec!["Hello world."]); + } + + // ── Test 2: Dedup overlapping chunks ───────────────────────── + + #[test] + fn test_pipeline_dedup_overlapping_chunks() { + let mut buffer = String::new(); + + // Chunk 1 + dedup_chunk_overlap(&mut buffer, "Hello world this is"); + // Chunk 2 overlaps: "this is" appears at end of chunk 1 and start of chunk 2 + dedup_chunk_overlap(&mut buffer, "this is a test"); + + assert_eq!(buffer, "Hello world this is a test"); + } + + // ── Test 3: Suffix dedup across utterances ─────────────────── + + #[test] + fn test_pipeline_suffix_dedup_utterances() { + let mut transcript = String::new(); + // Utterance 1 + let utt1 = "Hello world."; + transcript.push_str(utt1); + let last_suffix = utt1; + + // Utterance 2 — overlaps with end of utterance 1 + let utt2 = "world. And more."; + let stripped = strip_suffix_overlap(last_suffix, utt2); + if !stripped.is_empty() { + if !transcript.ends_with(' ') && !stripped.starts_with(' ') { + transcript.push(' '); + } + transcript.push_str(&stripped); + } + + assert_eq!(transcript, "Hello world. And more."); + } + + // ── Test 4: Delta accumulation equals transcript ───────────── + + #[test] + fn test_delta_accumulation_equals_transcript() { + let collector = Arc::new(CollectorSink::new()); + let sink: Arc = collector.clone(); + + let chunks = ["Cześć, ", "jestem ", "weterynarzem."]; + let mut buffer = String::new(); + + for chunk in &chunks { + let before = buffer.clone(); + dedup_chunk_overlap(&mut buffer, chunk); + // Build delta as difference + let delta_str = &buffer[before.len()..]; + if !delta_str.is_empty() { + sink.apply(&TranscriptDelta::from_raw(delta_str)); + } + } + + // Reassemble from deltas + let reassembled: String = collector.collected().join(""); + assert_eq!(buffer, "Cześć, jestem weterynarzem."); + assert_eq!(reassembled, buffer); + } + + // ── Test 5: CallbackSink bridges to Fn(&str) ──────────────── + + #[test] + fn test_callback_sink_integration() { + let received = Arc::new(std::sync::Mutex::new(Vec::new())); + let r = received.clone(); + let sink: Arc = Arc::new(CallbackSink::new(Arc::new(move |s: &str| { + r.lock().unwrap().push(s.to_string()); + }))); + + sink.apply(&TranscriptDelta::from_raw("Hello")); + sink.apply(&TranscriptDelta::from_raw(" world")); + + let result = received.lock().unwrap(); + assert_eq!(*result, vec!["Hello", " world"]); + } + + // ── Test 6: Multi-chunk fuzzy dedup end-to-end ─────────────── + + #[test] + fn test_pipeline_fuzzy_dedup_chain() { + let mut buffer = String::new(); + + // 3 chunks with overlapping regions, some fuzzy + dedup_chunk_overlap(&mut buffer, "The quick brown fox"); + dedup_chunk_overlap(&mut buffer, "brown fox jumps over"); + dedup_chunk_overlap(&mut buffer, "jumps over the lazy dog"); + + assert_eq!(buffer, "The quick brown fox jumps over the lazy dog"); + } + + // ── Test 7: Empty and whitespace-only chunks ───────────────── + + #[test] + fn test_pipeline_empty_chunks_ignored() { + let mut buffer = String::new(); + + dedup_chunk_overlap(&mut buffer, "Hello"); + dedup_chunk_overlap(&mut buffer, ""); + dedup_chunk_overlap(&mut buffer, " "); + dedup_chunk_overlap(&mut buffer, "world"); + + assert_eq!(buffer, "Hello world"); + } +} diff --git a/core/quality/quality_report.rs b/core/quality/quality_report.rs index 4196aaf2..a4943305 100644 --- a/core/quality/quality_report.rs +++ b/core/quality/quality_report.rs @@ -1315,7 +1315,7 @@ fn ensure_model_path() -> Result<()> { } Err(anyhow!( - "Missing local model. Set CODESCRIBE_MODEL_PATH or download via ./scripts/download-model.sh" + "Missing local model. Set CODESCRIBE_MODEL_PATH or run: hf download LibraxisAI/whisper-large-v3-turbo-mlx-q8" )) } diff --git a/core/stt/adapter.rs b/core/stt/adapter.rs new file mode 100644 index 00000000..6d36138a --- /dev/null +++ b/core/stt/adapter.rs @@ -0,0 +1,111 @@ +//! STT adapter — bridges the Whisper singleton to the `TranscriptionAdapter` contract. +//! +//! This is a thin wrapper: no logic changes, just type translation. +//! Future providers (cloud STT, etc.) would implement the same trait. +//! +//! Created by M&K (c)2026 VetCoders + +use anyhow::Result; + +use crate::pipeline::contracts::{RawTranscript, SpeechUtterance, TranscriptionAdapter}; + +/// Adapter wrapping the global Whisper singleton. +/// +/// Uses `whisper::transcribe(samples, sample_rate, language)` under the hood. +/// Thread-safe: the singleton uses `OnceLock>`. +pub struct WhisperSingletonAdapter; + +impl Default for WhisperSingletonAdapter { + fn default() -> Self { + Self + } +} + +impl WhisperSingletonAdapter { + pub fn new() -> Self { + Self + } +} + +impl TranscriptionAdapter for WhisperSingletonAdapter { + fn transcribe( + &self, + utterance: &SpeechUtterance, + language: Option<&str>, + ) -> Result { + let text = + crate::stt::whisper::transcribe(&utterance.samples, utterance.sample_rate, language)?; + Ok(RawTranscript { + text, + segments: Vec::new(), // Whisper singleton doesn't expose segments yet + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// Verify the adapter satisfies Send + Sync (required by trait bound). + #[test] + fn adapter_is_send_sync() { + fn assert_send_sync() {} + assert_send_sync::(); + } + + /// Verify construction doesn't panic. + #[test] + fn adapter_construction() { + let _adapter = WhisperSingletonAdapter::new(); + } + + /// Test with a mock: any struct implementing TranscriptionAdapter works. + struct MockAdapter { + response: String, + } + + impl TranscriptionAdapter for MockAdapter { + fn transcribe( + &self, + _utterance: &SpeechUtterance, + _language: Option<&str>, + ) -> Result { + Ok(RawTranscript { + text: self.response.clone(), + segments: Vec::new(), + }) + } + } + + #[test] + fn mock_adapter_returns_text() { + let adapter = MockAdapter { + response: "Cześć, to jest test".into(), + }; + let utterance = SpeechUtterance { + samples: vec![0.0; 16000], + sample_rate: 16000, + start_ts: 0.0, + end_ts: 1.0, + }; + let result = adapter.transcribe(&utterance, Some("pl")).unwrap(); + assert_eq!(result.text, "Cześć, to jest test"); + assert!(result.segments.is_empty()); + } + + /// Verify trait object works (dyn dispatch). + #[test] + fn adapter_as_trait_object() { + let adapter: Box = Box::new(MockAdapter { + response: "hello".into(), + }); + let utterance = SpeechUtterance { + samples: vec![0.0; 8000], + sample_rate: 16000, + start_ts: 0.0, + end_ts: 0.5, + }; + let result = adapter.transcribe(&utterance, None).unwrap(); + assert_eq!(result.text, "hello"); + } +} diff --git a/core/stt/mod.rs b/core/stt/mod.rs index e658f364..5056cc48 100644 --- a/core/stt/mod.rs +++ b/core/stt/mod.rs @@ -1 +1,2 @@ +pub mod adapter; pub mod whisper; diff --git a/core/stt/whisper/embedded.rs b/core/stt/whisper/embedded.rs index 6e1d678b..0f5ed509 100644 --- a/core/stt/whisper/embedded.rs +++ b/core/stt/whisper/embedded.rs @@ -21,14 +21,15 @@ mod data { } /// Check if embedded model is available +/// +/// Note: We only check weights_size, not cfg!(embed_model). +/// The cfg!() macro can return false in workspace builds even when +/// the data was correctly embedded via #[cfg(embed_model)]. +/// If weights exist (len > 0), the model is available. pub fn is_embedded_available() -> bool { - let cfg_set = cfg!(embed_model); let weights_size = data::WEIGHTS.len(); - eprintln!( - "[DEBUG] Embedded check: cfg={}, weights_size={}", - cfg_set, weights_size - ); - cfg_set && weights_size > 0 + tracing::debug!(weights_size, "Embedded model check"); + weights_size > 0 } /// Get embedded model data if available diff --git a/core/stt/whisper/engine.rs b/core/stt/whisper/engine.rs index 34bf364b..7edf7389 100644 --- a/core/stt/whisper/engine.rs +++ b/core/stt/whisper/engine.rs @@ -51,7 +51,17 @@ impl LocalWhisperEngine { tracing::debug!("LocalWhisperEngine using device: {:?}", device); let config_path = model_path.join("config.json"); - let weights_path = model_path.join("weights.safetensors"); + let weights_path = if model_path.join("weights.safetensors").exists() { + model_path.join("weights.safetensors") + } else { + model_path.join("model.safetensors") + }; + if !weights_path.exists() { + anyhow::bail!( + "Whisper weights not found (expected weights.safetensors or model.safetensors) in {}", + model_path.display() + ); + } let tokenizer_path = model_path.join("tokenizer.json"); let mel_filters_path = model_path.join("mel_filters.npz"); @@ -558,6 +568,21 @@ impl LocalWhisperEngine { tokens.push(t); } + // Initial prompt: tokenize and prepend to decoder context (helps with vocabulary/formatting) + if let Some(ref prompt) = self.decoding_params.initial_prompt + && let Ok(encoding) = self.tokenizer.encode(prompt.as_str(), false) + { + let prompt_tokens: Vec = encoding.get_ids().to_vec(); + if !prompt_tokens.is_empty() { + tracing::debug!( + "Initial prompt: {} ({} tokens)", + prompt, + prompt_tokens.len() + ); + tokens.extend(prompt_tokens); + } + } + let mut all_tokens = Vec::new(); // Run encoder once diff --git a/core/stt/whisper/params.rs b/core/stt/whisper/params.rs index 20bf9ae8..222c7749 100644 --- a/core/stt/whisper/params.rs +++ b/core/stt/whisper/params.rs @@ -10,7 +10,7 @@ pub struct DecodingParams { /// mlx_whisper default: 0 pub temperature: f32, /// Prevent repetitions of n-grams with this size (0 = disabled) - /// faster-whisper default: 3 + /// Size 5 catches more variants than default 3 (e.g., "jest." vs "jest") pub no_repeat_ngram_size: usize, /// Suppress blank/silence tokens early pub suppress_blank: bool, @@ -24,17 +24,33 @@ pub struct DecodingParams { /// Log probability threshold - if avg logprob < this, decoding failed /// mlx_whisper default: -1.0 pub logprob_threshold: f32, + /// Initial prompt to guide transcription (domain vocabulary, formatting hints) + /// Tokenized and prepended to decoder context after special tokens + /// Env: WHISPER_INITIAL_PROMPT + pub initial_prompt: Option, } impl Default for DecodingParams { fn default() -> Self { Self { - temperature: 0.0, // greedy (mlx_whisper default) - no_repeat_ngram_size: 4, // block 4-gram repetitions (increased for better coverage) + temperature: 0.0, // greedy (mlx_whisper default) + // Block 5-gram repetitions during decoding (preventive, not reactive) + // Size 5 catches more repetition variants than default 3: + // - "jest." vs "jest" variations + // - longer phrase loops like "w tej chwili w tej chwili" + // faster-whisper/whisper.cpp often use 4-5 for better quality + no_repeat_ngram_size: 5, suppress_blank: true, - no_speech_threshold: 0.6, // mlx_whisper default - compression_ratio_threshold: 2.4, // mlx_whisper default - logprob_threshold: -1.0, // mlx_whisper default + // Stricter thresholds (aligned with faster-whisper / API): + // - Reduces hallucinations by rejecting low-quality decodings + // - Matches lbrx-services/stt-engine defaults for consistency + no_speech_threshold: 0.5, // was 0.6 - stricter silence detection + compression_ratio_threshold: 2.0, // was 2.4 - stricter hallucination detection + logprob_threshold: -0.5, // was -1.0 - reject low-confidence output + // Initial prompt from env - helps with domain vocabulary and formatting + initial_prompt: std::env::var("WHISPER_INITIAL_PROMPT") + .ok() + .filter(|s| !s.is_empty()), } } } diff --git a/core/stt/whisper/singleton.rs b/core/stt/whisper/singleton.rs index c3fe05d0..c3a2289c 100644 --- a/core/stt/whisper/singleton.rs +++ b/core/stt/whisper/singleton.rs @@ -17,12 +17,14 @@ use tracing::{info, warn}; use crate::config::Config; use crate::config::models::ModelManager; +use crate::hf_cache; use super::engine::LocalWhisperEngine; use super::params::DecodingParams; /// Default model name (for dev/fallback mode) pub const DEFAULT_MODEL: &str = "whisper-large-v3-turbo-mlx-q8"; +const DEFAULT_WHISPER_REPO: &str = "LibraxisAI/whisper-large-v3-turbo-mlx-q8"; /// Global singleton engine static ENGINE: OnceLock> = OnceLock::new(); @@ -50,6 +52,27 @@ fn resolve_model_path_fallback() -> Result { // 2. Configured model (LOCAL_MODEL) let config = Config::load(); let configured_model = config.local_model; + if !configured_model.trim().is_empty() { + if configured_model.contains('/') { + if let Some(snapshot) = hf_cache::find_snapshot_with_any( + configured_model.trim(), + &["config.json", "tokenizer.json", "mel_filters.npz"], + &["weights.safetensors", "model.safetensors"], + ) { + info!("Using HF cache model: {}", snapshot.display()); + return Ok(snapshot); + } + } else if configured_model == DEFAULT_MODEL + && let Some(snapshot) = hf_cache::find_snapshot_with_any( + DEFAULT_WHISPER_REPO, + &["config.json", "tokenizer.json", "mel_filters.npz"], + &["weights.safetensors", "model.safetensors"], + ) + { + info!("Using HF cache model: {}", snapshot.display()); + return Ok(snapshot); + } + } if !configured_model.trim().is_empty() && let Ok(manager) = ModelManager::new() { @@ -76,7 +99,8 @@ fn resolve_model_path_fallback() -> Result { "Whisper model not available.\n\ Debug builds: Set CODESCRIBE_MODEL_PATH\n\ Release builds: Model should be embedded\n\n\ - Download with: ./scripts/download-model.sh" + Download with: hf download {}", + DEFAULT_WHISPER_REPO )) } diff --git a/core/tts/embedded.rs b/core/tts/embedded.rs index 7e0a05ae..78ded94e 100644 --- a/core/tts/embedded.rs +++ b/core/tts/embedded.rs @@ -23,15 +23,15 @@ mod data { } /// Check if embedded model is available +/// +/// Note: We only check weights_size, not cfg!(embed_tts). +/// The cfg!() macro can return false in workspace builds even when +/// the data was correctly embedded via #[cfg(embed_tts)]. +/// If weights exist (len > 0), the model is available. pub fn is_embedded_available() -> bool { - let cfg_set = cfg!(embed_tts); let weights_size = data::WEIGHTS.len(); - tracing::debug!( - "[TTS] Embedded check: cfg={}, weights_size={}", - cfg_set, - weights_size - ); - cfg_set && weights_size > 0 + tracing::debug!("[TTS] Embedded check: weights_size={}", weights_size); + weights_size > 0 } /// Get embedded model data if available diff --git a/core/tts/engine.rs b/core/tts/engine.rs index d1d7d192..2aad4c57 100644 --- a/core/tts/engine.rs +++ b/core/tts/engine.rs @@ -22,7 +22,7 @@ use tokenizers::Tokenizer; use tracing::{debug, info}; use super::embedded::EmbeddedTts; -use crate::safe_path; +use crate::{hf_cache, safe_path}; /// Default CSM model output sample rate const SAMPLE_RATE: u32 = 24000; @@ -85,21 +85,38 @@ impl TtsEngine { }; let csm = CsmModel::new(&config, vb).context("Failed to create CSM model")?; - // Load Mimi codec with default v0.1 config + // Load Mimi codec with config matching CSM's audio_num_codebooks + // NOTE: Mimi MUST use F32 - Metal conv1d doesn't support BF16 + // NOTE: num_codebooks MUST match CSM config (32 for csm-1b) let mimi_weights_path = model_path.join("mimi.safetensors"); - let mimi_config = MimiConfig::v0_1(None); + let mimi_num_codebooks = config.audio_num_codebooks; + let mimi_config = MimiConfig::v0_1(Some(mimi_num_codebooks)); + let mimi_dtype = DType::F32; // Mimi codec requires F32 for Metal conv1d + debug!( + "Mimi config: num_codebooks={} (from CSM audio_num_codebooks)", + mimi_num_codebooks + ); let mimi = if mimi_weights_path.exists() { let mimi_vb = unsafe { - VarBuilder::from_mmaped_safetensors(&[&mimi_weights_path], dtype, &device) + VarBuilder::from_mmaped_safetensors(&[&mimi_weights_path], mimi_dtype, &device) + .context("Failed to load Mimi weights")? + }; + MimiModel::new(mimi_config, mimi_vb).context("Failed to create Mimi model")? + } else if let Some(snapshot) = + hf_cache::find_snapshot("kyutai/mimi", &["model.safetensors"]) + { + let cached_path = snapshot.join("model.safetensors"); + let mimi_vb = unsafe { + VarBuilder::from_mmaped_safetensors(&[&cached_path], mimi_dtype, &device) .context("Failed to load Mimi weights")? }; MimiModel::new(mimi_config, mimi_vb).context("Failed to create Mimi model")? } else { - // Try to load from HuggingFace cache or bundled location return Err(anyhow!( - "Mimi codec weights not found at {}. Download with: ./scripts/download-csm.sh", - mimi_weights_path.display() + "Mimi codec weights not found. Run:\n\ + - hf download kyutai/mimi\n\ + - hf download sesame/csm-1b" )); }; @@ -146,14 +163,22 @@ impl TtsEngine { let vb = VarBuilder::from_tensors(csm_tensors, dtype, &device); let csm = CsmModel::new(&config, vb).context("Failed to create CSM model from embedded")?; - // Load Mimi codec from bytes with default v0.1 config - let mimi_config = MimiConfig::v0_1(None); + // Load Mimi codec from bytes with config matching CSM's audio_num_codebooks + // NOTE: Mimi MUST use F32 - Metal conv1d doesn't support BF16 + // NOTE: num_codebooks MUST match CSM config (32 for csm-1b) + let mimi_num_codebooks = config.audio_num_codebooks; + let mimi_config = MimiConfig::v0_1(Some(mimi_num_codebooks)); + let mimi_dtype = DType::F32; + debug!( + "Mimi (embedded) config: num_codebooks={} (from CSM audio_num_codebooks)", + mimi_num_codebooks + ); let mimi_tensors: HashMap = candle_core::safetensors::load_buffer(embedded.mimi_weights, &Device::Cpu) .context("Failed to deserialize Mimi weights")?; - let mimi_tensors = move_tensors_to_device(mimi_tensors, &device, dtype)?; - let mimi_vb = VarBuilder::from_tensors(mimi_tensors, dtype, &device); + let mimi_tensors = move_tensors_to_device(mimi_tensors, &device, mimi_dtype)?; + let mimi_vb = VarBuilder::from_tensors(mimi_tensors, mimi_dtype, &device); let mimi = MimiModel::new(mimi_config, mimi_vb).context("Failed to create Mimi from embedded")?; @@ -181,46 +206,63 @@ impl TtsEngine { } /// Synthesize text with specific speaker index + /// + /// Uses CSM's iterative generation: + /// 1. Convert text to tokens via `text_tokens_and_mask()` → 3D tensor [1, seq, cb+1] + /// 2. Generate first audio frame + /// 3. Convert frame to tokens via `audio_tokens_and_mask()` → next input + /// 4. Repeat until end-of-generation (all zeros) pub fn synthesize_with_speaker(&mut self, text: &str, speaker_idx: usize) -> Result> { // Clear KV cache for fresh generation self.csm.clear_kv_cache(); - // Format text with speaker index + // Format text with speaker index (CSM format: [speaker_idx]text<|end_of_text|>) let prompt = format!("[{}]{}{}", speaker_idx, text, EOT_TOKEN); debug!("TTS prompt: {}", prompt); // Tokenize text let encoding = self .tokenizer - .encode(prompt.as_str(), false) + .encode(prompt.as_str(), true) // add_special_tokens=true like candle example .map_err(|e| anyhow!("Tokenization failed: {}", e))?; let text_tokens: Vec = encoding.get_ids().to_vec(); debug!("Text tokens: {} tokens", text_tokens.len()); - // Convert to tensors - shape [batch, seq_len] - let tokens = Tensor::new(text_tokens.as_slice(), &self.device)? - .unsqueeze(0)? - .to_dtype(DType::I64)?; - let mask = Tensor::ones(tokens.dims(), DType::F32, &self.device)?; + // Convert to 3D tensors via CSM helper - shape [1, seq_len, codebooks+1] + // This creates proper format: [0,0,...,0,text_id] for each token + let (mut tokens, mut mask) = self.csm.text_tokens_and_mask(&text_tokens)?; - // Generate audio codes + // Position counter for KV cache + let mut pos: usize = 0; + + // Generate audio codes iteratively let mut logits_processor = LogitsProcessor::new(42, None, None); let mut all_codes: Vec> = Vec::new(); - for frame_idx in 0..MAX_FRAMES { + for _frame_idx in 0..MAX_FRAMES { // Generate one frame of audio codes - // API: generate_frame(&tokens, &mask, pos, &mut logits_processor) -> Result> let frame: Vec = self.csm - .generate_frame(&tokens, &mask, frame_idx, &mut logits_processor)?; + .generate_frame(&tokens, &mask, pos, &mut logits_processor)?; + + // Advance position by sequence length + pos += tokens.dim(1)?; - // Check for end of generation (all zeros) + // Check for end of generation (all zeros = silence marker) if frame.iter().all(|&x| x == 0) { - debug!("Generation complete at frame {}", frame_idx); + debug!("Generation complete after {} frames", all_codes.len()); + // Generate one more frame after EOS for clean ending (like candle example) + let _ = self + .csm + .generate_frame(&tokens, &mask, pos, &mut logits_processor); break; } - all_codes.push(frame); + all_codes.push(frame.clone()); + + // Convert audio frame to next input tokens + // This creates: [code0,code1,...,codeN,0] format + (tokens, mask) = self.csm.audio_tokens_and_mask(frame)?; } if all_codes.is_empty() { @@ -236,6 +278,9 @@ impl TtsEngine { } /// Decode RVQ audio codes (Vec format) to PCM samples using Mimi codec + /// + /// Input: codes[frame_idx][codebook_idx] - list of frames, each with codebook values + /// Output: f32 PCM samples at 24kHz fn decode_audio_codes_vec(&mut self, codes: &[Vec]) -> Result> { if codes.is_empty() { return Ok(Vec::new()); @@ -244,8 +289,13 @@ impl TtsEngine { let num_frames = codes.len(); let num_codebooks = codes[0].len(); + debug!( + "decode_audio_codes_vec: {} frames × {} codebooks", + num_frames, num_codebooks + ); + // Create tensor [num_codebooks, num_frames] by transposing the data - // codes is [frames, codebooks], we need [codebooks, frames] + // Input: codes[frame][codebook], Output: tensor[codebook][frame] let mut flat_codes: Vec = Vec::with_capacity(num_codebooks * num_frames); for cb_idx in 0..num_codebooks { for frame in codes { @@ -258,8 +308,16 @@ impl TtsEngine { .to_dtype(DType::I64)?; // Add batch dimension: [codebooks, frames] -> [batch, codebooks, frames] + // Mimi expects [B, K, T] where K=codebooks, T=frames let codes_tensor = codes_tensor.unsqueeze(0)?; + debug!( + "Mimi decode input shape: {:?} (expected [1, {}, {}])", + codes_tensor.shape(), + num_codebooks, + num_frames + ); + // Decode with Mimi let audio = self.mimi.decode(&codes_tensor)?; diff --git a/core/tts/singleton.rs b/core/tts/singleton.rs index 44e7a95e..54b20f98 100644 --- a/core/tts/singleton.rs +++ b/core/tts/singleton.rs @@ -16,9 +16,11 @@ use tracing::{info, warn}; use super::audio_output::AudioPlayer; use super::engine::TtsEngine; +use crate::hf_cache; /// Default TTS model name (for dev/fallback mode) pub const DEFAULT_MODEL: &str = "csm-1b"; +const DEFAULT_TTS_REPO: &str = "sesame/csm-1b"; /// Global singleton engine static ENGINE: OnceLock> = OnceLock::new(); @@ -46,7 +48,16 @@ fn resolve_model_path_fallback() -> Result { warn!("CODESCRIBE_TTS_PATH set but model incomplete: {}", path); } - // 2. Default models directory + // 2. HuggingFace cache (hf download sesame/csm-1b) + if let Some(snapshot) = hf_cache::find_snapshot( + DEFAULT_TTS_REPO, + &["config.json", "tokenizer.json", "model.safetensors"], + ) { + info!("Using HF cache TTS model: {}", snapshot.display()); + return Ok(snapshot); + } + + // 3. Default models directory let base_dirs = BaseDirs::new().ok_or_else(|| anyhow!("Cannot determine home directory"))?; let home = base_dirs.home_dir(); let models_dir = home.join(".codescribe").join("models").join(DEFAULT_MODEL); @@ -55,7 +66,7 @@ fn resolve_model_path_fallback() -> Result { return Ok(models_dir); } - // 3. Bundled .app fallback (Tauri builds without embedding) + // 4. Bundled .app fallback (Tauri builds without embedding) let exe = std::env::current_exe().context("Failed to get executable path")?; let exe_dir = exe.parent().context("Failed to get executable directory")?; @@ -71,7 +82,8 @@ fn resolve_model_path_fallback() -> Result { "TTS model not available.\n\ Debug builds: Set CODESCRIBE_TTS_PATH\n\ Release builds: Model should be embedded\n\n\ - Download with: ./scripts/download-csm.sh" + Download with: hf download {}", + DEFAULT_TTS_REPO )) } diff --git a/core/vad/config.rs b/core/vad/config.rs index 23bb6b65..cb49ab56 100644 --- a/core/vad/config.rs +++ b/core/vad/config.rs @@ -26,31 +26,36 @@ pub struct VadConfig { impl Default for VadConfig { fn default() -> Self { + // CLEAN API: One env var per parameter, no competing aliases + // See docs/ENV_REGISTRY.toml for full documentation Self { - // Clamp threshold to valid probability range [0.1, 0.95] - threshold: env_f32_clamped("CODESCRIBE_VAD_THRESHOLD", 0.5, 0.1, 0.95), - // Clamp durations to reasonable ranges + // Speech probability threshold (0.0-1.0) + // Lower = more sensitive (catches quiet speech), Higher = more conservative + // Default 0.35 - catches quiet speech without triggering on silence + threshold: env_f32_clamped("CODESCRIBE_VAD_THRESHOLD", 0.35, 0.1, 0.95), + + // Minimum speech duration before triggering (filters clicks/pops) min_speech_duration_sec: env_f32_clamped( "CODESCRIBE_VAD_MIN_SPEECH_SEC", 0.1, 0.01, 1.0, ), - // Sync with default_env.txt: 1.2s (was 0.8s) - max_silence_duration_sec: env_f32_clamped( - "CODESCRIBE_VAD_MAX_SILENCE_SEC", - 1.2, - 0.1, - 10.0, - ), - // Sync with default_env.txt: 60s (was 30s) + + // Silence duration before utterance flush (allows natural pauses) + // Default 2.5s - human speech pauses are typically 1-2s + max_silence_duration_sec: env_f32_clamped("CODESCRIBE_VAD_SILENCE_SEC", 2.5, 0.1, 10.0), + + // Maximum utterance length (force flush after this) max_utterance_sec: env_f32_clamped( "CODESCRIBE_VAD_MAX_UTTERANCE_SEC", 60.0, 1.0, 300.0, ), - pre_roll_sec: env_f32_clamped("CODESCRIBE_VAD_PRE_ROLL_SEC", 0.3, 0.0, 2.0), + + // Pre-roll buffer (captures context before speech onset) + pre_roll_sec: env_f32_clamped("CODESCRIBE_VAD_PRE_ROLL_SEC", 0.5, 0.0, 2.0), } } } @@ -94,15 +99,26 @@ impl VadConfig { } fn env_f32(key: &str, default: f32) -> f32 { - std::env::var(key) - .ok() - .and_then(|v| v.parse::().ok()) - .unwrap_or(default) + match std::env::var(key) { + Ok(val) => match val.parse::() { + Ok(v) => v, + Err(_) => { + tracing::warn!("{key}={val:?} is not a valid f32, using default {default}"); + default + } + }, + Err(_) => default, + } } /// Parse env var as f32 with clamping to valid range fn env_f32_clamped(key: &str, default: f32, min: f32, max: f32) -> f32 { - env_f32(key, default).clamp(min, max) + let raw = env_f32(key, default); + let clamped = raw.clamp(min, max); + if raw != clamped { + tracing::warn!("{key}={raw} out of range [{min}, {max}], clamped to {clamped}"); + } + clamped } #[cfg(test)] diff --git a/core/vad/embedded.rs b/core/vad/embedded.rs new file mode 100644 index 00000000..27a991ce --- /dev/null +++ b/core/vad/embedded.rs @@ -0,0 +1,45 @@ +//! Embedded Silero VAD model - direct include via generated code +//! +//! Release builds: Model file included directly in binary (~2.3MB) +//! Debug builds: Empty slice, use runtime file path +//! +//! Created by M&K (c)2026 VetCoders + +#[cfg(embed_vad)] +mod data { + include!(concat!(env!("OUT_DIR"), "/embedded_vad_data.rs")); +} + +#[cfg(not(embed_vad))] +mod data { + pub static MODEL: &[u8] = &[]; +} + +/// Check if embedded VAD model is available +pub fn is_embedded_available() -> bool { + let size = data::MODEL.len(); + tracing::debug!(size, "Embedded VAD check"); + size > 0 +} + +/// Get embedded model bytes if available +pub fn get_embedded_data() -> Option<&'static [u8]> { + if !is_embedded_available() { + return None; + } + Some(data::MODEL) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_embedded_availability() { + let available = is_embedded_available(); + println!("Embedded VAD available: {}", available); + if available { + println!("VAD size: {:.2} MB", data::MODEL.len() as f64 / 1_000_000.0); + } + } +} diff --git a/core/vad/mod.rs b/core/vad/mod.rs index 88f50100..f3a4b25a 100644 --- a/core/vad/mod.rs +++ b/core/vad/mod.rs @@ -1,6 +1,6 @@ //! Voice Activity Detection (VAD) module using Silero neural network. //! -//! Custom wrapper that shares ort runtime with fastembed (no dependency conflicts). +//! Custom wrapper that uses a shared ort runtime (no dependency conflicts). //! Uses worker thread to avoid blocking audio callbacks. //! //! ## Quick Start @@ -26,6 +26,7 @@ //! Created by M&K (c)2026 VetCoders pub mod config; +pub mod embedded; pub mod silero_ort; pub use config::VadConfig; diff --git a/core/vad/silero_ort.rs b/core/vad/silero_ort.rs index 4a4597df..3b26803b 100644 --- a/core/vad/silero_ort.rs +++ b/core/vad/silero_ort.rs @@ -1,22 +1,24 @@ //! Silero VAD wrapper using ort directly. //! -//! Custom implementation that shares ort runtime with fastembed. +//! Custom implementation using ort runtime directly. //! Model: silero_vad.onnx v5 from https://github.com/snakers4/silero-vad //! //! Created by M&K (c)2026 VetCoders -use std::path::Path; +use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{OnceLock, mpsc}; use std::thread; use anyhow::{Context, Result}; -use ndarray::{Array1, Array2, Array3}; +use ndarray::ArrayD; use ort::session::Session; -use ort::value::Tensor; +use ort::value::Value; use tracing::{debug, info}; use super::config::VadConfig; +use super::embedded; +use crate::hf_cache; /// Silero VAD sample rate (always 16kHz) pub const VAD_SAMPLE_RATE: u32 = 16000; @@ -24,9 +26,15 @@ pub const VAD_SAMPLE_RATE: u32 = 16000; /// Chunk size for Silero (512 samples = 32ms at 16kHz) const CHUNK_SIZE: usize = 512; -/// State dimensions for Silero v5 model -const STATE_DIM: usize = 64; -const STATE_LAYERS: usize = 2; +/// Context size for Silero v5 (64 samples at 16kHz). +/// Each inference call requires prepending the last 64 samples from the +/// previous chunk. Without this the model receives incomplete input and +/// returns unreliable speech probabilities. +const CONTEXT_SIZE: usize = 64; + +/// Unified state shape for Silero v5: [2, 1, 128]. +/// v4 used separate h/c tensors with dim 64; v5 merged them. +const STATE_SHAPE: [usize; 3] = [2, 1, 128]; /// Global VAD worker static VAD_WORKER: OnceLock = OnceLock::new(); @@ -77,31 +85,47 @@ impl Resampler { } } -/// Silero VAD model wrapper +/// Silero VAD v5 model wrapper. +/// +/// v5 API differences from v4: +/// - Unified state tensor `[2, 1, 128]` (v4 had separate h/c `[2, 1, 64]`) +/// - Input order: `input, state, sr` (v4: `input, sr, h, c`) +/// - Output names: `output`, `stateN` (v4: positional) +/// - Context window: 64 samples prepended to each 512-sample chunk pub struct SileroVad { session: Session, - state_h: Array3, - state_c: Array3, + state: ArrayD, + context: Vec, config: VadConfig, resampler: Option, } impl SileroVad { - /// Load Silero VAD model from path + /// Load Silero VAD model from embedded bytes (if available) or path pub fn new(model_path: &Path, config: VadConfig) -> Result { - info!("Loading Silero VAD model from: {}", model_path.display()); - - let session = Session::builder()? - .with_intra_threads(1)? - .commit_from_file(model_path) - .context("Failed to load Silero VAD ONNX model")?; + let session = if let Some(model_bytes) = embedded::get_embedded_data() { + info!( + "Loading Silero VAD model from embedded bytes ({:.2} MB)", + model_bytes.len() as f64 / 1_000_000.0 + ); + Session::builder()? + .with_intra_threads(1)? + .commit_from_memory(model_bytes) + .context("Failed to load embedded Silero VAD ONNX model")? + } else { + info!("Loading Silero VAD model from: {}", model_path.display()); + Session::builder()? + .with_intra_threads(1)? + .commit_from_file(model_path) + .context("Failed to load Silero VAD ONNX model")? + }; debug!("Silero VAD model loaded successfully"); Ok(Self { session, - state_h: Array3::zeros((STATE_LAYERS, 1, STATE_DIM)), - state_c: Array3::zeros((STATE_LAYERS, 1, STATE_DIM)), + state: ArrayD::zeros(STATE_SHAPE.as_slice()), + context: vec![0.0; CONTEXT_SIZE], config, resampler: None, }) @@ -116,87 +140,76 @@ impl SileroVad { } } - /// Get speech probability for audio chunk (0.0 - 1.0) + /// Get speech probability for a single CHUNK_SIZE (512) frame at 16kHz. /// - /// Automatically resamples if input rate was set. + /// The caller is responsible for providing exactly CHUNK_SIZE samples + /// already at 16kHz. The internal resampler path is kept for + /// backwards-compat but callers in streaming_recorder pre-resample. pub fn predict(&mut self, samples: &[f32]) -> Result { if samples.is_empty() { return Ok(0.0); } - // Resample if needed (get owned Vec to avoid borrow issues) + // Resample if needed let samples_16k: Vec = if let Some(ref mut resampler) = self.resampler { resampler.resample(samples) } else { samples.to_vec() }; - // Process in chunks and return max probability let mut max_prob = 0.0f32; - for chunk in samples_16k.chunks(CHUNK_SIZE) { - // Pad chunk if needed - let padded: Vec = if chunk.len() < CHUNK_SIZE { - let mut p = chunk.to_vec(); - p.resize(CHUNK_SIZE, 0.0); - p - } else { - chunk.to_vec() - }; - - let prob = self.predict_chunk(&padded)?; + if chunk.len() < CHUNK_SIZE { + break; + } + let prob = self.predict_chunk(chunk)?; max_prob = max_prob.max(prob); } - Ok(max_prob) } - /// Predict on a single 512-sample chunk + /// Predict on a single 512-sample chunk using Silero v5 API. + /// + /// Prepends 64-sample context, sends `[input, state, sr]`, + /// reads `output` (prob) and `stateN` (updated state). fn predict_chunk(&mut self, chunk: &[f32]) -> Result { - // Input: (batch=1, samples) - let input_array = Array2::from_shape_vec((1, chunk.len()), chunk.to_vec())?; - - // Sample rate as i64 - let sr_array = Array1::from_vec(vec![VAD_SAMPLE_RATE as i64]); - - // Create input tensors (Tensor::from_array in ort rc.11) - let input = Tensor::from_array(input_array)?; - let sr = Tensor::from_array(sr_array)?; - let h = Tensor::from_array(self.state_h.clone())?; - let c = Tensor::from_array(self.state_c.clone())?; - - // Run inference with named inputs - // Silero VAD v5 expects: input, sr, h, c - let outputs = self.session.run(ort::inputs![ - "input" => input, - "sr" => sr, - "h" => h, - "c" => c + // Build context + chunk → [1, 576] + let mut input_data = Vec::with_capacity(CONTEXT_SIZE + chunk.len()); + input_data.extend_from_slice(&self.context); + input_data.extend_from_slice(chunk); + + // Update context for next call + let ctx_start = chunk.len().saturating_sub(CONTEXT_SIZE); + self.context[..].copy_from_slice(&chunk[ctx_start..]); + + // Input tensors — v5 order: input, state, sr + let input = ndarray::Array2::from_shape_vec([1, input_data.len()], input_data) + .map_err(|e| anyhow::anyhow!("input shape: {}", e))?; + let sr = ndarray::Array1::from_vec(vec![VAD_SAMPLE_RATE as i64]); + let state = std::mem::replace(&mut self.state, ArrayD::zeros(STATE_SHAPE.as_slice())); + + let input_value = Value::from_array(input)?; + let state_value = Value::from_array(state)?; + let sr_value = Value::from_array(sr)?; + + let outputs = self.session.run([ + (&input_value).into(), + (&state_value).into(), + (&sr_value).into(), ])?; - // Extract probability from first output + // Read probability from "output" let prob = { - let output_value = &outputs[0]; - let (_shape, data) = output_value.try_extract_tensor::()?; + let (_shape, data) = outputs["output"].try_extract_tensor::()?; data.first().copied().unwrap_or(0.0) }; - // Update states if model returns them (outputs 1 and 2) - if outputs.len() > 2 { - // Extract new h state (let chains - Rust 2024) - if let Ok((_shape, h_data)) = outputs[1].try_extract_tensor::() - && let Ok(arr) = - Array3::from_shape_vec((STATE_LAYERS, 1, STATE_DIM), h_data.to_vec()) - { - self.state_h = arr; - } - - // Extract new c state - if let Ok((_shape, c_data)) = outputs[2].try_extract_tensor::() - && let Ok(arr) = - Array3::from_shape_vec((STATE_LAYERS, 1, STATE_DIM), c_data.to_vec()) - { - self.state_c = arr; + // Read updated state from "stateN" + { + let (shape, data) = outputs["stateN"].try_extract_tensor::()?; + let shape_usize: Vec = shape.as_ref().iter().map(|&d| d as usize).collect(); + if let Ok(arr) = ArrayD::from_shape_vec(shape_usize.as_slice(), data.to_vec()) { + self.state = arr; } } @@ -205,8 +218,8 @@ impl SileroVad { /// Reset internal state pub fn reset(&mut self) { - self.state_h.fill(0.0); - self.state_c.fill(0.0); + self.state = ArrayD::zeros(STATE_SHAPE.as_slice()); + self.context.fill(0.0); } /// Get current threshold @@ -223,12 +236,8 @@ use std::sync::{Arc, atomic::AtomicU32}; /// Message to VAD worker (fire-and-forget, no response channel) enum VadMessage { - Predict { - samples: Vec, - sample_rate: u32, - }, + Predict { samples: Vec, sample_rate: u32 }, Reset, - #[allow(dead_code)] Shutdown, } @@ -241,6 +250,8 @@ struct VadWorker { initialized: AtomicBool, /// Last computed probability (f32 as bits for atomic access) last_prob: Arc, + /// Worker thread handle for clean shutdown + thread: Option>, } impl VadWorker { @@ -265,7 +276,7 @@ impl VadWorker { // Oneshot channel to confirm model loaded successfully let (init_tx, init_rx) = mpsc::sync_channel::>(1); - thread::spawn(move || { + let handle = thread::spawn(move || { let mut vad = match SileroVad::new(&path, config) { Ok(v) => { // Signal success to main thread @@ -312,6 +323,7 @@ impl VadWorker { sender: tx, initialized: AtomicBool::new(true), last_prob, + thread: Some(handle), }) } Ok(Err(e)) => { @@ -345,6 +357,17 @@ impl VadWorker { } } +impl Drop for VadWorker { + fn drop(&mut self) { + // Send shutdown signal; if the channel is full or closed, the thread + // will exit anyway when the sender is dropped and `for msg in rx` ends. + let _ = self.sender.try_send(VadMessage::Shutdown); + if let Some(handle) = self.thread.take() { + let _ = handle.join(); + } + } +} + // ═══════════════════════════════════════════════════════════ // Public singleton API // ═══════════════════════════════════════════════════════════ @@ -389,7 +412,7 @@ pub fn init_with_config(model_path: &Path, config: VadConfig) -> Result<()> { } // Try to create worker - if it fails, VAD stays uninitialized - // (speech_probability will return 1.0, disabling auto-stop) + // (speech_probability will return 1.0, effectively disabling segmentation) match VadWorker::new(model_path, config.clone()) { Ok(worker) => { // Only set config/path AFTER successful init @@ -400,7 +423,7 @@ pub fn init_with_config(model_path: &Path, config: VadConfig) -> Result<()> { Ok(()) } Err(e) => { - tracing::error!("VAD init failed: {} - auto-stop disabled", e); + tracing::error!("VAD init failed: {} - segmentation disabled", e); Err(e) } } @@ -423,8 +446,8 @@ pub fn is_initialized() -> bool { /// This "eventual consistency" approach avoids blocking the audio thread. /// After a few calls, the returned value will reflect recent audio. /// -/// **Important:** Returns 1.0 when VAD not initialized (assume speech) -/// to prevent immediate auto-stop in recorders. +/// **Important:** Returns 1.0 when VAD not initialized (assume speech), +/// which effectively disables silence-based segmentation. pub fn speech_probability(samples: &[f32], sample_rate: u32) -> f32 { if let Some(worker) = VAD_WORKER.get() { // Submit new audio (non-blocking) @@ -432,7 +455,7 @@ pub fn speech_probability(samples: &[f32], sample_rate: u32) -> f32 { // Return last computed probability (instant, atomic read) worker.last_probability() } else { - // VAD not initialized - assume speech to prevent premature auto-stop + // VAD not initialized - assume speech to prevent premature segmentation 1.0 } } @@ -450,14 +473,37 @@ pub fn reset() { } } -/// Get default model path (~/.codescribe/models/silero_vad.onnx) -pub fn default_model_path() -> std::path::PathBuf { +/// HuggingFace repo for Silero VAD model +const SILERO_VAD_REPO: &str = "snakers4/silero-vad"; +const SILERO_VAD_FILE: &str = "silero_vad.onnx"; + +/// Get default model path (bundled/models dir -> HF cache -> ~/.codescribe/models/) +pub fn default_model_path() -> PathBuf { + // 1) Bundled / models dir (app Resources/models or ./models) + if let Ok(manager) = crate::config::models::ModelManager::new() { + let candidate = manager.models_dir().join(SILERO_VAD_FILE); + if candidate.exists() { + debug!("Using Silero VAD from models dir: {}", candidate.display()); + return candidate; + } + } + + // Try HF cache first (from `hf download snakers4/silero-vad`) + if let Some(snapshot) = hf_cache::find_snapshot(SILERO_VAD_REPO, &[SILERO_VAD_FILE]) { + let model_path = snapshot.join(SILERO_VAD_FILE); + if model_path.exists() { + debug!("Using Silero VAD from HF cache: {}", model_path.display()); + return model_path; + } + } + + // Fallback to legacy path directories::BaseDirs::new() .map(|d| d.home_dir().to_path_buf()) - .unwrap_or_else(|| std::path::PathBuf::from(".")) + .unwrap_or_else(|| PathBuf::from(".")) .join(".codescribe") .join("models") - .join("silero_vad.onnx") + .join(SILERO_VAD_FILE) } #[cfg(test)] diff --git a/docs/ADR/2026-01-16-LICENSE_NOTES.md b/docs/ADR/2026-01-16-LICENSE_NOTES.md new file mode 100644 index 00000000..13e77580 --- /dev/null +++ b/docs/ADR/2026-01-16-LICENSE_NOTES.md @@ -0,0 +1,23 @@ +# CodeScribe License Notes + +CodeScribe is distributed under the **BSD 4-Clause License (Original BSD)**. The clause that +usually raises questions is the *advertising clause*: + +> "All advertising materials mentioning features or use of this software must display the +> acknowledgement: This product includes software developed by Loctree." + +What this means in practice: + +| Scenario | Requirement | +|----------------------------------------------------------------------|------------------------------------------------------------------------------------| +| README / documentation inside this repo | already satisfied (see README §License). | +| Public marketing site, blog post, product page mentioning CodeScribe | include the acknowledgement sentence verbatim. | +| Binary redistributions (DMG, PKG) | include the acknowledgement in the splash / "About" dialog or accompanying README. | +| Internal deployments (no external advertising) | no additional action beyond keeping the LICENSE file intact. | + +If you fork CodeScribe and redistribute it, you must keep the LICENSE file and the acknowledgement +sentence wherever you describe the fork. If you embed CodeScribe inside another product, place the +acknowledgement next to other third‑party credits. + +For questions about alternative licensing please reach out Loctree team +at [contact@loctree.io](mailto:contact@loctree.io). diff --git a/docs/ADR/2026-01-18-WHISPER_LIVE.md b/docs/ADR/2026-01-18-WHISPER_LIVE.md new file mode 100644 index 00000000..74b0d5cc --- /dev/null +++ b/docs/ADR/2026-01-18-WHISPER_LIVE.md @@ -0,0 +1,152 @@ +# WHISPER LIVE (Embedded + Streaming Transcription) + +> **Status:** DONE ✅ (2026-01-16) +> +> **Tagline:** Whisper is welded into the binary, and transcription happens *during recording*. + +## TL;DR + +CodeScribe’s core power-up is: + +1. **Embedded Whisper model** (`whisper-large-v3-turbo-mlx-q8`) in the release binary + - **Zero disk I/O** for local STT + - Model loads once into GPU/Metal and stays in-process +2. **Live (streaming) transcription** while the user is recording + - Audio is chunked and transcribed in the background + - On `stop()` we only “close” the last fragment → **near-instant time-to-paste** + +## What we shipped + +### 1) Embedded Whisper (Strict Policy) + +- **ALWAYS Embedded:** The model (`whisper-large-v3-turbo-mlx-q8`) is welded into the release binary. + - **Zero Exceptions:** We never bundle the `models/` folder in the `.app`. + - **Zero Disk I/O:** Model loads directly from memory to Metal GPU. + - **Native Power:** Minimal abstraction layers (approx. 4) compared to typical 32+ in heavy JS/Python bridges. +- **Global Singleton:** A process-wide engine instance loads once and stays resident. + +Key behavior: + +- **Release:** Strict embedded mode. If the model isn't found at build time, the build fails. +- **Development:** Local debug builds can still resolve external paths for rapid iteration, but the release pipeline enforces embedding. + +### 2) Streaming transcription (during recording) + +We removed the old bottleneck: + +```text +Audio callback → buffer → stop() → WAV write/read → transcribe entire audio → LLM +``` + +And replaced it with: + +```text +Audio callback → non-blocking channel → chunking worker → spawn_blocking(Whisper) → transcript buffer + ↓ + overlap dedup + +stop() → transcribe last pending samples → return final transcript → LLM/paste +``` + +Practical win: + +- **~35s recording:** `stop()` is ~0.5s (last chunk only) instead of ~4s (whole audio) + +## What’s new around Whisper Live + +- **Stream postprocess** (`src/stream_postprocess.rs`) — semantic gating and cleanup of chunk output + before final paste/LLM, reducing low‑quality fragments in live mode. +- **IPC server** (`src/ipc/`) — stable runtime interface for GUI/clients; Whisper Live can be + consumed and extended outside the tray flow. +- **Quality loop/report** (`src/quality_loop.rs`, `src/quality_report.rs`) — automated scoring and + batch diagnostics for streaming accuracy and regressions. + +## How it works (high level) + +```mermaid +flowchart TD + A[CPAL input callback (audio thread)] -->|try_send f32 samples| B[mpsc channel] + B --> C[StreamingRecorder worker (tokio task)] + C -->|accumulate| D[chunk buffer] + D -->|every ~15s with ~2s overlap| E[spawn_blocking] + E --> F[Whisper singleton engine (Metal)] + F --> G[chunk text] + G --> H[append_with_overlap_dedup] + H --> I[transcript_buffer] + I --> J[controller stop(): finalize + paste / LLM] +``` + +## Where in the code + +### Embedded + singleton engine + +- `src/whisper/embedded.rs` — embedded model bytes and accessors +- `src/whisper/singleton.rs` — global engine singleton (loads embedded model and exposes `transcribe*()`) +- `src/whisper/engine.rs` — Candle/Whisper inference, chunking, overlap dedup (`append_with_overlap_dedup`) + +### Live streaming recorder + +- `src/audio/recorder.rs` + - CPAL input stream at **native device rate** (often `48000Hz`) + - callback hook for raw `f32` samples + - exposes `Recorder::actual_sample_rate()` +- `src/audio/streaming_recorder.rs` + - connects recorder callback → `mpsc::channel` (non-blocking) + - chunking (default: `15s` chunks + `2s` overlap) + - background transcription via `tokio::spawn_blocking` + - dedup between chunks via `append_with_overlap_dedup` +- `src/controller.rs` + - uses `StreamingRecorder` and prefers the streaming transcript on `stop()` + - can still save the WAV for logs and/or cloud fallback + +## Build & distribution + +### Install from source (embedded model) + +```bash +make download-model # ensures models/whisper-large-v3-turbo-mlx-q8 exists for embedding +make install # builds + installs an ~888MB binary with embedded model +``` + +### Bundle / DMG (embedded-only) + +```bash +make bundle +make dmg-full +``` + +Notes: + +- DMG ships the `.app` with **the embedded model only** (no `Resources/models/*` duplication) +- A “too small” release binary is treated as a build error (guardrail in `scripts/build-release.sh`) + +## Troubleshooting / FAQ + +### “My binary is small — is the model embedded?” + +If the release binary is far below ~500MB, embedding likely didn’t happen. + +Checklist: + +- ensure `models/whisper-large-v3-turbo-mlx-q8/` exists (download step) +- ensure `CODESCRIBE_NO_EMBED` is not set +- rebuild with `cargo build --release` + +### “Why does streaming care about actual sample rate?” + +Microphones usually run at `48kHz`. We record at the device’s native rate for compatibility, +and Whisper internally resamples to `16kHz`. + +**Important:** streaming must pass the **real** `sample_rate` to the engine — otherwise you +get hallucinations and low confidence (classic “gibberish” pattern). + +## Benchmarks (rule of thumb) + +- Model load: ~7s (first time after app start, embedded → GPU) +- Live transcription: overlaps with recording +- After `stop()`: usually just final chunk, typically well below 1s + +--- + +**Made with (งಠ_ಠ)ง by the ⌜ CodeScribe ⌟ 𝖙𝖊𝖆𝖒 (c) 2024-2026 +Maciej & Monika + Klaudiusz (AI) + Junie (AI)** diff --git a/docs/ADR/2026-01-19-INSTALLATION.md b/docs/ADR/2026-01-19-INSTALLATION.md new file mode 100644 index 00000000..dd84c741 --- /dev/null +++ b/docs/ADR/2026-01-19-INSTALLATION.md @@ -0,0 +1,194 @@ +# CodeScribe Installation and Launch Guide + +This document describes the installation methods, configuration paths, and how the application locates its resources. + +## Installation Methods + +### Method 1: CLI Install (Recommended for Development) + +```bash +make download-model # Download Whisper model (required for embedding) +make install # Build and install to ~/.cargo/bin/ +``` + +**Result**: Binary `codescribe` installed to `~/.cargo/bin/` (~888MB with embedded model). + +**How it runs**: Direct execution from terminal or as background daemon. + +### Method 2: App Bundle (For Distribution) + +```bash +make bundle # Creates bundle/CodeScribe.app +make install-app # Copies to /Applications/CodeScribe.app +``` + +**Result**: Standard macOS .app bundle in `/Applications/`. + +**How it runs**: Double-click or launch from Spotlight. + +### Method 3: DMG Distribution (For End Users) + +```bash +make dmg-signed # Build signed DMG +make notarize # Notarize with Apple (requires Developer ID) +``` + +**Result**: `CodeScribe_X.Y.Z.dmg` ready for distribution. + +## Configuration + +### Config Directory + +All configuration is stored in: + +``` +~/.codescribe/ +├── .env # Main configuration file +├── prompts/ # Custom AI prompts +│ ├── formatting.txt +│ └── assistive.txt +├── history/ # Transcription history +├── reports/ # Quality reports +└── repo_path # Path to source repo (set during install) +``` + +### Environment Variables (.env) + +The application loads configuration from `~/.codescribe/.env` using these priorities: + +1. **Environment variables** (highest priority) +2. **~/.codescribe/.env** (main config file) +3. **Default values** (fallback) + +```mermaid +flowchart TD + A[Application Start] --> B{Check ENV vars} + B -->|Set| C[Use ENV value] + B -->|Not set| D{Check ~/.codescribe/.env} + D -->|Exists| E[Load with dotenvy] + D -->|Missing| F[Create from template] + F --> G{Find template} + G -->|Bundle| H[../Resources/.env.example] + G -->|Repo| I[.env.example in repo root] + G -->|None| J[Generate minimal .env] + E --> K[Apply defaults for missing keys] + H --> K + I --> K + J --> K + C --> L[Config Ready] + K --> L +``` + +### Key Configuration Variables + +```env +# Speech-to-Text +WHISPER_LANGUAGE=pl # pl | en | de | fr +USE_LOCAL_STT=1 # 1 = embedded Whisper + +# Hotkeys +HOLD_MODS=ctrl # ctrl | ctrl_alt | ctrl_shift +TOGGLE_TRIGGER=double_option # double_option | none + +# AI Formatting +AI_FORMATTING_ENABLED=1 +LLM_ENDPOINT=https://api.openai.com/v1/responses +LLM_MODEL=gpt-4.1-mini +LLM_API_KEY=sk-xxx + +# Optional: Separate providers for modes +LLM_FORMATTING_{ENDPOINT,MODEL,API_KEY}=... +LLM_ASSISTIVE_{ENDPOINT,MODEL,API_KEY}=... +``` + +## Bundle Structure + +``` +CodeScribe.app/ +└── Contents/ + ├── Info.plist # Bundle metadata (icon, identifier, version) + ├── MacOS/ + │ └── codescribe # Main executable (~888MB with embedded model) + └── Resources/ + └── AppIcon.icns # Application icon +``` + +### Info.plist Keys + +| Key | Value | Purpose | +|-----|-------|---------| +| CFBundleIdentifier | io.loctree.codescribe | Unique app identifier | +| CFBundleIconFile | AppIcon | Points to AppIcon.icns | +| CFBundleExecutable | codescribe | Main binary name | +| LSMinimumSystemVersion | 14.0 | Requires macOS Sonoma+ | +| NSMicrophoneUsageDescription | ... | Microphone permission prompt | + +## Icons + +### Tray Icon +- **Source**: `assets/icon.png` (embedded via `include_bytes!`) +- **Location in code**: `src/tray/icons.rs` +- **Size**: 44x44 pixels (Retina), 22x22 logical + +### Dock Icon +- **For CLI**: Programmatically set via `set_dock_icon()` in `src/ui.rs` +- **For Bundle**: Uses `CFBundleIconFile` from Info.plist pointing to `AppIcon.icns` +- **Source**: `assets/AppIcon.icns` + +### Icon Loading Flow + +```mermaid +flowchart LR + subgraph CLI["CLI Mode (codescribe)"] + A1[Start] --> A2[set_dock_icon] + A2 --> A3[NSImage from include_bytes] + A3 --> A4[setApplicationIconImage] + end + + subgraph Bundle["Bundle Mode (.app)"] + B1[Start] --> B2[macOS reads Info.plist] + B2 --> B3[CFBundleIconFile = AppIcon] + B3 --> B4[Load AppIcon.icns from Resources] + end + + subgraph Tray["Tray Icon (both modes)"] + C1[Tray init] --> C2[load_custom_icon] + C2 --> C3[include_bytes icon.png] + C3 --> C4[tray_icon::Icon] + end +``` + +## Permissions Required + +Grant in **System Settings > Privacy & Security**: + +| Permission | Purpose | When Prompted | +|------------|---------|---------------| +| Microphone | Audio recording | First recording attempt | +| Accessibility | Global hotkeys, paste | First hotkey press | +| Input Monitoring | Keyboard event capture | First hotkey press | + +## Troubleshooting + +### Empty Dock Icon +- **CLI mode**: `set_dock_icon()` should set it programmatically +- **Bundle mode**: Check that `Info.plist` exists and has `CFBundleIconFile` +- **Verify**: `plutil -lint /Applications/CodeScribe.app/Contents/Info.plist` + +### Empty Tray Icon +- Check that `assets/icon.png` exists and is valid PNG +- Rebuild with `cargo build --release` + +### Config Not Loading +- Check `~/.codescribe/.env` exists +- Verify syntax: `cat ~/.codescribe/.env` +- Check logs: `codescribe -v` for verbose output + +### Hotkeys Not Working +- Grant Accessibility permission +- Grant Input Monitoring permission +- Restart the application after granting + +--- + +*Copyright © 2024–2026 VetCoders* diff --git a/docs/ADR/2026-01-19-TEAM_SETUP.md b/docs/ADR/2026-01-19-TEAM_SETUP.md new file mode 100644 index 00000000..710a3ede --- /dev/null +++ b/docs/ADR/2026-01-19-TEAM_SETUP.md @@ -0,0 +1,181 @@ +# CodeScribe - Team Setup (Pure Rust Era) + +## Quick Start + +### 1. Prerequisites + +- macOS 14+ (Apple Silicon ARM64 only) +- Rust 1.83+ + +### 2. Build & Run (CLI) + +```bash +# Clone +git clone git@github.com:VetCoders/CodeScribe.git +cd CodeScribe + +# Build and run CLI +cargo build --release -p codescribe +./target/release/codescribe +``` + +### 3. Development Mode + +```bash +# Run debug binary +cargo run +``` + +## Permissions Required + +Grant in: System Settings > Privacy & Security + +1. **Microphone** - for audio recording +2. **Accessibility** - for global hotkeys +3. **Input Monitoring** - for hotkey capture + +## Hotkeys + +| Key | Action | AI Mode | +|----------------------------|-----------------------------------------|--------------------| +| Hold **Ctrl** | Record → paste raw transcript | ALWAYS RAW (no AI) | +| Hold **Ctrl+Shift** | Record → AI assistant response | ALWAYS Assistive | +| Double-tap **Option** | Toggle recording (hands-free) | Respects AI toggle | +| Triple-tap **Option** | Toggle AI Formatting on/off | Shows toast | +| **Shift** during Ctrl hold | Upgrade to Assistive mode mid-recording | — | + +### Mode Behavior + +- **RAW mode (Ctrl)**: Fast dictation. Transcript is pasted as-is (only local repetition cleanup). + Ignores AI_FORMATTING_ENABLED setting. +- **Toggle mode (Double Option)**: Respects the AI Formatting toggle. If enabled, sends to AI + for formatting. If disabled, pastes raw. +- **Assistive mode (Ctrl+Shift)**: Full AI assistant. Model can answer questions, expand ideas, + or pass through dictation based on detected intent (KURIER/ASYSTENT system). + +## Model + +**Strictly Embedded (Release Policy)**: `whisper-large-v3-turbo-mlx-q8` (~888MB) + +- **Zero Exceptions:** Release binaries ALWAYS contain the model. +- **No external files:** We never bundle `Resources/models/*`. +- **Zero I/O:** Model loads from memory directly to Metal. + +**Developer note (Build Time):** +You still need the model files locally to *build* the app (because they are `include_bytes!`-ed into the binary). + +```bash +make download-model # Required for build +``` + +Location (build-time only): `models/whisper-large-v3-turbo-mlx-q8/` + +## CLI Usage + +```bash +# Transcribe audio file +codescribe transcribe audio.wav + +# With AI formatting +codescribe transcribe audio.wav --format + +# Specify language +codescribe transcribe audio.wav --language pl +``` + +## Quality & Tools + +New CLI tools for batch processing and automation: + +```bash +# Batch quality report +codescribe-quality --help + +# Self-improving quality loop +codescribe-loop --help +``` + +## Configuration + +File: `~/.codescribe/.env` + +```env +USE_LOCAL_STT=1 + +# Whisper +WHISPER_LANGUAGE=pl + +# AI formatting (optional) - separate providers for formatting vs assistive +AI_FORMATTING_ENABLED=1 + +# Formatting mode (fast, cheap) - for Ctrl Hold with AI toggle +LLM_FORMATTING_ENDPOINT=https://api.libraxis.cloud/v1/responses +LLM_FORMATTING_MODEL=gpt-5-mini +LLM_FORMATTING_API_KEY=sk-xxx + +# Assistive mode (smart) - for Ctrl+Shift Hold +LLM_ASSISTIVE_ENDPOINT=https://api.libraxis.cloud/v1/responses +LLM_ASSISTIVE_MODEL=gpt-5.2 +LLM_ASSISTIVE_API_KEY=sk-xxx + +# Shared fallback (if mode-specific not set) +LLM_ENDPOINT=https://api.openai.com/v1/responses +LLM_MODEL=gpt-4.1-mini +LLM_API_KEY=sk-proj-xxx +``` + +### Custom Prompts + +Prompts are loaded from `~/.codescribe/prompts/` at each request (no restart needed): + +- `formatting.txt` - System prompt for formatting mode (punctuation, structure) +- `assistive.txt` - System prompt for assistive mode (KURIER/ASYSTENT logic) + +Edit these files to customize AI behavior. Changes take effect immediately. + +## Quality Assurance + +### Local (recommended) + +```bash +# Install pre-commit hooks (runs check/fmt on commit, clippy/semgrep on push) +make hooks + +# Manual quality gate +make check # fmt + clippy + unit tests + +# E2E tests with real API +make test-sse # SSE streaming tests (requires ~/.codescribe/.env) +``` + +### CI (GitHub Actions) + +**Note:** Full build requires macOS + Swift 6.0 (CoreML, Metal). GitHub runners have Swift 5.10, so CI only runs: + +- **Format check** (`cargo fmt --check`) on Linux +- **Semgrep** security scan on Linux + +Clippy and tests run **locally** via pre-commit hooks or `make check`. + +For full CI, configure a self-hosted macOS runner (Dragon recommended). + +## Troubleshooting + +### App doesn't start + +- Check Console.app for crash logs +- If building locally: ensure the model exists in `models/` (for embedding at build time) + +### Hotkeys don't work + +- Grant Accessibility permission +- Grant Input Monitoring permission +- Restart app after granting + +### No transcription + +- Check `USE_LOCAL_STT=1` in config +- If using local STT: confirm the app is using the embedded engine (default in release builds) + +--- +*Copyright © 2024–2026 VetCoders* diff --git a/docs/ADR/2026-01-25-ARCHITECTURE.md b/docs/ADR/2026-01-25-ARCHITECTURE.md new file mode 100644 index 00000000..46f9efb5 --- /dev/null +++ b/docs/ADR/2026-01-25-ARCHITECTURE.md @@ -0,0 +1,247 @@ +# CodeScribe Architecture + +> Created by M&K (c)2026 VetCoders + +## System Overview + +```mermaid +flowchart TB + %% High-level packaging / layers + + subgraph APP[codescribe crate - bin/daemon] + direction LR + HK[hotkeys.rs] + CTRL[controller/] + IPC_SERVER[ipc/server.rs] + TRAY[tray/] + OVERLAY[voice_chat_ui/] + + subgraph CORE[codescribe-core crate] + direction LR + WH[whisper/] + CO[config/] + AU[audio/] + IPC_CORE[ipc types] + end + + APP --> CORE + end + + WH --> MODEL[Whisper Model\nlarge-v3-turbo\nmlx-q8 ~888MB\nembedded in bin] + + subgraph TOOLS[Quality & CLI Tools] + CLI[codescribe-quality] + LOOP[codescribe-loop] + end + + APP -.-> TOOLS +``` + +## Module Architecture + +### Recording Flow + +``` +┌─────────────┐ ┌────────────┐ ┌───────────────┐ ┌──────────────┐ +│ CGEventTap │───►│ hotkeys.rs │───►│ controller/ │───►│ whisper/ │ +│ (macOS API) │ │ │ │ mod.rs │ │ engine.rs │ +└─────────────┘ └────────────┘ └───────────────┘ └──────────────┘ + │ │ │ + │ ▼ ▼ + │ ┌──────────────┐ ┌──────────────┐ + │ │ voice_chat │ │ transcription│ + │ │ _ui/ │ │ _overlay.rs │ + │ └──────────────┘ └──────────────┘ + │ + Ctrl hold → Raw mode (no AI) + Ctrl+Shift hold → Assistive mode (AI) + Double Option → Toggle mode (respects AI setting) +``` + +### Voice Chat UI (Mission Control) + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Status Header [Collapse] │ +├─────────────────────────────────────┬───────────────────────────┤ +│ LEFT PANEL (60%) │ RIGHT PANEL (40%) │ +│ │ │ +│ Chat bubbles (NSStackView) │ [Transcriptions][Settings]│ +│ ┌─────────────────────────────┐ │ │ +│ │ User message (blue, right) │ │ Draft files list │ +│ └─────────────────────────────┘ │ [Format] [Copy] [Augment] │ +│ ┌─────────────────────────┐ │ │ +│ │ AI response (gray,left) │ │ Settings toggles │ +│ └─────────────────────────┘ │ [Edit Config] [Edit Prompt]│ +│ │ │ +│ [Auto] [📎] [Input...] [Send] │ │ +└─────────────────────────────────────┴───────────────────────────┘ +``` + +## File Structure + +``` +CodeScribe/ +├── codescribe-core/ # Core library (portable, no macOS deps) +│ └── src/ +│ ├── whisper/ # Embedded + singleton Whisper engine +│ │ ├── engine.rs # Transcription logic +│ │ ├── singleton.rs # Global instance (lazy init) +│ │ └── embedded.rs # Model bytes (include_bytes!) +│ ├── audio/ # Recorder + StreamingRecorder +│ │ ├── recorder.rs # cpal audio capture +│ │ └── streaming_recorder.rs # Live transcription +│ ├── config/ # Configuration management +│ ├── stream_postprocess.rs # Semantic gating for live chunks +│ ├── quality_loop.rs # Self-improvement loop +│ ├── quality_report.rs # Batch quality reports +│ └── ipc/ # IPC types +│ +├── src/ # codescribe crate (macOS-specific) +│ ├── main.rs # CLI entry (daemon/transcribe) +│ ├── lib.rs # Library exports +│ │ +│ ├── controller/ # Recording state machine +│ │ ├── mod.rs # RecordingController impl +│ │ ├── types.rs # State, HotkeyInput, etc. +│ │ ├── helpers.rs # Session state, callbacks +│ │ └── tests.rs # Controller tests +│ │ +│ ├── voice_chat_ui/ # Voice Chat Overlay (Mission Control) +│ │ ├── mod.rs # UI creation (AppKit) +│ │ ├── state.rs # VoiceChatOverlayState +│ │ ├── handlers.rs # Objective-C callbacks +│ │ └── api.rs # Public API functions +│ │ +│ ├── tray/ # System tray menu +│ │ ├── mod.rs # Tray setup +│ │ ├── menu.rs # Menu creation +│ │ ├── handlers.rs # Menu actions +│ │ ├── icons.rs # Icon generation +│ │ └── types.rs # MenuIds, TrayMenuEvent +│ │ +│ ├── hotkeys.rs # CGEventTap handler +│ ├── transcription_overlay.rs # Simple text overlay +│ ├── ui.rs # Badge, Dock icon +│ ├── ui_helpers.rs # AppKit utilities +│ ├── clipboard.rs # Paste to active app +│ ├── permissions.rs # macOS permission checks +│ └── ipc/ # IPC server (Unix socket) +│ +├── src/bin/ # CLI tools +│ ├── codescribe_quality.rs # Batch quality reports +│ └── codescribe_loop.rs # Self-improving loop +│ +├── docs/ +│ ├── guide/ # User documentation +│ │ ├── README.md # Quick start +│ │ ├── installation.md +│ │ ├── modes.md +│ │ ├── chat-overlay.md +│ │ ├── settings.md +│ │ ├── troubleshooting.md +│ │ └── privacy.md +│ ├── ARCHITECTURE.md # This file +│ ├── WHISPER_LIVE.md # Streaming transcription +│ ├── TEAM_SETUP.md # Developer setup +│ └── future/ # Aspirational docs +│ ├── ARCHITECTURE_VISION.md +│ └── FEASIBILITY_ANALYSIS.md +│ +└── tests/ # Integration tests +``` + +## Key Components + +### Controller State Machine + +```rust +// src/controller/types.rs +pub enum State { + Idle, // Ready for input + RecHold, // Recording (hold mode) + RecToggle, // Recording (toggle mode) + Busy, // Processing transcription +} +``` + +State transitions: +- `Idle` + Ctrl down → (800ms delay) → `RecHold` +- `Idle` + Double Option → `RecToggle` +- `RecHold` + Ctrl up → `Busy` → `Idle` +- `RecToggle` + Double Option → `Busy` → `Idle` +- `RecToggle` + 5s silence (VAD) → `Busy` → `Idle` + +### Mode Determination + +```rust +// src/controller/mod.rs - handle_hotkey_event() +match (hotkey, flags) { + (Hold, no_shift) => force_raw = true, // Ctrl: always raw + (Hold, shift) => assistive = true, // Ctrl+Shift: always AI + (Toggle, force_ai)=> force_ai = true, // Left Option x2: force AI + (Toggle, _) => /* respects AI_FORMATTING_ENABLED */ +} +``` + +### Voice Chat UI Components + +| Module | LOC | Purpose | +|--------|-----|---------| +| `mod.rs` | 632 | UI creation with AppKit | +| `api.rs` | 589 | Public API (update_status, etc.) | +| `handlers.rs` | 450 | Objective-C action handlers | +| `state.rs` | 148 | VoiceChatOverlayState struct | + +### Whisper Engine + +- **Singleton pattern**: One global instance, lazy initialized +- **Metal acceleration**: Uses Apple GPU via candle-core +- **Streaming**: Chunks processed during recording +- **Embedded**: Model bytes in binary (~888MB) + +## Implementation Status + +| Feature | Status | +|---------|--------| +| Local Whisper STT (Metal GPU) | ✅ | +| Embedded model (~888MB binary) | ✅ | +| Global hotkeys (CGEventTap) | ✅ | +| Three recording modes (Raw/Assistive/Toggle) | ✅ | +| Voice Chat UI (split panel) | ✅ | +| Chat bubbles (NSStackView) | ✅ | +| Drafts panel with tabs | ✅ | +| Settings in overlay | ✅ | +| AI formatting (Responses API) | ✅ | +| Streaming AI responses | ✅ | +| Tray app with submenus | ✅ | +| History with slug filenames | ✅ | +| IPC server (runtime interface) | ✅ | +| Stream postprocess (semantic gating) | ✅ | +| Quality loop + report | ✅ | +| CodeScribe Core separation | ✅ | +| VAD (utterance boundary on silence) | ✅ | +| Transcription overlay | ✅ | +| Tauri GUI (future) | 📋 | + +## Model Location + +**Release Builds**: Model embedded via `include_bytes!` (~888MB total). +Zero disk I/O, model bytes loaded directly into GPU memory. + +**Development**: External model from: +1. `CODESCRIBE_MODEL_PATH` environment variable +2. `~/.codescribe/models/whisper-large-v3-turbo-mlx-q8/` +3. `./models/whisper-large-v3-turbo-mlx-q8/` in repo + +## Related Documentation + +- [`guide/README.md`](guide/README.md) — User documentation +- [`WHISPER_LIVE.md`](WHISPER_LIVE.md) — Embedded + streaming transcription +- [`TEAM_SETUP.md`](TEAM_SETUP.md) — Developer setup guide +- [`BACKLOG.md`](BACKLOG.md) — Feature backlog +- [`future/ARCHITECTURE_VISION.md`](future/ARCHITECTURE_VISION.md) — Libraxis Qube Protocol vision + +--- + +**Made with ⌜ CodeScribe ⌟ by Maciej & Monika + Klaudiusz (AI) (c) 2024-2026** diff --git a/docs/ADR/2026-01-25-ARCHITECTURE_VISION.md b/docs/ADR/2026-01-25-ARCHITECTURE_VISION.md new file mode 100644 index 00000000..e572c0bb --- /dev/null +++ b/docs/ADR/2026-01-25-ARCHITECTURE_VISION.md @@ -0,0 +1,186 @@ +# Architecture Vision: The Libraxis Qube Protocol + +> **Status:** Draft / Conceptual +> **Date:** 2026-01-19 +> **Author:** Junie (AI) for VetCoders + +## 1. Core Concept: The Libraxis Qube + +We are renaming the central orchestration node (formerly "CodeScribe Core") to **The Libraxis Qube**. + +The **Libraxis Qube** is a central, location-agnostic **Stream Router & Orchestrator**. It is not just a backend for an app; +it is the "infinite cube" that holds the state of the conversation and manages all flows of information (Audio, Text, +Artifacts). + +### Key Philosophy: Deployment Neutrality + +The architecture removes the distinction between "Local" and "Cloud". + +- **Distance is irrelevant.** +- The Libraxis Qube can run on `localhost` (your laptop) or on `Dragon` (remote workstation). +- The **Client** (Microphone/Speaker) connects to the Libraxis Qube via a **WebSocket**. +- Whether that WebSocket connects to `ws://127.0.0.1:8000` or `wss://dragon.lan:8000` changes nothing in the system + logic. It only changes the latency of the wire. + +## 2. System Topology + +```mermaid +graph TD + subgraph Client [User / Client Side] + Mic[Microphone] + Speaker[Speaker] + UI[Visual Overlay/Logs] + end + + subgraph Libraxis QubeNode [The Libraxis Qube Node] + Router[Libraxis Qube Orchestrator
(State, Routing, Demux)] + + subgraph Modules [Attached Modules] + ASR[Candle Transformers
(Whisper v3 Turbo - Streaming)] + Agent[LLM Agent
(Reasoning & Tools)] + TTS[TTS Engine
(Voice Synthesis)] + Artifacts[Artifact Generator
(PDF, Files, JSON)] + end + end + + %% Connections + Mic ==>|"WebSocket Stream (Audio Bytes)"| Router + Router ==>|"WebSocket Stream (Audio/Events)"| Speaker + Router -.->|"Updates (SSE/WS)"| UI + + %% Internal Flows + Router <==>|"Raw Audio / Tokens"| ASR + Router <==>|"Context / Intention"| Agent + Router ==>|"Text to Speak"| TTS + TTS ==>|"Audio Bytes"| Router + Router ==>|"Content"| Artifacts +``` + +## 3. The "Single Stream" Protocol + +To maintain synchronization and simplicity, the Agent/Model does not open multiple TCP connections. Instead, it emits a +**single continuous stream** of tokens. The Libraxis Qube is responsible for **demuxing** (splitting) this stream based on +**Tags**. + +### Stream Structure + +The stream contains interspersed content types, delimited by XML-like tags or special markers. + +**Example Stream Flow (Server to Client):** + +```text +[STREAM_START] +... (thinking tokens, internal log) ... +Here is the summary of the patient's condition. +[ROUTER DETECTS TAG -> Routes content to TTS -> Sends Audio Bytes to Client] +... + + { "patient": "Fretka Ziggy", "diagnosis": "Healthy" } + +[ROUTER DETECTS TAG -> Routes content to PDF Generator -> Saves File -> Notifies Client] +... +Displaying chart... +[ROUTER DETECTS TAG -> Routes to Overlay] +... +[STREAM_END] +``` + +### Routing Logic (The Demuxer) + +1. **Default Channel**: Tokens flow to the internal buffer/context. +2. **Audio Channel (`` / `