From 95be92886b2cc0a17597d2601eead9a875fe9e7b Mon Sep 17 00:00:00 2001 From: Ankush Malaker <43288948+AnkushMalaker@users.noreply.github.com> Date: Tue, 14 Jul 2026 22:22:05 +0000 Subject: [PATCH] docs: consolidate documentation under lowercase docs --- .easignore | 2 +- CLAUDE.md => AGENTS.md | 36 +- README-K8S.md | 3 +- README.md | 6 +- backends/advanced/Docs/README.md | 191 --- .../Docs/README_speaker_enrollment.md | 292 ----- backends/advanced/Docs/UI.md | 244 ---- backends/advanced/Docs/architecture.md | 1077 ----------------- backends/advanced/Docs/contribution.md | 35 - backends/advanced/Docs/flowdiagram.excalidraw | 856 ------------- backends/advanced/README.md | 4 +- backends/advanced/init.py | 4 +- .../workers/conversation_jobs.py | 2 +- config/README.md | 4 +- config/plugins.yml.template | 2 +- docs/README.md | 24 + {Docs => docs}/audio-pipeline-architecture.md | 0 .../advanced/Docs => docs/backend}/auth.md | 0 .../Docs => docs/backend}/memories.md | 2 +- .../backend}/plugin-configuration.md | 8 +- .../backend}/plugin-development-guide.md | 0 {Docs => docs}/init-system.md | 2 +- {Docs => docs}/overview.md | 5 +- {Docs => docs}/podman.md | 2 +- {Docs => docs}/ssl-certificates.md | 0 extras/speaker-recognition/README.md | 5 +- plugins/email_summarizer/README.md | 3 +- quickstart.md | 6 +- tests/setup/README.md | 2 +- 29 files changed, 71 insertions(+), 2746 deletions(-) rename CLAUDE.md => AGENTS.md (94%) delete mode 100644 backends/advanced/Docs/README.md delete mode 100644 backends/advanced/Docs/README_speaker_enrollment.md delete mode 100644 backends/advanced/Docs/UI.md delete mode 100644 backends/advanced/Docs/architecture.md delete mode 100644 backends/advanced/Docs/contribution.md delete mode 100644 backends/advanced/Docs/flowdiagram.excalidraw create mode 100644 docs/README.md rename {Docs => docs}/audio-pipeline-architecture.md (100%) rename {backends/advanced/Docs => docs/backend}/auth.md (100%) rename {backends/advanced/Docs => docs/backend}/memories.md (98%) rename {backends/advanced/Docs => docs/backend}/plugin-configuration.md (97%) rename {backends/advanced/Docs => docs/backend}/plugin-development-guide.md (100%) rename {Docs => docs}/init-system.md (99%) rename {Docs => docs}/overview.md (96%) rename {Docs => docs}/podman.md (99%) rename {Docs => docs}/ssl-certificates.md (100%) diff --git a/.easignore b/.easignore index ea15bfd9..a5360919 100644 --- a/.easignore +++ b/.easignore @@ -6,7 +6,7 @@ backends/ extras/ tests/ -Docs/ +docs/ sdk/ untracked/ diff --git a/CLAUDE.md b/AGENTS.md similarity index 94% rename from CLAUDE.md rename to AGENTS.md index 7e690c01..e07cf736 100644 --- a/CLAUDE.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ -# CLAUDE.md +# AGENTS.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file provides guidance to coding agents working in this repository. ## Project Overview @@ -40,7 +40,7 @@ uv run --with-requirements setup-requirements.txt python wizard.py ### Setup Documentation For detailed setup instructions and troubleshooting, see: - **[@quickstart.md](quickstart.md)**: Beginner-friendly step-by-step setup guide -- **[@Docs/init-system.md](Docs/init-system.md)**: Complete initialization system architecture and design +- **[@docs/init-system.md](docs/init-system.md)**: Complete initialization system architecture and design ### Wizard Architecture The initialization system uses a **root orchestrator pattern**: @@ -552,7 +552,7 @@ tailscale ip -4 ### Package Management - **Backend**: Uses `uv` for Python dependency management (faster than pip) - **Mobile**: Uses `npm` with React Native and Expo -- **Containers**: The service lifecycle (`services.py`/`status.py`/service-manager) supports **both Docker and Podman**, selected via `container_engine: docker|podman` in `config/config.yml` (or the `CONTAINER_ENGINE`/`COMPOSE_CMD` env vars). The repo's compose files run unmodified under either engine. For Podman it drives `podman-compose` and needs CDI for GPU. See **[@Docs/podman.md](Docs/podman.md)** for rootless/GPU setup and migration notes. +- **Containers**: The service lifecycle (`services.py`/`status.py`/service-manager) supports **both Docker and Podman**, selected via `container_engine: docker|podman` in `config/config.yml` (or the `CONTAINER_ENGINE`/`COMPOSE_CMD` env vars). The repo's compose files run unmodified under either engine. For Podman it drives `podman-compose` and needs CDI for GPU. See **[@docs/podman.md](docs/podman.md)** for rootless/GPU setup and migration notes. ### Testing Strategy - **Makefile-Based**: All test operations through simple `make` commands (`make test`, `make start`, `make stop`) @@ -592,22 +592,18 @@ The system includes comprehensive health checks: - **Log Preservation**: All cleanup operations save logs to `tests/logs/` automatically - **CI Compatibility**: Same test logic runs locally and in GitHub Actions -### Cursor Rule Integration -Project includes `.cursor/rules/always-plan-first.mdc` requiring understanding before coding. Always explain the task and confirm approach before implementation. - ## Extended Documentation For detailed technical documentation, see: -- **[@Docs/overview.md](Docs/overview.md)**: Architecture overview and technical deep dive -- **[@Docs/init-system.md](Docs/init-system.md)**: Initialization system and service management -- **[@Docs/ssl-certificates.md](Docs/ssl-certificates.md)**: HTTPS/SSL setup details -- **[@Docs/podman.md](Docs/podman.md)**: Running with Podman instead of Docker (engine selection, rootless/GPU setup) -- **[@Docs/audio-pipeline-architecture.md](Docs/audio-pipeline-architecture.md)**: Audio pipeline design -- **[@backends/advanced/Docs/auth.md](backends/advanced/Docs/auth.md)**: Authentication architecture -- **[backends/advanced/Docs/architecture.md](backends/advanced/Docs/architecture.md)**: Backend architecture details -- **[backends/advanced/Docs/memories.md](backends/advanced/Docs/memories.md)**: Memory system documentation -- **[backends/advanced/Docs/data-audit.md](backends/advanced/Docs/data-audit.md)**: Data Audit (VAD analysis, split/merge, audio preview) -- **[backends/advanced/Docs/plugin-development-guide.md](backends/advanced/Docs/plugin-development-guide.md)**: Plugin development guide +- **[@docs/README.md](docs/README.md)**: Documentation index +- **[@docs/overview.md](docs/overview.md)**: Architecture overview and technical deep dive +- **[@docs/init-system.md](docs/init-system.md)**: Initialization system and service management +- **[@docs/ssl-certificates.md](docs/ssl-certificates.md)**: HTTPS/SSL setup details +- **[@docs/podman.md](docs/podman.md)**: Running with Podman instead of Docker (engine selection, rootless/GPU setup) +- **[@docs/audio-pipeline-architecture.md](docs/audio-pipeline-architecture.md)**: Audio pipeline design +- **[@docs/backend/auth.md](docs/backend/auth.md)**: Authentication architecture +- **[@docs/backend/memories.md](docs/backend/memories.md)**: Memory system documentation +- **[@docs/backend/plugin-development-guide.md](docs/backend/plugin-development-guide.md)**: Plugin development guide ## Robot Framework Testing @@ -634,14 +630,14 @@ Key Testing Rules: - Use space-separated tags (must be tab-separated) - Skip reading the guidelines before writing tests -## Notes for Claude +## Notes for Coding Agents Check if the src/ is volume mounted. If not, do compose build so that code changes are reflected. Do not simply run `docker compose restart` as it will not rebuild the image. -Check backends/advanced/Docs for up to date information on advanced backend. +Check `docs/backend/` for up-to-date information on the advanced backend. All docker projects have .dockerignore following the exclude pattern. That means files need to be included for them to be visible to docker. The uv package manager is used for all python projects. Wherever you'd call `python3 main.py` you'd call `uv run python main.py` **Container Engine (Docker or Podman):** -- The project supports **both Docker and Podman**. The active engine is set by `container_engine` in `config/config.yml` (default `docker`); prefer the lifecycle scripts (`./start.sh`/`./stop.sh`/`./restart.sh`) which route through the selected engine. For one-off manual commands under Podman use `podman-compose` (not `docker compose`). See **[@Docs/podman.md](Docs/podman.md)**. +- The project supports **both Docker and Podman**. The active engine is set by `container_engine` in `config/config.yml` (default `docker`); prefer the lifecycle scripts (`./start.sh`/`./stop.sh`/`./restart.sh`) which route through the selected engine. For one-off manual commands under Podman use `podman-compose` (not `docker compose`). See **[@docs/podman.md](docs/podman.md)**. **Docker Build Guidelines:** - Use `docker compose build` (or `podman-compose build`) without `--no-cache` by default for faster builds diff --git a/README-K8S.md b/README-K8S.md index aa3509a0..5a243400 100644 --- a/README-K8S.md +++ b/README-K8S.md @@ -817,8 +817,7 @@ This script handles speaker recognition service deployment with proper environme For additional support: - Check the main [README.md](README.md) -- Review [CLAUDE.md](CLAUDE.md) for development notes -- Check [README-skaffold.md](README-skaffold.md) for Skaffold-specific information +- Review [AGENTS.md](AGENTS.md) for development notes --- diff --git a/README.md b/README.md index 550691d6..a775f83f 100644 --- a/README.md +++ b/README.md @@ -38,8 +38,8 @@ For step-by-step instructions, see the [setup guide](quickstart.md). ## Links - **๐Ÿ“š [Setup Guide](quickstart.md)** - Start here -- **๐Ÿ”ง [Full Documentation](CLAUDE.md)** - Comprehensive reference -- **๐Ÿ—๏ธ [Project Overview](Docs/overview.md)** - Architecture and vision +- **๐Ÿ”ง [Full Documentation](AGENTS.md)** - Comprehensive reference +- **๐Ÿ—๏ธ [Project Overview](docs/overview.md)** - Architecture and vision - **๐Ÿณ [Docker/K8s](README-K8S.md)** - Container deployment ## Project Structure @@ -60,7 +60,7 @@ chronicle/ โ”‚ โ”œโ”€โ”€ speaker-recognition/ # Voice identification service โ”‚ โ”œโ”€โ”€ asr-services/ # Offline speech-to-text (Parakeet) โ”‚ โ””โ”€โ”€ openmemory-mcp/ # External memory server -โ”œโ”€โ”€ Docs/ # Technical documentation +โ”œโ”€โ”€ docs/ # Technical documentation โ”œโ”€โ”€ config/ # Central configuration files โ”œโ”€โ”€ tests/ # Integration & unit tests โ”œโ”€โ”€ wizard.py # Root setup orchestrator diff --git a/backends/advanced/Docs/README.md b/backends/advanced/Docs/README.md deleted file mode 100644 index 09bc934a..00000000 --- a/backends/advanced/Docs/README.md +++ /dev/null @@ -1,191 +0,0 @@ -# Chronicle Backend Documentation Guide - -## ๐Ÿ“– **New Developer Reading Order** - -Welcome to chronicle! This guide provides the optimal reading sequence to understand the complete voice โ†’ transcript โ†’ memories system. - ---- - -## ๐ŸŽฏ **Start Here: System Overview** - -### 1. **[Overview & Quick Start](./quickstart.md)** โญ *START HERE* -**Read first** - Complete system overview and setup guide -- What the system does (voice โ†’ memories) -- Key features and capabilities -- Basic setup and configuration -- **Code References**: `src/advanced_omi_backend/main.py`, `config/config.yml`, `docker-compose.yml` - -### 2. **[System Architecture](./architecture.md)** -**Read second** - Complete technical architecture with diagrams -- Component relationships and data flow -- Authentication and security architecture -- Deployment structure and containers -- **Code References**: `src/advanced_omi_backend/main.py`, `src/advanced_omi_backend/auth.py`, `src/advanced_omi_backend/users.py` - ---- - -## ๐Ÿ”ง **Core Components Deep Dive** - -### 3. **[Memory System](./memories.md)** -**Memory extraction and vault retrieval** - -- How conversations become memories -- Agentic Markdown vault (the single source of truth) -- Configuration and customization options -- **Code References**: - - `src/advanced_omi_backend/memory/memory_service.py` (main processing) - - `src/advanced_omi_backend/transcript_coordinator.py` (event coordination) - - `src/advanced_omi_backend/conversation_repository.py` (data access) - - `src/advanced_omi_backend/conversation_manager.py` (lifecycle management) - -### 4. **[Authentication System](./auth.md)** -**User management and security** -- Dual authentication (email + user_id) -- JWT tokens and OAuth integration -- User-centric data architecture -- **Code References**: - - `src/advanced_omi_backend/auth.py` (authentication logic) - - `src/advanced_omi_backend/users.py` (user management) - - `src/advanced_omi_backend/routers/api_router.py` (API router setup) - ---- - -## ๐Ÿ› **Advanced Topics** - -### 5. **Memory Debug System** โ†’ `../MEMORY_DEBUG_IMPLEMENTATION.md` -**Pipeline tracking and debugging** -- How to track transcript โ†’ memory conversion -- Debug database schema and API endpoints -- Performance monitoring and troubleshooting -- **Code References**: - - `src/advanced_omi_backend/routers/modules/system_routes.py` (debug endpoints) - ---- - -## ๐Ÿ” **Configuration & Customization** - -### 6. **Configuration File** โ†’ `../config/config.yml` -**Central configuration for all extraction** -- Memory extraction settings and prompts -- Quality control and debug settings -- **Code References**: - - `src/advanced_omi_backend/model_registry.py` (config loading) - - `src/advanced_omi_backend/memory/memory_service.py` (config usage) - ---- - -## ๐Ÿš€ **Quick Reference by Use Case** - -### **"I want to understand the system quickly"** (30 min) -1. [quickstart.md](./quickstart.md) - System overview -2. [architecture.md](./architecture.md) - Technical architecture -3. `src/advanced_omi_backend/main.py` - Core imports and setup -4. `config/config.yml` - Configuration overview - -### **"I want to work on memory extraction"** -1. [memories.md](./memories.md) - Memory system details -2. `../config/config.yml` - Models and memory configuration -3. `src/advanced_omi_backend/memory/memory_service.py` - Implementation -4. `src/advanced_omi_backend/controllers/memory_controller.py` - Processing triggers - -### **"I want to debug pipeline issues"** -1. `../MEMORY_DEBUG_IMPLEMENTATION.md` - Debug system overview -3. API: `GET /api/debug/memory/stats` - Live debugging -4. `src/advanced_omi_backend/routers/modules/system_routes.py` - Debug endpoints - -### **"I want to understand authentication"** -1. [auth.md](./auth.md) - Authentication system -2. `src/advanced_omi_backend/auth.py` - Authentication implementation -3. `src/advanced_omi_backend/users.py` - User management -4. `src/advanced_omi_backend/routers/api_router.py` - Auth router setup - ---- - -## ๐Ÿ“‚ **File Organization Reference** - -``` -backends/advanced-backend/ -โ”œโ”€โ”€ Docs/ # ๐Ÿ“– Documentation -โ”‚ โ”œโ”€โ”€ README.md # ๐Ÿ‘ˆ This file (start here) -โ”‚ โ”œโ”€โ”€ quickstart.md # System overview & setup -โ”‚ โ”œโ”€โ”€ architecture.md # Technical architecture -โ”‚ โ”œโ”€โ”€ memories.md # Memory system details -โ”‚ โ””โ”€โ”€ auth.md # Authentication system -โ”‚ -โ”œโ”€โ”€ src/advanced_omi_backend/ # ๐Ÿ”ง Source Code -โ”‚ โ”œโ”€โ”€ main.py # Core application (WebSocket) -โ”‚ โ”œโ”€โ”€ auth.py # Authentication system -โ”‚ โ”œโ”€โ”€ users.py # User management -โ”‚ โ”œโ”€โ”€ routers/ # API route definitions -โ”‚ โ”‚ โ”œโ”€โ”€ api_router.py # Main API router -โ”‚ โ”‚ โ””โ”€โ”€ modules/ # Modular route organization -โ”‚ โ”œโ”€โ”€ controllers/ # Business logic controllers -โ”‚ โ”œโ”€โ”€ memory/ -โ”‚ โ”‚ โ””โ”€โ”€ memory_service.py # Memory system (Mem0) -โ”‚ โ””โ”€โ”€ model_registry.py # Configuration loading -โ”‚ -โ”œโ”€โ”€ config/config.yml # ๐Ÿ“‹ Central configuration -โ”œโ”€โ”€ MEMORY_DEBUG_IMPLEMENTATION.md # Debug system details -``` - ---- - -## ๐ŸŽฏ **Key Code Entry Points** - -### **Audio Processing Pipeline** -- **Entry**: WebSocket endpoints in `src/advanced_omi_backend/main.py` -- **Transcription**: Audio processing pipeline in `src/advanced_omi_backend/processors.py` -- **Memory Trigger**: Memory processing in `src/advanced_omi_backend/controllers/memory_controller.py` - -### **Data Storage** -- **Memories**: `src/advanced_omi_backend/services/memory/providers/chronicle.py` โ†’ agentic Markdown vault at `data/conversation_docs//` (the single source of truth) - -### **Configuration** -- **Loading**: `src/advanced_omi_backend/model_registry.py` -- **File**: `config/config.yml` -- **Usage**: `src/advanced_omi_backend/memory/memory_service.py` - -### **Authentication** -- **Setup**: `src/advanced_omi_backend/auth.py` -- **Users**: `src/advanced_omi_backend/users.py` -- **Integration**: `src/advanced_omi_backend/routers/api_router.py` - ---- - -## ๐Ÿ’ก **Reading Tips** - -1. **Follow the references**: Each doc links to specific code files and line numbers -2. **Use the debug API**: `GET /api/debug/memory/stats` shows live system status -3. **Check configuration first**: Many behaviors are controlled by `config/config.yml` -4. **Understand the memory pipeline**: Memories (end-of-conversation) -5. **Test with curl**: All API endpoints have curl examples in the docs - ---- - -## ๐ŸŽฏ **After Reading This Guide** - -### **Next Steps for New Developers** - -1. **Set up the system**: Follow [quickstart.md](./quickstart.md) to get everything running -2. **Test the API**: Use the curl examples in the documentation to test endpoints -3. **Explore the debug system**: Check `GET /api/debug/memory/stats` to see live data -4. **Modify configuration**: Edit `config/config.yml` (memory section) to see how it affects extraction -5. **Read the code**: Start with `src/advanced_omi_backend/main.py` and follow the references in each doc - -### **Contributing Guidelines** - -- **Add code references**: When updating docs, include file paths and line numbers -- **Test your changes**: Use the debug API to verify your modifications work -- **Update configuration**: Add new settings to `config/config.yml` when needed -- **Follow the architecture**: Keep memories in their respective services - -### **Getting Help** - -- **Debug API**: `GET /api/debug/memory/*` endpoints show real-time system status -- **Configuration**: Check `config/config.yml` for behavior controls -- **Logs**: Check Docker logs with `docker compose logs chronicle-backend` -- **Documentation**: Each doc file links to relevant code sections - ---- - -This documentation structure ensures you understand both the **big picture** and **implementation details** in a logical progression! diff --git a/backends/advanced/Docs/README_speaker_enrollment.md b/backends/advanced/Docs/README_speaker_enrollment.md deleted file mode 100644 index de3fa736..00000000 --- a/backends/advanced/Docs/README_speaker_enrollment.md +++ /dev/null @@ -1,292 +0,0 @@ -# Speaker Recognition and Enrollment Guide - -The advanced backend now includes sophisticated speaker recognition functionality using pyannote.audio for diarization and SpeechBrain for speaker embeddings. This guide shows you how to use the speaker enrollment and identification features. - -## Overview - -The speaker recognition system provides: - -1. **Speaker Diarization**: Automatically detect and separate different speakers in audio -2. **Speaker Enrollment**: Register known speakers with audio samples -3. **Speaker Identification**: Identify enrolled speakers in new audio -4. **API Endpoints**: RESTful API for all speaker operations -5. **Command Line Tools**: Easy-to-use scripts for speaker management - -## Setup and Requirements - -### Environment Variables - -Make sure you have your HuggingFace token set for pyannote.audio models: - -```bash -export HF_TOKEN="your_huggingface_token_here" -``` - -You can get a token from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). - -### Python Dependencies - -The speaker recognition system requires additional packages. Install them with: - -```bash -# For audio recording (optional) -pip install sounddevice soundfile - -# For API calls -pip install aiohttp requests - -# Core dependencies (should already be installed) -# pyannote.audio, speechbrain, faiss-cpu, scipy -``` - -## Speaker Enrollment - -### Method 1: Using the Enrollment Script - -The easiest way to enroll speakers is using the provided script: - -```bash -# Navigate to the backend directory -cd backends/advanced-backend - -# List currently enrolled speakers -python enroll_speaker.py --list - -# Enroll a speaker from an existing audio file -python enroll_speaker.py --id alice --name "Alice Smith" --file "audio_chunk_file.wav" - -# Enroll from a specific segment of an audio file (useful for clean speech) -python enroll_speaker.py --id bob --name "Bob Jones" --file "recording.wav" --start 10.0 --end 15.0 - -# Record new audio and enroll (requires microphone) -python enroll_speaker.py --id charlie --name "Charlie Brown" --record --duration 5.0 - -# Test identification on an audio file -python enroll_speaker.py --identify "test_audio.wav" -``` - -### Method 2: Using the API Directly - -You can also use the REST API endpoints: - -```bash -# Enroll a speaker -curl -X POST "http://localhost:8000/api/speakers/enroll" \ - -H "Content-Type: application/json" \ - -d '{ - "speaker_id": "alice", - "speaker_name": "Alice Smith", - "audio_file_path": "audio_chunk_file.wav" - }' - -# List enrolled speakers -curl "http://localhost:8000/api/speakers" - -# Get specific speaker info -curl "http://localhost:8000/api/speakers/alice" - -# Identify speaker from audio -curl -X POST "http://localhost:8000/api/speakers/identify" \ - -H "Content-Type: application/json" \ - -d '{ - "audio_file_path": "test_audio.wav" - }' - -# Remove a speaker -curl -X DELETE "http://localhost:8000/api/speakers/alice" -``` - -## Integration with Laptop Client - -The laptop client (`laptop_client.py`) can be used to create audio for speaker enrollment: - -### Step 1: Record Audio with Laptop Client - -```bash -# Start the backend server -python main.py - -# In another terminal, record audio with a specific user ID -python laptop_client.py --user-id alice_recording - -# Speak for 10-30 seconds, then stop the client (Ctrl+C) -``` - -This will create audio chunks in the `audio_chunks/` directory. - -### Step 2: Enroll Speaker from Recorded Audio - -```bash -# Find the audio file created (check audio_chunks/ directory) -ls audio_chunks/ - -# Enroll the speaker using one of the audio chunks -python enroll_speaker.py --id alice --name "Alice" --file "audio_chunk_alice_recording_12345.wav" -``` - -### Step 3: Test Recognition - -```bash -# Record new audio with the same speaker -python laptop_client.py --user-id test_recognition - -# Test identification -python enroll_speaker.py --identify "audio_chunk_test_recognition_67890.wav" -``` - -## How Speaker Recognition Works - -### During Audio Processing - -1. **Diarization**: When audio is processed, pyannote.audio separates different speakers -2. **Embedding Extraction**: For each speaker segment, a SpeechBrain embedding is computed -3. **Speaker Identification**: Embeddings are compared against enrolled speakers using FAISS -4. **Database Storage**: Results are stored in MongoDB with speaker assignments - -### Speaker Enrollment Process - -1. **Audio Loading**: Load audio file (optionally cropped to specific segment) -2. **Embedding Extraction**: Generate speaker embedding using SpeechBrain -3. **Normalization**: L2-normalize embedding for cosine similarity -4. **FAISS Storage**: Add embedding to FAISS index for fast similarity search -5. **Database Storage**: Store speaker metadata in MongoDB - -### Identification Process - -1. **Embedding Extraction**: Generate embedding from unknown audio -2. **Similarity Search**: Search FAISS index for most similar enrolled speaker -3. **Threshold Check**: Only identify if similarity > 0.85 (configurable) -4. **Return Result**: Return speaker ID if identified, or "unknown" if not - -## API Endpoints Reference - -| Method | Endpoint | Description | -|--------|----------|-------------| -| POST | `/api/speakers/enroll` | Enroll a new speaker | -| GET | `/api/speakers` | List all enrolled speakers | -| GET | `/api/speakers/{speaker_id}` | Get speaker details | -| DELETE | `/api/speakers/{speaker_id}` | Remove a speaker | -| POST | `/api/speakers/identify` | Identify speaker from audio | - -## Configuration - -### Speaker Recognition Settings - -Edit `speaker_recognition/speaker_recognition.py` to adjust: - -- `SIMILARITY_THRESHOLD = 0.85`: Cosine similarity threshold for identification -- `device`: CUDA device for GPU acceleration -- Embedding model: Currently uses `speechbrain/spkrec-ecapa-voxceleb` -- Diarization model: Currently uses `pyannote/speaker-diarization-community-1` - -### Audio Settings - -The system supports: -- Sample rate: Dynamic detection (commonly 16kHz, 44.1kHz, or 48kHz) -- Channels: Mono (stereo converted to mono automatically) -- Format: WAV files (recommended), WebM, MP4 - -## Troubleshooting - -### Common Issues - -1. **HuggingFace Token Issues** - ``` - Error: pyannote models require authentication - Solution: Set HF_TOKEN environment variable - ``` - -2. **CUDA Out of Memory** - ``` - Error: CUDA out of memory - Solution: Reduce batch size or use CPU-only mode - ``` - -3. **Audio File Not Found** - ``` - Error: Audio file not found - Solution: Ensure audio files are in audio_chunks/ directory - ``` - -4. **Poor Recognition Accuracy** - ``` - Issue: Speakers not being identified correctly - Solutions: - - Use cleaner audio for enrollment (less background noise) - - Enroll with longer audio segments (5-10 seconds) - - Lower similarity threshold if needed - ``` - -### Debug Mode - -Enable debug logging by setting: - -```bash -export PYTHONPATH=/path/to/backend -python -c " -import logging -logging.basicConfig(level=logging.DEBUG) -# your code here -" -``` - -## Performance Notes - -- **GPU Acceleration**: Enable CUDA for faster processing -- **Memory Usage**: ~500MB for models, ~4MB per 1000 enrolled speakers -- **Processing Speed**: ~2-5x real-time on GPU, ~0.5x real-time on CPU -- **Accuracy**: >95% for clean speech, >85% for noisy environments - -## Advanced Usage - -### Batch Enrollment - -```python -import asyncio -from enroll_speaker import enroll_speaker_api - -async def batch_enroll(): - speakers = [ - ("alice", "Alice Smith", "alice.wav"), - ("bob", "Bob Jones", "bob.wav"), - ("charlie", "Charlie Brown", "charlie.wav") - ] - - for speaker_id, name, file in speakers: - await enroll_speaker_api("localhost", 8000, speaker_id, name, file) - -asyncio.run(batch_enroll()) -``` - -### Custom Similarity Threshold - -```python -import speaker_recognition -speaker_recognition.SIMILARITY_THRESHOLD = 0.75 # More permissive -``` - -### Integration with Other Systems - -The speaker recognition module can be imported and used directly: - -```python -from speaker_recognition import enroll_speaker, identify_speaker, list_enrolled_speakers - -# Enroll speaker -success = enroll_speaker("john", "John Doe", "/path/to/audio.wav") - -# Get embedding and identify -embedding = extract_embedding_from_audio("/path/to/unknown.wav") -speaker_id = identify_speaker(embedding) - -# List all speakers -speakers = list_enrolled_speakers() -``` - -## Next Steps - -1. **Improve Accuracy**: Collect more training data for your specific use case -2. **Real-time Processing**: Implement streaming speaker recognition -3. **Speaker Adaptation**: Fine-tune models on your specific speakers -4. **Multi-language Support**: Add support for different languages -5. **Speaker Verification**: Add 1:1 verification in addition to 1:N identification diff --git a/backends/advanced/Docs/UI.md b/backends/advanced/Docs/UI.md deleted file mode 100644 index 5559b452..00000000 --- a/backends/advanced/Docs/UI.md +++ /dev/null @@ -1,244 +0,0 @@ -# React Web Dashboard Documentation - -## Overview - -The Chronicle web dashboard provides a comprehensive interface for managing conversations, memories, users, and system debugging. Built with modern React and TypeScript, it offers real-time access to audio processing pipelines, advanced memory search, and administrative functions. - -> **Note**: This documentation covers the modern React interface located in `./webui/`. The legacy Streamlit interface has been moved to `src/_webui_original/` for reference. - -## Access & Authentication - -### Dashboard URL -- **HTTP**: `http://localhost:5173` (development) or `http://localhost:3000` (production) -- **HTTPS**: `https://localhost/` (automatic via setup wizard - see [Docs/ssl-certificates.md](../../../Docs/ssl-certificates.md)) -- **Live Recording**: Available at `/live-record` page for real-time audio streaming -- **Network Access**: Configure `BACKEND_PUBLIC_URL` for remote device access via Tailscale/LAN - -### Authentication Methods -1. **Email/Password Login**: Standard authentication via backend API -2. **JWT Token**: Direct token authentication (for programmatic access) -3. **Google OAuth**: Social login integration (if configured) - -### User Roles -- **Regular Users**: Access to own conversations and memories -- **Admin Users**: Full system access including debug tools and user management - -## Dashboard Sections - -### 1. Conversations Tab -**Purpose**: View and manage audio conversations and transcripts - -**Features**: -- Real-time conversation listing with metadata -- Audio playback and transcript viewing -- Conversation status tracking (open/closed) -- Speaker identification and timing information -- Audio file upload for processing existing recordings - -**Admin Features**: -- View all users' conversations -- Advanced filtering and search capabilities - -### 2. Memories Tab -**Purpose**: Browse and search the agentic Markdown vault (Obsidian-style notes that are the single source of truth for memories) - -**Core Search Features**: -- **Text Search**: Keyword-based filtering of vault notes -- **Vault Search Button**: AI-powered search that runs a read-only retrieval agent (ripgrep over the vault) and returns a synthesized answer plus cited note paths -- **Memory Count Display**: Shows the notes/memories surfaced from the vault - -**Advanced Features**: -- **Reset Functionality**: Clear filters and return to the full memory list -- Memory source tracking (which conversation / note) -- Export capabilities - -**UI Improvements**: -- **Search Input Enhancement**: Vault search button integrated into search field -- **Visual Feedback**: Loading states and cited note paths -- **Responsive Design**: Works across desktop and mobile devices - -**Admin Features**: -- **Admin Debug Section**: Load and view all user memories for debugging -- **System-wide Memory View**: Access all memories across users -- **Memory Statistics**: Processing success rates and performance metrics - -### 3. User Management Tab -**Purpose**: Administrative user and client management - -**Features**: -- **Create New Users**: Email-based user registration -- **User Listing**: View all registered users with details -- **User Deletion**: Remove users and optionally clean up their data -- **Client Management**: View active audio clients and connections - -**Admin Only**: This entire tab requires superuser privileges - -### 4. Conversation Management Tab -**Purpose**: Real-time conversation and client control - -**Features**: -- **Active Clients**: View currently connected audio clients -- **Conversation Control**: Manually close open conversations -- **Connection Monitoring**: Real-time client status and metadata -- **WebSocket Information**: Authentication tokens and connection details - -### 5. ๐Ÿ”ง System State Tab (Admin Only) -**Purpose**: Real-time system monitoring, debugging, and failure recovery status - -**Important**: This tab uses **lazy loading** - click the buttons to load specific data sections. This design prevents performance issues and allows selective monitoring. - -**Features**: - -#### System Overview (Click "๐Ÿ“ˆ Load Debug Stats") -- **Processing Metrics**: Total memory sessions, success rates, processing times -- **Failure Analysis**: Failed extractions and error tracking -- **Performance Monitoring**: Average processing times and bottlenecks -- **Live Statistics**: Real-time system performance data - -#### Recent Memory Sessions (Click "๐Ÿ“‹ Load Recent Sessions") -- **Session Listing**: Recent memory processing attempts with status -- **Session Details**: Deep dive into specific processing sessions with full JSON data -- **Pipeline Tracing**: Step-by-step processing flow analysis -- **Error Debugging**: Detailed error messages and stack traces for failed sessions - -#### System Configuration (Click "๐Ÿ“‹ Load Memory Config") -- **Memory Config**: Current memory extraction settings and LLM configuration -- **Debug Settings**: System debug mode, logging levels, and performance settings -- **Live Config**: Real-time configuration without restart required - -#### Failure Recovery System (Click "๐Ÿ“Š Load System Overview") -- **System Health**: Overall system status (healthy/degraded/critical) -- **Queue Statistics**: Processing queue metrics, backlogs, and throughput -- **Service Health**: Real-time health checks for all dependencies: - - MongoDB connectivity and response times - - Ollama/OpenAI API availability and model status - - ASR service connectivity and transcription status -- **Recovery Metrics**: Automatic recovery attempts and success rates - -#### Service Health Monitoring (Click "๐Ÿ” Check Service Health") -- **Service Status Grid**: Visual status indicators for all services -- **Response Times**: Real-time latency metrics for each service -- **Failure Tracking**: Consecutive failure counts and error messages -- **Circuit Breaker Status**: Service protection states and thresholds - -#### Usage Tips -- **Click buttons to load data**: Content appears only after clicking section buttons -- **Refresh data**: Use the "๐Ÿ”„ Refresh Debug Data" button to clear cache and reload -- **Monitor continuously**: Regularly check different sections for system health -- **Error investigation**: Use session details to debug processing failures - -## API Integration - -### Debug API Endpoints (Admin) -The dashboard integrates with comprehensive debug APIs: - -**Memory Debug APIs:** -- `GET /api/debug/memory/stats` - Processing statistics -- `GET /api/debug/memory/sessions` - Recent sessions -- `GET /api/debug/memory/session/{uuid}` - Session details -- `GET /api/debug/memory/config` - Configuration -- `GET /api/debug/memory/pipeline/{uuid}` - Pipeline trace - -**Failure Recovery APIs:** -- `GET /api/failure-recovery/system-overview` - System status -- `GET /api/failure-recovery/queue-stats` - Queue metrics -- `GET /api/failure-recovery/health` - Service health -- `GET /api/failure-recovery/circuit-breakers` - Circuit breaker status - -### Authentication Requirements -- All debug APIs require admin authentication -- JWT tokens must have `is_superuser: true` -- Regular users see filtered data based on their user ID - -## Advanced Features - -### Real-time Updates -- **Auto-refresh**: Configurable refresh intervals for live data -- **WebSocket Status**: Live connection monitoring -- **Health Monitoring**: Real-time service status updates - -### Data Export -- **Memory Export**: Download memories in JSON format -- **Conversation Export**: Export transcripts and audio metadata -- **Debug Reports**: Export system performance reports - -### Troubleshooting Tools -- **Connection Testing**: Verify backend API connectivity -- **Authentication Debugging**: Token validation and user info display -- **Service Diagnostics**: Health checks for all system components - -## Configuration - -### Environment Variables -```bash -# Backend API connection -BACKEND_API_URL=http://localhost:8000 -BACKEND_PUBLIC_URL=http://your-domain:8000 - -# Debug mode -DEBUG=true # Enable detailed logging - -# Authentication (inherited from backend) -AUTH_SECRET_KEY=your-secret-key -ADMIN_PASSWORD=your-admin-password -``` - -## Usage Patterns - -### Admin Workflow -1. **Login** with superuser credentials -2. **System Health**: Check debug logs tab for service status -3. **Monitor Processing**: Review memory debug statistics -4. **User Management**: Create/manage user accounts as needed -5. **Troubleshooting**: Use debug tools to investigate issues - -### User Workflow -1. **Authentication**: Login via sidebar -2. **View Conversations**: Browse recent audio sessions -3. **Search Memories**: Find relevant conversation insights -5. **Connect Clients**: Use provided tokens for audio devices - -## Security Considerations - -### Access Control -- **Role-based UI**: Admin features hidden from regular users -- **API Security**: All requests include proper authentication headers -- **Token Management**: Secure token storage and automatic refresh - -### Data Privacy -- **User Isolation**: Non-admin users only see their own data -- **Audit Logging**: All admin actions logged for accountability -- **Secure Communication**: HTTPS recommended for production - -## Troubleshooting - -### Common Issues - -#### Connection Problems -- Verify `BACKEND_API_URL` points to running backend -- Check firewall/port settings -- Ensure backend health endpoint responds - -#### Authentication Failures -- Verify admin credentials in backend `.env` -- Check JWT token expiration (1-hour default) -- Confirm user has appropriate permissions - -#### Missing Debug Tab -- Only visible to admin users (`is_superuser: true`) -- Verify authentication with admin account -- Check backend user creation and superuser flag - -#### API Errors -- Check backend logs for detailed error information -- Verify all required services are running (MongoDB, Redis, etc.) -- Test API endpoints directly with curl for debugging - -### Debug Steps -1. **Check Logs**: `./logs/streamlit.log` for frontend issues -2. **Backend Health**: Use `/health` endpoint to verify backend status -3. **API Testing**: Test endpoints directly with admin token -4. **Service Status**: Use debug tab to check component health -5. **Configuration**: Verify all environment variables are set correctly - -This dashboard provides comprehensive system management capabilities with particular strength in debugging and monitoring the audio processing pipeline and memory extraction systems. diff --git a/backends/advanced/Docs/architecture.md b/backends/advanced/Docs/architecture.md deleted file mode 100644 index da52b6a1..00000000 --- a/backends/advanced/Docs/architecture.md +++ /dev/null @@ -1,1077 +0,0 @@ -# Chronicle Backend Architecture - -> ๐Ÿ“– **Prerequisite**: Read [quickstart.md](./quickstart.md) first for basic system understanding. - -## System Overview - - -Chronicle is a comprehensive real-time conversation processing system that captures audio streams, performs speech-to-text transcription, and extracts memories. The system features a FastAPI backend with WebSocket audio streaming, versioned transcript and memory processing, a React web dashboard with search capabilities, and user authentication with role-based access control. - - -**Core Implementation**: The complete system is implemented in `src/advanced_omi_backend/main.py` with supporting services in dedicated modules, using a modular router/controller architecture pattern. - -## Architecture Diagram - -```mermaid -graph TB - %% External Inputs - subgraph "Audio Sources" - MIC[Mic/Wearables] - APP[Mobile App] - end - - %% Main WebSocket Server - subgraph "WebSocket Server" - WS["/ws?codec=pcm endpoint"] - AUTH[JWT Auth] - end - - %% Wyoming Protocol Handler - subgraph "Wyoming Protocol" - WP[Protocol Parser] - AS[audio-start event] - AC[audio-chunk event] - AST[audio-stop event] - end - - %% Client Management - subgraph "Client Management" - CM[ClientManager] - CS[ClientState] - CT[Client Tracking] - CONV_MGR[ConversationManager
Lifecycle Coordination] - end - - %% Processor System - subgraph "ProcessorManager" - direction TB - PM[Manager Coordinator] - - subgraph "Global Queues" - AQ[Audio Queue] - TQ[Transcription Queue] - MQ[Memory Queue] - CQ[Cropping Queue] - end - - subgraph "Processors" - AP[Audio Processor] - TP[Transcription Processor] - MP[Memory Processor] - CP[Cropping Processor] - end - - subgraph "Event Coordination" - TC[TranscriptCoordinator
AsyncIO Events] - end - end - - %% Storage Systems - subgraph "Storage" - FS[File System
Audio Files] - - subgraph "MongoDB" - AC_COL[audio_chunks
All Audio Sessions] - CONV_COL[conversations
Speech-Detected Only] - USERS[users] - CONV_REPO[ConversationRepository
Clean Data Access] - end - - subgraph "Markdown Vault" - MEM[conversation_docs
Obsidian-style notes] - end - end - - %% Transcription Services - subgraph "Transcription Services" - DG[Deepgram
Nova-3] - WY[Wyoming ASR
Fallback] - end - - %% Memory Services - subgraph "LLM Services" - OAI[OpenAI] - OLL[Ollama] - end - - %% Data Flow - MIC --> WS - APP --> WS - WS --> AUTH - AUTH --> WP - - %% Wyoming Protocol Flow - WP --> AS - WP --> AC - WP --> AST - - AS --> CM - AC --> CM - AST --> CM - - %% Client State Management - CM --> CS - CS --> CT - - %% Audio Processing Flow - CS -->|queue audio| AQ - AQ --> AP - AP -->|save| FS - AP -->|queue| TQ - - %% Transcription Flow - TQ --> TP - TP --> DG - TP --> WY - TP -->|save transcript| AC_COL - TP -->|signal completion| TC - - %% Conversation Closure Flow (Critical Path) - AST -->|close_conversation| CS - CS -->|delegate to| CONV_MGR - CONV_MGR -->|wait for event| TC - TC -->|transcript ready| CONV_MGR - CONV_MGR -->|queue memory| MQ - CONV_MGR -->|if enabled| CQ - - %% Memory Processing Flow - MQ --> MP - MP -->|read via| CONV_REPO - CONV_REPO -->|access| AC_COL - MP --> OAI - MP --> OLL - MP -->|memory agent writes notes| MEM - MP -->|update via| CONV_REPO - - %% Cropping Flow - CQ --> CP - CP -->|read| FS - CP -->|save cropped| FS - CP -->|update| AC_COL - - %% Management - PM -.-> AQ - PM -.-> TQ - PM -.-> MQ - PM -.-> CQ - PM -.-> AP - PM -.-> TP - PM -.-> MP - PM -.-> CP - - style WS fill:#f9f,stroke:#333,stroke-width:4px - style CS fill:#bbf,stroke:#333,stroke-width:2px - style MQ fill:#fbb,stroke:#333,stroke-width:2px - style AST fill:#bfb,stroke:#333,stroke-width:2px - style TC fill:#ffb,stroke:#333,stroke-width:3px - style CONV_MGR fill:#bff,stroke:#333,stroke-width:2px - style CONV_REPO fill:#fbf,stroke:#333,stroke-width:2px -``` - -## Component Descriptions - -### Core Application - -#### FastAPI Backend (`src/advanced_omi_backend/main.py`) -- **Authentication-First Design**: All endpoints require JWT authentication -- **Wyoming Protocol WebSocket**: Real-time audio streaming using Wyoming protocol (JSONL + binary) for structured session management -- **WebSocket Audio Streaming**: Real-time Opus/PCM audio ingestion with per-client isolation -- **Conversation Management**: Session-based conversation lifecycle using Wyoming audio-start/stop events -- **REST API Suite**: Comprehensive endpoints organized in modular routers (`src/advanced_omi_backend/routers/`) -- **Health Monitoring**: Detailed service health checks and performance metrics (`src/advanced_omi_backend/routers/modules/system_routes.py`) -- **Audio Cropping**: Intelligent speech segment extraction using FFmpeg - -#### Authentication System (`src/advanced_omi_backend/auth.py`) -- **FastAPI-Users Integration**: Complete user lifecycle management -- **Email Authentication**: User authentication via email and password -- **Multi-Authentication**: JWT tokens, and cookie-based sessions -- **Role-Based Access Control**: Admin vs regular user permissions with data isolation -- **WebSocket Security**: Custom authentication for real-time connections with token/cookie support -- **Admin User Bootstrap**: Automatic admin account creation -- **Client ID Generation**: Automatic `objectid_suffix-device_name` format for client identification - -> ๐Ÿ“– **Read more**: [Authentication Architecture](./auth.md) for complete authentication system details - -#### React Dashboard (`webui/`) -- **Modern Interface**: Complete React/TypeScript web-based management interface -- **Advanced Memory Search**: Agentic vault search (ripgrep over notes, synthesized answer with cited note paths) -- **Memory Count Display**: Notes/memories surfaced from the vault -- **Authentication Integration**: Login with backend JWT tokens and user management -- **Real-Time Monitoring**: Live client status and conversation management -- **Live Recording**: Real-time audio recording with WebSocket streaming (HTTPS) -- **Data Management**: User, conversation, and memory interfaces with enhanced UX -- **Audio Playback**: Smart audio player with original/cropped audio options -- **System Health**: Visual service status and configuration display - -### Wyoming Protocol Implementation - -The system implements Wyoming protocol for structured WebSocket audio communication, providing clear session boundaries and improved conversation management. - -#### Protocol Overview -Wyoming is a peer-to-peer protocol for voice assistants that combines JSONL (JSON Lines) headers with binary audio payloads: - -**Protocol Format**: -``` -{JSON_HEADER}\n - -``` - -#### Supported Events - -**Audio Session Events**: -- **audio-start**: Signals the beginning of an audio recording session - ```json - {"type": "audio-start", "data": {"rate": 16000, "width": 2, "channels": 1}, "payload_length": null} - ``` - -- **audio-chunk**: Contains raw audio data with format metadata - ```json - {"type": "audio-chunk", "data": {"rate": 16000, "width": 2, "channels": 1}, "payload_length": 320} - <320 bytes of PCM/Opus audio data> - ``` - -- **audio-stop**: Signals the end of an audio recording session - ```json - {"type": "audio-stop", "data": {"timestamp": 1234567890}, "payload_length": null} - ``` - -#### Backend Implementation - -**Advanced Backend (`/ws?codec=pcm`)**: -- **Full Wyoming Protocol Support**: Parses all Wyoming events for comprehensive session management -- **Session State Tracking**: Only processes audio chunks when session is active (after receiving audio-start) -- **Conversation Boundaries**: Uses Wyoming audio-start/stop events to define precise conversation segments -- **PCM Audio Processing**: Direct processing of PCM audio data from all apps - -**Advanced Backend (`/ws?codec=opus`)**: -- **Wyoming Protocol + Opus Decoding**: Combines Wyoming session management with OMI Opus decoding -- **Continuous Streaming**: OMI devices stream continuously, audio-start/stop events are optional -- **Timestamp Preservation**: Uses timestamps from Wyoming headers when provided -- **Dynamic Sample Rate**: Automatically detects and adapts to client sample rate (typically 16kHz for OMI devices, but supports other rates) - -**Simple Backend (`/ws`)**: -- **Minimal Wyoming Support**: Parses audio-chunk events, silently ignores control events -- **Opus Processing**: Handles Opus-encoded audio chunks from Wyoming protocol messages -- **Legacy Compatibility**: Maintains support for raw Opus packet streaming - -#### Protocol Parser Implementation -Both backends include a Wyoming protocol parser (`parse_wyoming_protocol()`) that: -- Detects message type (JSONL header vs. raw binary) -- Parses JSON headers and extracts payload length -- Reads binary payloads when specified -- Provides backward compatibility for non-Wyoming clients - -#### Benefits -- **Clear Session Boundaries**: Eliminates timeout-based conversation detection -- **Structured Communication**: Consistent protocol across all audio streaming interfaces -- **Future Extensibility**: Foundation for additional event types (pause, resume, metadata) -- **Universal App Integration**: Simple implementation for all WebSocket clients (mobile, desktop, web) -- **Backward Compatibility**: Existing raw audio streaming clients continue to work - -### Audio Processing Pipeline - -#### Transcription Architecture - -The system implements a dual transcription approach with Deepgram as primary and Wyoming ASR as fallback: - -**Deepgram Batch Processing**: -- **Model**: Nova-3 (Deepgram's latest high-accuracy model) -- **Features**: Smart formatting, punctuation, speaker diarization -- **Processing**: Collect-then-process approach using REST API -- **Timeout**: 1.5 minute collection timeout for optimal quality -- **Client Manager Integration**: Uses centralized ClientManager for clean client state access -- **Configuration**: Auto-enables when `DEEPGRAM_API_KEY` environment variable is present - -**Wyoming ASR Fallback**: -- **Purpose**: Offline fallback when Deepgram unavailable -- **Protocol**: TCP connection to self-hosted Wyoming ASR service -- **Event-Driven**: Asynchronous event processing with background queue management -- **Graceful Degradation**: Seamless fallback without service interruption - -**TranscriptionManager Architecture**: -```python -# Clean dependency injection pattern -TranscriptionManager( - chunk_repo=database_repo, - # Uses get_client_manager() singleton for client state access -) -``` - -#### Client Manager Architecture - -The system uses a centralized **ClientManager** for managing active client connections and state: - -**Centralized Client Management**: -```python -# Singleton pattern for global client state access -client_manager = get_client_manager() - -# Client state management -client_state = ClientState( - client_id="user_id_suffix-device_name", - chunk_repo=database_repo, - chunk_dir=audio_storage_path -) -``` - -**Client ID Format**: `{objectid_suffix}-{device_name}` -- Uses last 6 characters of MongoDB ObjectId + device name -- Examples: `cd7994-laptop`, `e26efe-upload-001` -- Ensures unique identification across users and devices - -**Key Features**: -- **Connection Tracking**: Real-time monitoring of active clients -- **State Management**: Simplified client state for conversation tracking only -- **Centralized Processing**: Application-level processors handle all background tasks -- **Dynamic Sample Rate**: Client state tracks actual sample rate from audio chunks -- **Audio Buffer Management**: Sophisticated buffer system with timing and collection management - -### Audio Buffer Management - -The system implements advanced audio buffer management for reliable processing: - -**Buffer Collection**: -- **Retention**: Configurable buffer retention (default 120 seconds for speaker identification) -- **Timeout**: 1.5 minute collection timeout to prevent indefinite buffering -- **Isolation**: Each client maintains isolated buffer state -- **Dynamic Sizing**: Adapts to actual sample rate and chunk sizes - -**Buffer State Tracking**: -- Sample rate detection from incoming audio chunks -- Automatic fallback to default rates when not specified -- Buffer timing synchronization for accurate segment extraction -- Memory-efficient circular buffer implementation - -### Application-Level Processing Architecture - -The system has been refactored from per-client processing to application-level processing for improved reliability and resource management: - -**Processor Manager**: -```python -# Centralized processing with global queues -ProcessorManager( - audio_queue=asyncio.Queue(), # Audio file management - transcription_queue=asyncio.Queue(), # Deepgram/Wyoming ASR - memory_queue=asyncio.Queue(), # LLM processing - cropping_queue=asyncio.Queue() # Speech segment extraction -) -``` - -**Background Task Manager**: -```python -# Centralized task tracking prevents orphaned processes -BackgroundTaskManager( - tasks={}, # Active task tracking - completed_tasks=[], # Completed task history - max_completed_history=1000 # Task history limit -) -``` - -**Key Benefits**: -- **Prevents Orphaned Processes**: All background tasks tracked centrally -- **Processing Continues After Disconnect**: Tasks complete even if client disconnects -- **Resource Leak Prevention**: Proper cleanup of TCP connections, file handles, threads -- **Robust Error Handling**: Comprehensive exception handling with guaranteed cleanup -- **Task Timeout Management**: Automatic cancellation of long-running tasks - -**Architecture Changes**: -- **ClientState**: Simplified to state management only (conversation tracking, speech segments) -- **ProcessorManager**: Handles all audio, transcription, memory, and cropping tasks -- **TaskManager**: Tracks all background tasks with health monitoring -- **Exception Safety**: WebSocket handlers use try/finally for guaranteed cleanup -- **Resource Management**: Automatic cleanup on client disconnect -- **Multi-Device Support**: Single user can have multiple active clients -- **Thread-Safe Operations**: Concurrent client access with proper synchronization - -### Conversation Closure and Memory Processing - -**Recent Architecture Changes**: -1. Memory processing decoupled from transcription to prevent duplicates -2. **Event-driven coordination** eliminates polling/retry race conditions - -#### Previous Architecture (Had Issues) -``` -Transcription Complete โ†’ Queue Memory Processing โŒ (Duplicate) - โ†“ -Conversation Close โ†’ Queue Memory Processing โŒ (Duplicate) -``` - -#### Current Architecture (Event-Driven) -``` -Transcription Complete โ†’ Save Transcript + Signal Event โœ“ - โ†“ -Conversation Close โ†’ Wait for Event โ†’ Queue Memory Processing โœ“ -``` - -#### Event Coordination System -The system now uses **TranscriptCoordinator** for proper async coordination: - -```python -class TranscriptCoordinator: - async def wait_for_transcript_completion(audio_uuid: str) -> bool: - # Wait for asyncio.Event (no polling!) - - def signal_transcript_ready(audio_uuid: str): - # Signal completion from TranscriptionManager -``` - -**Benefits:** -- โœ… **Zero polling** - Uses asyncio events instead of sleep/retry loops -- โœ… **Immediate processing** - Memory processor starts as soon as transcript is ready -- โœ… **No race conditions** - Proper async coordination prevents timing issues -- โœ… **Better performance** - No artificial delays or timeout-based waiting - -#### New Architecture Components - -**TranscriptCoordinator** (`transcript_coordinator.py`) -- **Event-Driven Coordination**: Manages asyncio.Event objects for transcript completion -- **Zero-Polling Design**: Eliminates sleep/retry loops in favor of proper async events -- **Global Singleton**: Centralized coordination across all processors -- **Automatic Cleanup**: Events are cleaned up after use to prevent memory leaks - -**ConversationManager** (`conversation_manager.py`) -- **Lifecycle Management**: Handles complete conversation closure workflow -- **Single Responsibility**: Separated from ClientState for cleaner architecture -- **Processing Coordination**: Manages memory processing and audio cropping queuing -- **Error Handling**: Comprehensive error handling with success/failure tracking - -**ConversationRepository** (`conversation_repository.py`) -- **Repository Pattern**: Clean abstraction for conversation data access -- **Event Integration**: Signals TranscriptCoordinator when transcripts are saved -- **Domain-Focused Methods**: High-level methods like `get_full_conversation_text()` -- **Proper Error Handling**: All database operations wrapped with try/catch and logging - -#### Conversation Closure Triggers - -The `close_conversation()` method in ClientState now delegates to ConversationManager: - -1. **Wyoming Protocol `audio-stop` Event** (Primary Path) - - Explicit session end from client - - Most reliable trigger - - Immediate processing - -2. **Client Disconnect** - - WebSocket connection closed - - Cleanup handler ensures closure - - Prevents orphaned conversations - -3. **Manual API Call** - - `/api/conversations/{client_id}/close` endpoint - - User-initiated closure - - UI button in React dashboard - -4. **Conversation Timeout** (Not shown in diagram) - - 1.5 minute silence detection - - Automatic closure for inactive sessions - -#### Memory Processing Flow - -When `close_conversation()` is called: - -```python -async def close_current_conversation(self): - # 1. Check if conversation has content - if not self.has_transcripts(): - return # No memory processing needed - - # 2. Save conversation to MongoDB - audio_uuid = await self.save_conversation() - - # 3. Queue memory processing (ONLY place this happens) - if audio_uuid and self.has_required_data(): - processor_manager.queue_memory_processing({ - 'audio_uuid': audio_uuid, - 'user_id': self.user_id, - 'transcript': self.get_transcript() - }) - - # 4. Queue audio cropping if enabled - if self.audio_cropping_enabled: - processor_manager.queue_audio_cropping(audio_path) - - # 5. Reset state for next conversation - self.reset_conversation_state() -``` - -#### Reliability Considerations - -**Q: Can `close_conversation` fail to be called?** - -Potential failure scenarios: -1. **Server crash before processing `audio-stop`** - Mitigated by disconnect handler -2. **Network failure during event transmission** - Client reconnect triggers new session -3. **Client bug that doesn't send `audio-stop`** - Disconnect handler ensures cleanup - -**Q: Is memory processing guaranteed?** - -Current guarantee: **Best effort with multiple fallbacks** -- Primary trigger: Wyoming protocol events -- Fallback 1: Client disconnect handler -- Fallback 2: Manual API endpoint -- Future improvement: Periodic sweep for unprocessed conversations - -#### Per-Client State Management -```mermaid -stateDiagram-v2 - [*] --> Connected: WebSocket Auth - Connected --> Processing: Audio Received - Processing --> Transcribing: Chunk Buffered - Transcribing --> ActiveConversation: Transcript Generated - ActiveConversation --> Processing: Continue Audio - ActiveConversation --> ConversationTimeout: 1.5min Silence - Processing --> ManualClose: User Action - ConversationTimeout --> ProcessingMemory: Extract Insights - ManualClose --> ProcessingMemory: Extract Insights - ProcessingMemory --> AudioCropping: Remove Silence - AudioCropping --> ConversationClosed: Cleanup Complete - ConversationClosed --> Connected: Ready for New - Connected --> [*]: Client Disconnect -``` - -#### Audio Processing Queues (Per-Client) -- **Chunk Queue**: Raw audio buffering with client isolation -- **Transcription Queue**: Audio chunks for real-time ASR processing with quality validation -- **Memory Queue**: Completed conversations for LLM memory extraction (with transcript validation) -- **Quality Control**: Multi-stage validation prevents empty/invalid transcripts from consuming LLM resources - -#### Speech Processing Features -- **Voice Activity Detection**: Automatic silence removal and speech segment extraction -- **Audio Cropping**: FFmpeg-based processing to create concise audio files -- **Multiple Format Support**: Opus (compressed) and PCM (uncompressed) audio input -- **Conversation Chunking**: 60-second segments with seamless processing - -### Intelligence Services - - -#### Memory Management (`src/advanced_omi_backend/memory/memory_service.py`) -- **User-Centric Storage**: All memories keyed by database user_id (not client_id) -- **Agentic Vault Writes**: A tool-calling memory agent records each conversation as `Conversations/.md` and edits `People/.md`, `Topics/.md`, and `/.md` notes -- **Single Source of Truth**: The Markdown vault at `data/conversation_docs//` is the only memory store -- **Agentic Retrieval**: A read-only retrieval agent drives ripgrep (grep/glob/read_note) over the vault and synthesizes an answer with cited note paths -- **User Isolation**: Complete data separation between users via per-user vault folders -- **Processing Trigger**: Conversation end events processed by `src/advanced_omi_backend/controllers/memory_controller.py` - -> ๐Ÿ“– **Read more**: [Memory System Documentation](./memories.md) for detailed memory extraction and storage - - -### Data Models & Access - -#### User Management (`src/advanced_omi_backend/users.py`) -- **Beanie ODM**: MongoDB document modeling with type safety -- **User ID System**: MongoDB ObjectId-based user identification -- **Authentication Data**: Secure password hashing, email verification, email-based login -- **Profile Management**: User preferences, display names, and permissions -- **Client Registration**: Tracking of registered clients per user with device names -- **Data Ownership**: All data (conversations, memories) associated via user_id -- **Client ID Generation**: Helper functions for `objectid_suffix-device_name` format - -#### Conversation Data Access (`src/advanced_omi_backend/conversation_repository.py`) -- **Audio Metadata**: File paths, timestamps, duration tracking -- **Transcript Management**: Speaker identification and timing information -- **Memory Links**: Connection between conversations and extracted memories - -#### Permission System -- **Dictionary-Based Mapping**: Clean client-user relationship tracking via in-memory dictionaries -- **Active Client Tracking**: `client_to_user_mapping` for currently connected clients -- **Persistent Tracking**: `all_client_user_mappings` for database query permission checks -- **Ownership Validation**: Simple dictionary lookup instead of regex pattern matching -- **Data Isolation**: User-scoped queries using client ID lists for efficient permission filtering - -## Deployment Architecture - -### Docker Compose Structure - -```mermaid -graph LR - subgraph "Docker Network" - Backend[chronicle-backend
uv + FastAPI] - WebUI[webui
React Dashboard] - Proxy[nginx
Load Balancer] - Mongo[mongo:4.4.18
Primary Database] - Vault[Markdown Vault
conversation_docs] - end - - subgraph "External Services" - Ollama[ollama
LLM Service] - ASRService[ASR Services
extras/asr-services] - end - - subgraph "Client Access" - WebBrowser[Web Browser
Dashboard] - AudioClient[Audio Client
Mobile/Desktop] - end - - WebBrowser -->|Port 5173 (dev)| WebUI - WebBrowser -->|Port 80 (prod)| Proxy - AudioClient -->|Port 8000| Backend - - Proxy --> Backend - Proxy --> WebUI - Backend --> Mongo - Backend --> Vault - Backend -.->|Optional| Ollama - Backend -.->|Optional| ASRService -``` - -### Container Specifications - -#### Backend Container (`chronicle-backend`) -- **Base**: Python 3.12 slim with uv package manager -- **Dependencies**: FastAPI, WebSocket libraries, audio processing tools -- **Volumes**: Audio chunk storage, debug directories -- **Health Checks**: Automated readiness and liveness probes -- **Environment**: All configuration via environment variables - -#### React WebUI Container (`webui`) -- **Purpose**: Modern React-based web dashboard interface -- **Dependencies**: React, TypeScript, Vite for development, Nginx for production -- **Backend Integration**: HTTP API client with JWT authentication -- **Configuration**: Backend URL configuration for API calls -- **Development**: Vite dev server on port 5173 -- **Production**: Nginx serving built React app on port 80 - -#### Infrastructure Containers -- **MongoDB 4.4.18**: Primary data storage with persistence -- **Redis**: Job queues (RQ workers) and session state -- **Markdown Vault**: Per-user Obsidian-style notes on disk at `data/conversation_docs//` (the single memory store) -- **Nginx Alpine**: Reverse proxy and load balancing (serves React app in production, proxies API calls to backend) - -## Detailed Data Flow Architecture - -> ๐Ÿ“– **Reference Documentation**: -> - [Authentication Details](./auth.md) - Complete authentication system documentation - -### Complete System Data Flow Diagram - -```mermaid -flowchart TB - %% External Clients - Client[๐Ÿ“ฑ Audio Client
Mobile/Desktop/HAVPE] - WebUI[๐ŸŒ Web Dashboard
React Web UI] - - %% Authentication Gateway - subgraph "๐Ÿ” Authentication Layer" - AuthGW[JWT/Cookie Auth
๐Ÿ• 1hr token lifetime] - ClientGen[Client ID Generator
user_suffix-device_name] - UserDB[(๐Ÿ‘ค User Database
MongoDB ObjectId)] - end - - %% Audio Processing Pipeline - subgraph "๐ŸŽต Audio Processing Pipeline" - WSAuth[WebSocket Auth
๐Ÿ• Connection timeout: 30s] - OpusDecoder[Opus/PCM Decoder
Real-time Processing] - - subgraph "โฑ๏ธ Per-Client State Management" - ClientState[Client State
๐Ÿ• Conversation timeout: 1.5min] - AudioQueue[Audio Chunk Queue
60s segments] - ConversationTimer[Conversation Timer
๐Ÿ”„ Auto-timeout tracking] - end - - subgraph "๐ŸŽ™๏ธ Transcription Layer" - ASRManager[Transcription Manager
๐Ÿ• Init timeout: 60s] - DeepgramWS[Deepgram WebSocket
Nova-3 Model, Smart Format
๐Ÿ”Œ Auto-reconnect on disconnect] - OfflineASR[Wyoming ASR Fallback
๐Ÿ• Connect timeout: 5s] - ClientManager[Client Manager
Centralized client state access] - TranscriptValidation[Transcript Validation
๐Ÿ“ Min 10 chars] - end - end - - %% Intelligence Services - subgraph "๐Ÿง  Intelligence Processing" - subgraph "๐Ÿ’ญ Memory Pipeline" - MemoryService[Memory Service
๐Ÿ• Init timeout: 60s
๐Ÿ• Processing timeout: 20min] - MemoryValidation[Memory Validation
๐Ÿ“ Min conversation length] - LLMProcessor[Ollama LLM
๐Ÿ”„ Circuit breaker protection] - VaultStore[Markdown Vault
๐Ÿ“ Agent-written notes] - end - - end - - - %% Data Storage - subgraph "๐Ÿ’พ Data Storage Layer" - MongoDB[(MongoDB
Users & Conversations
๐Ÿ• Health check: 5s)] - VaultDB[(Markdown Vault
conversation_docs
๐Ÿ“ Memory notes)] - AudioFiles[Audio Files
๐Ÿ“ Chunk storage + cropping] - end - - %% Connection Flow with Timeouts - Client -->|๐Ÿ” Auth Token| AuthGW - AuthGW -->|โŒ 401 Unauthorized
โฑ๏ธ Invalid/expired token| Client - AuthGW -->|โœ… Validated| ClientGen - ClientGen -->|๐Ÿท๏ธ Generate client_id| WSAuth - - %% Audio Processing Flow - Client -->|๐ŸŽต Opus/PCM Stream
๐Ÿ• 30s connection timeout| WSAuth - WSAuth -->|โŒ 1008 Policy Violation
๐Ÿ” Auth required| Client - WSAuth -->|โœ… Authenticated| OpusDecoder - OpusDecoder -->|๐Ÿ“ฆ Audio chunks| ClientState - ClientState -->|โฑ๏ธ 1.5min timeout check| ConversationTimer - ConversationTimer -->|๐Ÿ”„ Timeout exceeded| ClientState - - %% Transcription Flow with Failure Points - ClientState -->|๐ŸŽต Audio data| ASRManager - ASRManager -->|๐Ÿ”Œ Primary connection| DeepgramWS - ASRManager -->|๐Ÿ”Œ Fallback connection| OfflineASR - ASRManager -->|๐Ÿ“‹ Client state access| ClientManager - DeepgramWS -->|โŒ WebSocket disconnect
๐Ÿ”„ Auto-reconnect after 2s| ASRManager - OfflineASR -->|โŒ TCP connection timeout
๐Ÿ• 5s limit| ASRManager - ASRManager -->|๐Ÿ“ Raw transcript| TranscriptValidation - TranscriptValidation -->|โŒ Too short (<10 chars)
๐Ÿšซ Skip processing| QueueTracker - TranscriptValidation -->|โœ… Valid transcript| MemoryService - - %% Memory Processing with Timeouts - MemoryService -->|๐Ÿ• 20min timeout| LLMProcessor - LLMProcessor -->|โŒ Model stopped
๐Ÿ”„ Circuit breaker trip| CircuitBreaker - LLMProcessor -->|โŒ Empty response
๐Ÿ”„ Fallback memory| MemoryService - LLMProcessor -->|โœ… Memory extracted| VaultStore - MemoryService -->|๐Ÿ“Š Track processing| QueueTracker - - - %% Disconnect and Cleanup Flow - Client -->|๐Ÿ”Œ Disconnect| ClientState - ClientState -->|๐Ÿงน Cleanup tasks
๐Ÿ• Background memory: 5min
๐Ÿ• Transcription queue: 60s| ASRManager - ASRManager -->|๐Ÿ”Œ Graceful disconnect
๐Ÿ• 2s timeout| DeepgramWS - ClientState -->|๐Ÿ”„ Final conversation processing| MemoryService - - %% Storage Integration - MemoryService -->|๐Ÿ’พ Store memories| MongoDB - VaultStore -->|๐Ÿ’พ Notes| VaultDB - QueueTracker -->|๐Ÿ“Š Metrics & tracking| SQLiteTracking - ClientState -->|๐Ÿ“ Audio segments| AudioFiles - - %% Web Dashboard Flow - WebUI -->|๐Ÿ” Cookie/JWT auth
๐Ÿ• 1hr lifetime| AuthGW - WebUI -->|๐Ÿ“Š API calls| MongoDB - WebUI -->|๐ŸŽต Audio playback| AudioFiles -``` - -### Critical Timeout and Failure Points - -#### ๐Ÿ• **Timeout Configuration** -| Component | Timeout Value | Failure Behavior | Recovery Action | -|-----------|---------------|------------------|-----------------| -| **JWT Tokens** | 1 hour | 401 Unauthorized | Client re-authentication required | -| **WebSocket Connection** | 30 seconds | Connection dropped | Client reconnection with auth | -| **Conversation Auto-Close** | 1.5 minutes | New conversation started | Memory processing triggered | -| **Transcription Queue** | 60 seconds | Queue processing timeout | Graceful degradation | -| **Memory Service Init** | 60 seconds | Service unavailable | Health check failure | -| **Ollama Processing** | 20 minutes | LLM timeout | Fallback memory creation | -| **Background Memory Task** | 5 minutes | Task cancellation | Partial processing retained | -| **MongoDB Health Check** | 5 seconds | Service marked unhealthy | Circuit breaker activation | -| **Ollama Health Check** | 8 seconds | Service marked unhealthy | Circuit breaker activation | -| **ASR Connection** | 5 seconds | Connection failure | Fallback ASR or degraded mode | - -#### ๐Ÿ”Œ **Disconnection Scenarios** -1. **Client Disconnect**: Graceful cleanup with conversation finalization -2. **Network Interruption**: Auto-reconnection with exponential backoff -3. **Service Failure**: Circuit breaker protection and alternative routing -4. **Authentication Expiry**: Forced re-authentication with clear error codes - - -### Audio Ingestion & Processing -1. **Client Authentication**: JWT token validation for WebSocket connection (email or user_id based) -2. **Client ID Generation**: Automatic `user_id-device_name` format creation for client identification -3. **Permission Registration**: Client-user relationship tracking in permission dictionaries -4. **Audio Streaming**: Real-time Opus/PCM packets over WebSocket with user context -5. **Per-Client Processing**: Isolated audio queues and state management per user -6. **Audio Session Storage**: All audio sessions stored in `audio_chunks` collection by `audio_uuid` -7. **Speech Detection**: Automatic analysis of transcript content for meaningful speech -8. **Conversation Creation**: User-facing conversations created in `conversations` collection only when speech detected -9. **Transcription Pipeline**: Configurable ASR service integration with versioned storage -10. **Conversation Lifecycle**: Automatic timeout handling and memory processing -11. **Audio Optimization**: Speech segment extraction and silence removal - -#### Critical Timing Considerations -**Transcription Manager Creation Race Condition**: When processing uploaded audio files, there's a timing dependency between: -- Audio chunks being queued via `processor_manager.queue_audio()` (synchronous) -- Transcription manager creation in background `_transcription_processor()` (asynchronous) -- `close_client_audio()` call to flush final transcript (immediate) - -**Solution**: A 2-second delay is added before calling `close_client_audio()` to ensure the transcription manager is created by the background processor. Without this delay, the flush call fails silently and transcription never completes. - -**File Upload Flow**: -1. Audio chunks queued to `transcription_queue` -2. Background transcription processor creates `TranscriptionManager` on first chunk -3. 2-second delay ensures manager exists before flush -4. Client audio closure triggers transcript completion - -### Memory & Intelligence Processing -1. **Conversation Completion**: End-of-session trigger for memory extraction (speech-detected conversations only) -2. **Transcript Validation**: Multi-layer validation prevents empty/short transcripts from reaching LLM implemented in memory controller -3. **User Resolution**: Client-ID to database user mapping for proper data association -4. **Versioned Processing**: Multiple transcript and memory versions with provider/model tracking -5. **LLM Processing**: Ollama-based conversation summarization with user context (only for validated transcripts) -6. **Vault Storage**: A tool-calling memory agent writes notes into the per-user Markdown vault (`data/conversation_docs//`) -7. **Metadata Enhancement**: Client information and user email stored in metadata -8. **Reprocessing Capabilities**: Re-run transcript or memory extraction with different parameters -9. **Version Management**: Active version pointers with automatic legacy field population -10. **Search & Retrieval**: User-scoped agentic vault search (ripgrep over notes, synthesized answer with cited paths) - -### User Management & Security -1. **Registration**: Admin-controlled user creation with email/password and auto-generated user_id -2. **Dual Authentication**: JWT token generation for both email and user_id login methods -3. **Client Association**: Automatic client ID generation as `user_id-device_name` -4. **Permission Tracking**: Dictionary-based client-user relationship management -5. **Authorization**: Per-endpoint permission checking with simplified ownership validation -6. **Data Isolation**: User-scoped data access via client ID mapping and ownership validation - -## Speaker Recognition Integration - -The advanced backend integrates with an external speaker recognition service for real-time speaker identification during conversations. - -### Integration Architecture - -**Service Communication**: -- **HTTP API**: RESTful endpoints for speaker enrollment and management -- **Real-time Processing**: Speaker identification during live transcription -- **Asynchronous Pipeline**: Non-blocking speaker identification parallel to transcription - -**Key Features**: -- **Dynamic Enrollment**: Add speakers through audio samples -- **Live Identification**: Real-time speaker recognition during conversations -- **Confidence Scoring**: Adjustable thresholds for identification accuracy -- **Multi-speaker Support**: Handles conversations with multiple participants - -### Speaker Recognition Flow - -1. **Audio Collection**: Capture audio chunks with proper buffering -2. **Feature Extraction**: Generate speaker embeddings from audio segments -3. **Identity Matching**: Compare against enrolled speaker database -4. **Result Integration**: Enhance transcripts with speaker identification - -### Configuration - -```yaml -# Environment variables for speaker recognition -SPEAKER_SERVICE_URL: "http://speaker-recognition:8001" -SPEAKER_CONFIDENCE_THRESHOLD: 0.15 # Adjustable confidence level -``` - -### API Endpoints - -- `POST /api/speaker/enroll` - Enroll new speaker with audio samples -- `GET /api/speaker/list` - List enrolled speakers -- `POST /api/speaker/identify` - Identify speaker from audio segment -- `DELETE /api/speaker/{speaker_id}` - Remove enrolled speaker - -## Security Architecture - -### Authentication Layers -- **API Gateway**: JWT middleware on all protected endpoints with email/user_id support -- **WebSocket Security**: Custom authentication handler for real-time connections (token + cookie support) -- **Client ID Management**: Automatic generation and validation of `user_id-device_name` format -- **Permission Mapping**: Dictionary-based client-user relationship tracking -- **Role Validation**: Admin vs user permission matrix enforcement -- **Data Scoping**: Efficient user context filtering via client ID mapping - -### Access Control Matrix -| Resource | Regular User | Superuser | -|----------|-------------|-----------| -| Own Conversations | Full Access | Full Access | -| Other Users' Conversations | No Access | Full Access | -| User Management | Profile Only | Full CRUD | -| System Administration | Health Check Only | Full Access | -| Active Client Management | Own Clients Only | All Clients | -| Memory Management | Own Memories Only | All Memories (with client info) | - -### Data Protection -- **Encryption**: JWT token signing with configurable secret keys -- **Password Security**: Bcrypt hashing with salt rounds -- **User Identification**: MongoDB ObjectId-based user system -- **Data Isolation**: User ID validation on all data operations via client mapping -- **Permission Efficiency**: Dictionary-based ownership checking instead of regex patterns -- **Audit Logging**: Comprehensive request and authentication logging with user_id tracking - -## Configuration & Environment - -### Required Environment Variables -```bash -AUTH_SECRET_KEY=your-super-secret-jwt-key-here-make-it-long-and-random -ADMIN_PASSWORD=your-secure-admin-password -``` - -### Optional Service Configuration -```bash -# Database -MONGODB_URI=mongodb://mongo:27017 - -# LLM Processing -OLLAMA_BASE_URL=http://ollama:11434 - -# Memory Storage: per-user Markdown vault on disk (data/conversation_docs//) -# No vector/graph database required. - -# Transcription Services (Deepgram Primary, Wyoming Fallback) -DEEPGRAM_API_KEY=your-deepgram-api-key-here - -``` - -### Service Dependencies - -#### Critical Services (Required for Core Functionality) -- **MongoDB**: User data, conversations -- **Authentication**: JWT token validation and user sessions - -#### Enhanced Services (Optional but Recommended) -- **Ollama**: Memory processing -- **Deepgram**: Primary speech-to-text transcription service (WebSocket streaming) -- **Wyoming ASR**: Fallback transcription service (self-hosted) - -#### External Services (Optional) -- **Ngrok**: Public internet access for development -- **HAVPE Relay**: ESP32 audio streaming bridge with authentication (`extras/havpe-relay/`) - -### HAVPE Relay Integration -The HAVPE relay (`extras/havpe-relay/main.py`) provides ESP32 audio streaming capabilities: - -- **Authentication**: Supports both `AUTH_EMAIL` and `AUTH_USER_ID` environment variables -- **Client ID Generation**: Creates client ID as `user_id-havpe` automatically -- **Audio Processing**: Converts ESP32 32-bit stereo to 16-bit mono for backend -- **Reconnection**: Automatic JWT token refresh and WebSocket reconnection on auth failures -- **Device Name**: Configurable device identifier for multi-device support - -## REST API Architecture - -The system provides a comprehensive REST API organized into functional modules using a router/controller pattern: - -### Router/Controller Architecture -``` -src/advanced_omi_backend/ -โ”œโ”€โ”€ routers/ # Route definitions -โ”‚ โ”œโ”€โ”€ api_router.py # Main API router aggregation -โ”‚ โ””โ”€โ”€ modules/ # Modular route organization -โ”‚ โ”œโ”€โ”€ user_routes.py # User management routes -โ”‚ โ”œโ”€โ”€ client_routes.py # Client monitoring routes -โ”‚ โ”œโ”€โ”€ conversation_routes.py # Conversation routes -โ”‚ โ”œโ”€โ”€ memory_routes.py # Memory management routes -โ”‚ โ””โ”€โ”€ system_routes.py # System utilities routes -โ””โ”€โ”€ controllers/ # Business logic controllers - โ”œโ”€โ”€ user_controller.py # User management logic - โ”œโ”€โ”€ client_controller.py # Client monitoring logic - โ”œโ”€โ”€ conversation_controller.py # Conversation logic - โ”œโ”€โ”€ memory_controller.py # Memory processing logic - โ””โ”€โ”€ system_controller.py # System utilities logic -``` - -### API Organization -``` -/api/ -โ”œโ”€โ”€ /users # User management (admin only) -โ”œโ”€โ”€ /clients/active # Active client monitoring -โ”œโ”€โ”€ /conversations # Conversation CRUD operations (speech-detected only) -โ”‚ โ”œโ”€โ”€ /{conversation_id}/reprocess-transcript # Transcript reprocessing -โ”‚ โ”œโ”€โ”€ /{conversation_id}/reprocess-memory # Memory reprocessing -โ”‚ โ”œโ”€โ”€ /{conversation_id}/versions # Version history -โ”‚ โ”œโ”€โ”€ /{conversation_id}/activate-transcript # Switch transcript version -โ”‚ โ””โ”€โ”€ /{conversation_id}/activate-memory # Switch memory version -โ”œโ”€โ”€ /memories # Memory management and search -โ”‚ โ”œโ”€โ”€ /admin # Admin view (all users) -โ”‚ โ””โ”€โ”€ /search # Agentic vault search -โ”œโ”€โ”€ /admin/ # Admin compatibility endpoints -โ”‚ โ”œโ”€โ”€ /memories # Consolidated admin memory view -โ”‚ โ””โ”€โ”€ /memories/debug # Legacy debug endpoint -โ””โ”€โ”€ /active_clients # Client monitoring (compatibility) -``` - -### Key Endpoints - -#### User & Authentication -- `POST /auth/jwt/login` - Email/password authentication -- `GET /api/users` - User management (admin only) -- `POST /api/create_user` - User creation (admin only) - -#### Client Management -- `GET /api/clients/active` - Active client monitoring (unified endpoint) - -#### Memory Management -- `GET /api/memories` - User memories (with user_id filter for admin) -- `GET /api/memories/admin` - All memories grouped by user (admin only) -- `GET /api/memories/search?query=` - Agentic vault search - -#### Audio & Conversations -- `GET /api/conversations` - User conversations (speech-detected only) -- `GET /api/conversations/{conversation_id}` - Specific conversation details -- `POST /api/conversations/{conversation_id}/reprocess-transcript` - Re-run transcript processing -- `POST /api/conversations/{conversation_id}/reprocess-memory` - Re-extract memories -- `GET /api/conversations/{conversation_id}/versions` - Get version history -- `POST /api/conversations/{conversation_id}/activate-transcript` - Switch transcript version -- `POST /api/conversations/{conversation_id}/activate-memory` - Switch memory version -- `POST /api/audio/upload` - Batch audio file upload and processing -- WebSocket `/ws?codec=opus` - Real-time Opus audio streaming with Wyoming protocol (OMI devices) -- WebSocket `/ws?codec=pcm` - Real-time PCM audio streaming with Wyoming protocol (all apps) - -### Authentication & Authorization -- **JWT Tokens**: All API endpoints require valid JWT authentication -- **User Isolation**: Regular users see only their own data -- **Admin Access**: Superusers can access cross-user data with `user_id` filters -- **WebSocket Auth**: Token or cookie-based authentication for real-time connections - -### Data Formats -```json -// Active clients response -{ - "clients": [ - { - "client_id": "cd7994-laptop", - "user_id": "507f1f77bcf86cd799439011", - "connected_at": "2025-01-15T10:30:00Z", - "conversation_count": 3 // speech-detected conversations only - } - ], - "active_clients_count": 1, - "total_count": 1 -} - -// Conversation with versions response -{ - "conversation_id": "conv_12345", - "audio_uuid": "audio_67890", - "transcript": "Active transcript content", // from active version - "active_transcript_version": "version_abc", - "active_memory_version": "version_def", - "transcript_versions": [ - { - "version_id": "version_abc", - "provider": "deepgram", - "created_at": "2025-01-15T10:30:00Z", - "transcript": "Processed content" - } - ], - "memory_versions": [ - { - "version_id": "version_def", - "provider": "chronicle", - "created_at": "2025-01-15T10:32:00Z", - "memory_count": 5 - } - ] -} - -// Admin memories response -{ - "memories": [...], // Flat list for compatibility - "user_memories": {...}, // Grouped by user_id - "stats": { - "total_memories": 150, - "total_users": 5, - "users_with_memories": ["user1", "user2"], - "client_ids_with_memories": ["cd7994-laptop", "e26efe-upload"] - } -} -``` - -## Performance & Scalability - -### Client Isolation Design -- **Per-Client Queues**: Independent processing pipelines prevent cross-client interference -- **Async Processing**: Non-blocking audio ingestion with background processing -- **Resource Management**: Configurable timeouts and cleanup procedures -- **State Management**: Memory-efficient client state with automatic cleanup - -### Monitoring & Observability -- **Health Checks**: Comprehensive service dependency validation -- **Performance Metrics**: Audio processing latency, transcription accuracy -- **Resource Tracking**: Memory usage, connection counts, processing queues -- **Error Handling**: Graceful degradation with detailed logging -- **System Tracking**: Debug tracking and monitoring via SystemTracker diff --git a/backends/advanced/Docs/contribution.md b/backends/advanced/Docs/contribution.md deleted file mode 100644 index 592a8c62..00000000 --- a/backends/advanced/Docs/contribution.md +++ /dev/null @@ -1,35 +0,0 @@ - 1. Docs/quickstart.md (15 min) - 2. Docs/architecture.md (20 min) - 3. main.py - just the imports and WebSocket sections (15 min) - 4. config/config.yml (memory section) (10 min) - - ๐Ÿ”ง "I want to work on memory extraction" - - 1. Docs/quickstart.md โ†’ Docs/memories.md - 2. config/config.yml (memory.extraction section) - 3. main.py lines 1047-1065 (trigger) - 4. main.py lines 1163-1195 (processing) - 5. src/memory/memory_service.py - 6. src/memory_debug.py (for tracking) - - ๐Ÿ› "I want to debug pipeline issues" - - 1. MEMORY_DEBUG_IMPLEMENTATION.md - 2. src/memory_debug.py - 3. src/memory_debug_api.py - 4. API endpoints: /api/debug/memory/* - - ๐Ÿ—๏ธ "I want to understand the full architecture" - - 1. Docs/architecture.md - 2. main.py (full file, focusing on class structures) - 3. src/auth.py (authentication flow) - 4. src/users.py (user management) - 5. All service files (memory_service.py) - - ๐ŸŽฏ Key Concepts to Understand - - Data Flow - - Audio โ†’ Transcription โ†’ Dual Processing - โ””โ”€ Memory Pipeline (end-of-conversation) diff --git a/backends/advanced/Docs/flowdiagram.excalidraw b/backends/advanced/Docs/flowdiagram.excalidraw deleted file mode 100644 index 6613c57d..00000000 --- a/backends/advanced/Docs/flowdiagram.excalidraw +++ /dev/null @@ -1,856 +0,0 @@ -{ - "type": "excalidraw", - "version": 2, - "source": "https://marketplace.visualstudio.com/items?itemName=pomdtr.excalidraw-editor", - "elements": [ - { - "id": "OulhD4eIDipJVjvMqqHN9", - "type": "text", - "x": -240, - "y": 120, - "width": 35.93995666503906, - "height": 25, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a0", - "roundness": null, - "seed": 949825938, - "version": 58, - "versionNonce": 685271314, - "isDeleted": false, - "boundElements": [ - { - "id": "JuRztH6XRp6DvXdwfkTPv", - "type": "arrow" - }, - { - "id": "Cgan1Y-ZLLhdR0VAd-Y6l", - "type": "arrow" - } - ], - "updated": 1751344820715, - "link": null, - "locked": false, - "text": "/ws", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "left", - "verticalAlign": "top", - "containerId": null, - "originalText": "/ws", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "JuRztH6XRp6DvXdwfkTPv", - "type": "arrow", - "x": -214.25893698297295, - "y": 159.44245910644534, - "width": 6.126033573870387, - "height": 139.94669207277641, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a1", - "roundness": { - "type": 2 - }, - "seed": 1054454862, - "version": 411, - "versionNonce": 1344705618, - "isDeleted": false, - "boundElements": null, - "updated": 1751344820905, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - -6.126033573870387, - 139.94669207277641 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "OulhD4eIDipJVjvMqqHN9", - "focus": -0.4833607406656171, - "gap": 14.442459106445341 - }, - "endBinding": { - "elementId": "xa7OfpSu0rorDHXlV5sB6", - "focus": 0.3680000000000001, - "gap": 1.3585811725927346 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - }, - { - "id": "WM10Gvx91S8CZiVm-px6q", - "type": "rectangle", - "x": 160, - "y": 320, - "width": 440, - "height": 540, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a6", - "roundness": { - "type": 3 - }, - "seed": 685730766, - "version": 66, - "versionNonce": 20639442, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "uI1qhvTeIO-g5d7i_SeZe" - } - ], - "updated": 1751342004950, - "link": null, - "locked": false - }, - { - "id": "uI1qhvTeIO-g5d7i_SeZe", - "type": "text", - "x": 165, - "y": 325, - "width": 119.09991455078125, - "height": 25, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a7", - "roundness": null, - "seed": 1095549650, - "version": 49, - "versionNonce": 1134171598, - "isDeleted": false, - "boundElements": null, - "updated": 1751341649146, - "link": null, - "locked": false, - "text": "Client State", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "left", - "verticalAlign": "top", - "containerId": "WM10Gvx91S8CZiVm-px6q", - "originalText": "Client State", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "rxk6Z6Cs4gccgNpwrVkdK", - "type": "rectangle", - "x": 180, - "y": 360, - "width": 400, - "height": 240, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a8", - "roundness": { - "type": 3 - }, - "seed": 2010082002, - "version": 41, - "versionNonce": 1878576142, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "Uqq3VJow6LhzLDQvEYagw" - }, - { - "id": "JNs971qoZ58e4Q6rk-DVc", - "type": "arrow" - }, - { - "id": "qH2PkV-HogNFWoirkepSq", - "type": "arrow" - }, - { - "id": "7rgHqrDT-vJkJpLWGNRta", - "type": "arrow" - }, - { - "id": "oA3CO6Jh9f2JGbBuvqBm5", - "type": "arrow" - } - ], - "updated": 1751344829847, - "link": null, - "locked": false - }, - { - "id": "Uqq3VJow6LhzLDQvEYagw", - "type": "text", - "x": 235.20013427734375, - "y": 442.5, - "width": 289.5997314453125, - "height": 75, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "a9", - "roundness": null, - "seed": 999127186, - "version": 90, - "versionNonce": 955981970, - "isDeleted": false, - "boundElements": null, - "updated": 1751341691552, - "link": null, - "locked": false, - "text": "Raw Audio Stream\nTranscript, with speaker tags\nSegments/Metadata", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "center", - "verticalAlign": "middle", - "containerId": "rxk6Z6Cs4gccgNpwrVkdK", - "originalText": "Raw Audio Stream\nTranscript, with speaker tags\nSegments/Metadata", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "6ZE9XSb7Jvai-_AUUHAlT", - "type": "rectangle", - "x": -60, - "y": 100, - "width": 80, - "height": 60, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aA", - "roundness": { - "type": 3 - }, - "seed": 475694418, - "version": 75, - "versionNonce": 270989906, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "E-r7mTyIym633uouimXnJ" - }, - { - "id": "Cgan1Y-ZLLhdR0VAd-Y6l", - "type": "arrow" - } - ], - "updated": 1751344818914, - "link": null, - "locked": false - }, - { - "id": "E-r7mTyIym633uouimXnJ", - "type": "text", - "x": -47.139976501464844, - "y": 105, - "width": 54.27995300292969, - "height": 50, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aAV", - "roundness": null, - "seed": 1464795662, - "version": 75, - "versionNonce": 565698578, - "isDeleted": false, - "boundElements": null, - "updated": 1751344818914, - "link": null, - "locked": false, - "text": "Ws\nClient", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "center", - "verticalAlign": "middle", - "containerId": "6ZE9XSb7Jvai-_AUUHAlT", - "originalText": "Ws Client", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "Cgan1Y-ZLLhdR0VAd-Y6l", - "type": "arrow", - "x": -60.79765933128198, - "y": 120.76825196314995, - "width": 133.66243893531956, - "height": 3.7389022806945746, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aC", - "roundness": { - "type": 2 - }, - "seed": 360111122, - "version": 597, - "versionNonce": 850672146, - "isDeleted": false, - "boundElements": null, - "updated": 1751344820906, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - -133.66243893531956, - 3.7389022806945746 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "6ZE9XSb7Jvai-_AUUHAlT", - "focus": 0.333333333333333, - "gap": 1 - }, - "endBinding": { - "elementId": "OulhD4eIDipJVjvMqqHN9", - "focus": -0.5553967285156247, - "gap": 9.599945068359375 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - }, - { - "id": "tsk-eKUlAyTfy3XOz5ngq", - "type": "rectangle", - "x": 800, - "y": 320, - "width": 420, - "height": 300, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aD", - "roundness": { - "type": 3 - }, - "seed": 1912468626, - "version": 32, - "versionNonce": 1015597202, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "hJImn0HrTnJ3XXruBmZot" - }, - { - "id": "JNs971qoZ58e4Q6rk-DVc", - "type": "arrow" - } - ], - "updated": 1751341768677, - "link": null, - "locked": false - }, - { - "id": "hJImn0HrTnJ3XXruBmZot", - "type": "text", - "x": 805, - "y": 325, - "width": 248.07981872558594, - "height": 100, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aE", - "roundness": null, - "seed": 702178574, - "version": 120, - "versionNonce": 435427470, - "isDeleted": false, - "boundElements": null, - "updated": 1751341976772, - "link": null, - "locked": false, - "text": "Registered Processors\n- Action Items Processor\n- Summary Processor\n- HA pipeline", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "left", - "verticalAlign": "top", - "containerId": "tsk-eKUlAyTfy3XOz5ngq", - "originalText": "Registered Processors\n- Action Items Processor\n- Summary Processor\n- HA pipeline", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "JNs971qoZ58e4Q6rk-DVc", - "type": "arrow", - "x": 580, - "y": 440, - "width": 220, - "height": 0, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aF", - "roundness": { - "type": 2 - }, - "seed": 1879774994, - "version": 19, - "versionNonce": 1730627278, - "isDeleted": false, - "boundElements": null, - "updated": 1751341791392, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - 220, - 0 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "rxk6Z6Cs4gccgNpwrVkdK", - "focus": -0.33333333333333337, - "gap": 1 - }, - "endBinding": { - "elementId": "tsk-eKUlAyTfy3XOz5ngq", - "focus": 0.2, - "gap": 1 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - }, - { - "id": "qrBJ2gdiZzg2hhczdDE2v", - "type": "rectangle", - "x": 180, - "y": 640, - "width": 400, - "height": 200, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aG", - "roundness": { - "type": 3 - }, - "seed": 1722212946, - "version": 30, - "versionNonce": 2115688334, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "9Pe_0sRmtP4HofQmzMZs1" - }, - { - "id": "7rgHqrDT-vJkJpLWGNRta", - "type": "arrow" - }, - { - "id": "oA3CO6Jh9f2JGbBuvqBm5", - "type": "arrow" - } - ], - "updated": 1751344829847, - "link": null, - "locked": false - }, - { - "id": "9Pe_0sRmtP4HofQmzMZs1", - "type": "text", - "x": 327.24002838134766, - "y": 727.5, - "width": 105.51994323730469, - "height": 25, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aH", - "roundness": null, - "seed": 1736328014, - "version": 12, - "versionNonce": 2091351630, - "isDeleted": false, - "boundElements": null, - "updated": 1751342017436, - "link": null, - "locked": false, - "text": "ASR, VAD, ", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "center", - "verticalAlign": "middle", - "containerId": "qrBJ2gdiZzg2hhczdDE2v", - "originalText": "ASR, VAD, ", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "xa7OfpSu0rorDHXlV5sB6", - "type": "rectangle", - "x": -500, - "y": 300, - "width": 400, - "height": 200, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aI", - "roundness": { - "type": 3 - }, - "seed": 358506834, - "version": 57, - "versionNonce": 620616914, - "isDeleted": false, - "boundElements": [ - { - "type": "text", - "id": "1iUrqVPE_SUffFLTSSZBW" - }, - { - "id": "JuRztH6XRp6DvXdwfkTPv", - "type": "arrow" - }, - { - "id": "qH2PkV-HogNFWoirkepSq", - "type": "arrow" - } - ], - "updated": 1751344783097, - "link": null, - "locked": false - }, - { - "id": "1iUrqVPE_SUffFLTSSZBW", - "type": "text", - "x": -495, - "y": 305, - "width": 375.2796630859375, - "height": 75, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aJ", - "roundness": null, - "seed": 291928786, - "version": 74, - "versionNonce": 446529422, - "isDeleted": false, - "boundElements": null, - "updated": 1751344801293, - "link": null, - "locked": false, - "text": "Backend State\n- Clients List\n- Registered Processors List Per User", - "fontSize": 20, - "fontFamily": 5, - "textAlign": "left", - "verticalAlign": "top", - "containerId": "xa7OfpSu0rorDHXlV5sB6", - "originalText": "Backend State\n- Clients List\n- Registered Processors List Per User", - "autoResize": true, - "lineHeight": 1.25 - }, - { - "id": "qH2PkV-HogNFWoirkepSq", - "type": "arrow", - "x": -100, - "y": 340, - "width": 260, - "height": 0, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aL", - "roundness": { - "type": 2 - }, - "seed": 544283342, - "version": 18, - "versionNonce": 1854829202, - "isDeleted": false, - "boundElements": null, - "updated": 1751344783097, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - 260, - 0 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "xa7OfpSu0rorDHXlV5sB6", - "focus": -0.6000000000000001, - "gap": 1 - }, - "endBinding": { - "elementId": "rxk6Z6Cs4gccgNpwrVkdK", - "focus": 1.166666666666667, - "gap": 14 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - }, - { - "id": "7rgHqrDT-vJkJpLWGNRta", - "type": "arrow", - "x": 320, - "y": 600, - "width": 0, - "height": 40, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aM", - "roundness": { - "type": 2 - }, - "seed": 1930787982, - "version": 7, - "versionNonce": 1410327502, - "isDeleted": false, - "boundElements": null, - "updated": 1751344828421, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - 0, - 40 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "rxk6Z6Cs4gccgNpwrVkdK", - "focus": 0.3, - "gap": 1 - }, - "endBinding": { - "elementId": "qrBJ2gdiZzg2hhczdDE2v", - "focus": -0.30000000000000004, - "gap": 1 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - }, - { - "id": "oA3CO6Jh9f2JGbBuvqBm5", - "type": "arrow", - "x": 380, - "y": 640, - "width": 0, - "height": 40, - "angle": 0, - "strokeColor": "#1e1e1e", - "backgroundColor": "transparent", - "fillStyle": "solid", - "strokeWidth": 2, - "strokeStyle": "solid", - "roughness": 1, - "opacity": 100, - "groupIds": [], - "frameId": null, - "index": "aN", - "roundness": { - "type": 2 - }, - "seed": 246557586, - "version": 8, - "versionNonce": 1278522830, - "isDeleted": false, - "boundElements": null, - "updated": 1751344829847, - "link": null, - "locked": false, - "points": [ - [ - 0, - 0 - ], - [ - 0, - -40 - ] - ], - "lastCommittedPoint": null, - "startBinding": { - "elementId": "qrBJ2gdiZzg2hhczdDE2v", - "focus": 0, - "gap": 1 - }, - "endBinding": { - "elementId": "rxk6Z6Cs4gccgNpwrVkdK", - "focus": 0, - "gap": 1 - }, - "startArrowhead": null, - "endArrowhead": "arrow", - "elbowed": false - } - ], - "appState": { - "gridSize": 20, - "gridStep": 5, - "gridModeEnabled": true, - "viewBackgroundColor": "#ffffff" - }, - "files": {} -} diff --git a/backends/advanced/README.md b/backends/advanced/README.md index e85636a7..01bcb9e5 100644 --- a/backends/advanced/README.md +++ b/backends/advanced/README.md @@ -2,7 +2,7 @@ A FastAPI backend with pluggable memory providers, real-time audio processing, and comprehensive conversation management. -[QuickStart](../../quickstart.md) | [Memory Providers](./MEMORY_PROVIDERS.md) | [Configuration Guide](./Docs/memories.md) +[Quick Start](../../quickstart.md) | [Memory System](../../docs/backend/memories.md) | [Configuration Guide](../../config/README.md) ## Key Features @@ -74,7 +74,7 @@ docker compose --profile https up --build -d - ๐ŸŒ **Network Access** from other devices via Tailscale/LAN - ๐Ÿ”„ **Automatic protocol detection** - Frontend auto-configures for HTTP/HTTPS -See [Docs/ssl-certificates.md](../../Docs/ssl-certificates.md) for how SSL is configured. +See [SSL certificates](../../docs/ssl-certificates.md) for how SSL is configured. ## Testing diff --git a/backends/advanced/init.py b/backends/advanced/init.py index 34525519..e7b9fc02 100644 --- a/backends/advanced/init.py +++ b/backends/advanced/init.py @@ -1825,9 +1825,9 @@ def run(self): ) self.console.print() self.console.print("For detailed documentation, see:") - self.console.print(" โ€ข Docs/quickstart.md") + self.console.print(" โ€ข quickstart.md") self.console.print(" โ€ข MEMORY_PROVIDERS.md") - self.console.print(" โ€ข CLAUDE.md") + self.console.print(" โ€ข AGENTS.md") except KeyboardInterrupt: self.console.print() diff --git a/backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py b/backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py index 350e5464..1130d40b 100644 --- a/backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py +++ b/backends/advanced/src/advanced_omi_backend/workers/conversation_jobs.py @@ -421,7 +421,7 @@ async def handle_end_of_conversation( # Session still active - enqueue new speech detection for next conversation. # Clear any TTL: the hash must live as long as the connection does, otherwise # a quiet gap > TTL expires it mid-session โ†’ next close reads status=None โ†’ - # speech detection never restarts (deaf connection). See Docs/system-oddities.md #4. + # speech detection never restarts, leaving the connection unable to receive audio. await store.persist_session(session_id) logger.info( f"๐Ÿ”„ Enqueueing new speech detection (conversation #{conversation_count + 1})" diff --git a/config/README.md b/config/README.md index 4229ba76..52a70c8c 100644 --- a/config/README.md +++ b/config/README.md @@ -147,6 +147,6 @@ Before committing `config/plugins.yml`, manually verify: ## Documentation For detailed configuration guides, see: -- `/backends/advanced/Docs/memories.md` - Memory settings +- `/docs/backend/memories.md` - Memory settings - `/quickstart.md` - Setup guide -- `/CLAUDE.md` - Project overview and technical reference +- `/AGENTS.md` - Project overview and technical reference diff --git a/config/plugins.yml.template b/config/plugins.yml.template index 2eec468d..6f3f9db7 100644 --- a/config/plugins.yml.template +++ b/config/plugins.yml.template @@ -84,7 +84,7 @@ plugins: # Double press: cross-plugin call (e.g., toggle lights via Home Assistant) # # Actions are configured in: plugins/test_button_actions/config.yml - # See: backends/advanced/Docs/plugin-development-guide.md for button event data flow + # See: docs/backend/plugin-development-guide.md for button event data flow test_button_actions: enabled: false # Set to true to enable button actions diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..f281f0c1 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,24 @@ +# Chronicle Documentation + +This directory is the canonical home for Chronicle's technical documentation. +Start with the repository [Quick Start Guide](../quickstart.md) for installation and +day-to-day operation, and use [AGENTS.md](../AGENTS.md) for development conventions. + +## System + +- [Project overview](overview.md): components, deployment topology, and repository layout +- [Audio pipeline](audio-pipeline-architecture.md): session, transcription, and memory flow +- [Initialization system](init-system.md): setup wizard and service orchestration +- [Podman](podman.md): rootless containers, GPU access, and engine migration +- [SSL certificates](ssl-certificates.md): HTTPS setup and certificate trust + +## Backend + +- [Authentication](backend/auth.md): user identity, JWTs, and protected endpoints +- [Memory system](backend/memories.md): agentic Markdown vault and retrieval +- [Plugin configuration](backend/plugin-configuration.md): configuration and secret boundaries +- [Plugin development](backend/plugin-development-guide.md): plugin lifecycle and APIs + +Service-specific instructions remain beside their implementations, such as +[`extras/speaker-recognition/`](../extras/speaker-recognition/README.md) and +[`extras/tts/`](../extras/tts/README.md). diff --git a/Docs/audio-pipeline-architecture.md b/docs/audio-pipeline-architecture.md similarity index 100% rename from Docs/audio-pipeline-architecture.md rename to docs/audio-pipeline-architecture.md diff --git a/backends/advanced/Docs/auth.md b/docs/backend/auth.md similarity index 100% rename from backends/advanced/Docs/auth.md rename to docs/backend/auth.md diff --git a/backends/advanced/Docs/memories.md b/docs/backend/memories.md similarity index 98% rename from backends/advanced/Docs/memories.md rename to docs/backend/memories.md index 5d87a172..087def94 100644 --- a/backends/advanced/Docs/memories.md +++ b/docs/backend/memories.md @@ -1,6 +1,6 @@ # Memory Service: The Agentic Markdown Vault -> ๐Ÿ“– **Prerequisite**: Read [quickstart.md](./quickstart.md) first for system overview. +> ๐Ÿ“– **Prerequisite**: Read the [Quick Start Guide](../../quickstart.md) first for system overview. This document explains how Chronicle stores and retrieves memories. diff --git a/backends/advanced/Docs/plugin-configuration.md b/docs/backend/plugin-configuration.md similarity index 97% rename from backends/advanced/Docs/plugin-configuration.md rename to docs/backend/plugin-configuration.md index a4c7b222..79b71c70 100644 --- a/backends/advanced/Docs/plugin-configuration.md +++ b/docs/backend/plugin-configuration.md @@ -393,7 +393,7 @@ ha_token = prompt_token("Home Assistant", env_file_path="../../.env") ## See Also -- [CLAUDE.md](../../../CLAUDE.md) - Main documentation -- [Plugin Development Guide](plugin-development.md) - Creating custom plugins -- [Environment Variables](environment-variables.md) - Complete .env reference -- [setup_utils.py](../../../setup_utils.py) - Shared setup utility reference +- [AGENTS.md](../../AGENTS.md) - Main development reference +- [Plugin Development Guide](plugin-development-guide.md) - Creating custom plugins +- [Configuration Guide](../../config/README.md) - Configuration and environment reference +- [setup_utils.py](../../setup_utils.py) - Shared setup utility reference diff --git a/backends/advanced/Docs/plugin-development-guide.md b/docs/backend/plugin-development-guide.md similarity index 100% rename from backends/advanced/Docs/plugin-development-guide.md rename to docs/backend/plugin-development-guide.md diff --git a/Docs/init-system.md b/docs/init-system.md similarity index 99% rename from Docs/init-system.md rename to docs/init-system.md index 4952b53f..d9167b4f 100644 --- a/Docs/init-system.md +++ b/docs/init-system.md @@ -3,7 +3,7 @@ ## Quick Links - **๐Ÿ‘‰ [Start Here: Quick Start Guide](../quickstart.md)** - Main setup path for new users -- **๐Ÿ“š [Full Documentation](../CLAUDE.md)** - Comprehensive reference +- **๐Ÿ“š [Full Documentation](../AGENTS.md)** - Comprehensive reference - **๐Ÿ—๏ธ [Architecture Details](overview.md)** - Technical deep dive - **๐Ÿง [Running with Podman](podman.md)** - Use Podman instead of Docker (engine selection, rootless/GPU) diff --git a/Docs/overview.md b/docs/overview.md similarity index 96% rename from Docs/overview.md rename to docs/overview.md index 78487675..7c709e7a 100644 --- a/Docs/overview.md +++ b/docs/overview.md @@ -101,7 +101,7 @@ chronicle/ โ”‚ โ”œโ”€โ”€ havpe-relay/ # ESP32 audio bridge โ”‚ โ””โ”€โ”€ vault-sync/ # macOS menu bar app: vault โ‡„ Obsidian via Syncthing โ”œโ”€โ”€ config/ # Central configuration -โ”œโ”€โ”€ Docs/ # Documentation +โ”œโ”€โ”€ docs/ # Documentation โ”œโ”€โ”€ tests/ # Integration tests (Robot Framework) โ”œโ”€โ”€ wizard.py # Setup wizard โ””โ”€โ”€ services.py # Service lifecycle manager @@ -128,4 +128,5 @@ open http://localhost:5173 - [Initialization System](init-system.md) โ€” Setup wizard internals and port configuration - [Audio Pipeline Architecture](audio-pipeline-architecture.md) โ€” Deep technical reference - [SSL Certificates](ssl-certificates.md) โ€” HTTPS setup -- [Backend Architecture](../backends/advanced/Docs/architecture.md) โ€” Backend internals +- [Authentication Architecture](backend/auth.md) โ€” Backend authentication internals +- [Memory System](backend/memories.md) โ€” Agentic Markdown vault architecture diff --git a/Docs/podman.md b/docs/podman.md similarity index 99% rename from Docs/podman.md rename to docs/podman.md index 9c84d8ae..7913842e 100644 --- a/Docs/podman.md +++ b/docs/podman.md @@ -156,6 +156,6 @@ one-command choice for the user, the wizard / `services.py` should automate them | Data-dir ownership remap (`sudo chown` + `podman unshare chown 999`) | a `services.py migrate-podman` helper (one-shot) | only when migrating an existing docker install | | Clear Docker Desktop `credsStore` | wizard (detect `desktop` credsStore, offer to neutralize) | harmless on fresh hosts | -Until automated, `Docs/podman.md` (this file) is the checklist. The **engine +Until automated, `docs/podman.md` (this file) is the checklist. The **engine abstraction itself** (`container_engine`/`compose_cmd`) is already seamless โ€” only the host prerequisites need wiring into setup. diff --git a/Docs/ssl-certificates.md b/docs/ssl-certificates.md similarity index 100% rename from Docs/ssl-certificates.md rename to docs/ssl-certificates.md diff --git a/extras/speaker-recognition/README.md b/extras/speaker-recognition/README.md index ed81a04b..f6ee723b 100644 --- a/extras/speaker-recognition/README.md +++ b/extras/speaker-recognition/README.md @@ -59,7 +59,7 @@ manual generation step. `./wizard.sh` configures it based on your address: - **Real domain** โ†’ Let's Encrypt, auto-renewed. - **IP / `localhost`** โ†’ Caddy's internal-CA self-signed cert (accept the browser warning). -See [../../Docs/ssl-certificates.md](../../Docs/ssl-certificates.md) for the full model +See [../../docs/ssl-certificates.md](../../docs/ssl-certificates.md) for the full model (including the Docker-Desktop `static` fallback). ### 5. Start the system @@ -351,7 +351,6 @@ When you see issues with transcript processing: For detailed documentation, API reference, and advanced usage: - **[README.detailed.md](README.detailed.md)** - Comprehensive guide -- **[plan.md](plan.md)** - Implementation details ## ๐Ÿ”ง Configuration @@ -411,7 +410,7 @@ The React UI is configured with HTTPS enabled by default (`REACT_UI_HTTPS=true`) `docker inspect speaker-recognition-caddy-1 --format '{{range .Mounts}}{{.Destination}} {{end}}'` should list `/var/run/tailscale/tailscaled.sock`. - Confirm the served cert: `echo | openssl s_client -connect localhost:8444 -servername 2>/dev/null | openssl x509 -noout -issuer -enddate` -- See [../../Docs/ssl-certificates.md](../../Docs/ssl-certificates.md) for details. +- See [../../docs/ssl-certificates.md](../../docs/ssl-certificates.md) for details. **Can't access the web UI?** - Check if services are running: `docker compose --profile cpu ps` (or `--profile gpu`) diff --git a/plugins/email_summarizer/README.md b/plugins/email_summarizer/README.md index f1a21a52..474b298f 100644 --- a/plugins/email_summarizer/README.md +++ b/plugins/email_summarizer/README.md @@ -38,7 +38,8 @@ Chronicle uses a clean three-file separation for plugin configuration: - Event subscriptions - Trigger conditions -This separation keeps secrets secure and configuration organized. See [`plugin-configuration.md`](../../../Docs/plugin-configuration.md) for details. +This separation keeps secrets secure and configuration organized. See the +[plugin configuration guide](../../docs/backend/plugin-configuration.md) for details. ## Configuration diff --git a/quickstart.md b/quickstart.md index 1758c111..2dba06a0 100644 --- a/quickstart.md +++ b/quickstart.md @@ -287,6 +287,6 @@ Before connecting your phone, make sure everything works: ## Need Help? -- **Full Documentation**: [CLAUDE.md](CLAUDE.md) - Complete technical reference -- **Architecture Details**: [Docs/overview.md](Docs/overview.md) - How everything works -- **Advanced Setup**: [Docs/init-system.md](Docs/init-system.md) - Power user options +- **Full Documentation**: [AGENTS.md](AGENTS.md) - Complete technical reference +- **Architecture Details**: [docs/overview.md](docs/overview.md) - How everything works +- **Advanced Setup**: [docs/init-system.md](docs/init-system.md) - Power user options diff --git a/tests/setup/README.md b/tests/setup/README.md index 965ef2eb..99fff19c 100644 --- a/tests/setup/README.md +++ b/tests/setup/README.md @@ -233,6 +233,6 @@ FRESH_RUN=true robot tests/ ## Related Documentation -- **[@CLAUDE.md](../../CLAUDE.md)**: Project overview and development guide +- **[@AGENTS.md](../../AGENTS.md)**: Project overview and development guide - **[@tests/README.md](../README.md)**: Testing overview (if exists) - **[@backends/advanced/README.md](../../backends/advanced/README.md)**: Backend service documentation