This file provides guidance to coding agents working in this repository.
Chronicle is at the core an AI-powered personal system - various devices, including but not limited to wearables from OMI can be used for at the very least audio capture, speaker specific transcription, memory extraction and retrieval. On top of that - it is being designed to support other services, that can help a user with these inputs such as reminders, action items, personal diagnosis etc.
This supports a comprehensive web dashboard for management.
❌ No Backward Compatibility: Do NOT add backward compatibility code unless explicitly requested. This includes fallback logic, legacy field support, or compatibility layers. Always ask before adding backward compatibility - in most cases the answer is no during active development.
Chronicle includes an interactive setup wizard for easy configuration. The wizard guides you through:
- Service selection (backend + optional services)
- Authentication setup (admin account, JWT secrets)
- Transcription provider configuration (Deepgram or offline ASR)
- LLM provider setup (OpenAI or Ollama)
- Memory configuration (agentic Markdown vault — Chronicle's single memory provider)
- Network configuration and HTTPS setup
- Optional services (speaker recognition, Parakeet ASR)
# Run the interactive setup wizard from project root (recommended)
./wizard.sh
# Or use direct command:
uv run --with-requirements setup-requirements.txt python wizard.py
# For step-by-step instructions, see quickstart.mdNote on Convenience Scripts: Chronicle provides wrapper scripts (./wizard.sh, ./start.sh, ./restart.sh, ./stop.sh, ./status.sh) that simplify the longer uv run --with-requirements setup-requirements.txt python commands. Use these for everyday operations.
For detailed setup instructions and troubleshooting, see:
- @quickstart.md: Beginner-friendly step-by-step setup guide
- @docs/init-system.md: Complete initialization system architecture and design
The initialization system uses a root orchestrator pattern:
wizard.py: Root setup orchestrator for service selection and delegationbackends/advanced/init.py: Backend configuration wizardextras/speaker-recognition/init.py: Speaker recognition setup- Service setup scripts: Individual setup for ASR services
Key features:
- Interactive prompts with validation
- API key masking and secure credential handling
- Environment file generation with placeholders
- HTTPS configuration with SSL certificate generation
- Service status display and health checks
- Automatic backup of existing configurations
cd backends/advanced
# Start full stack with Docker
docker compose up --build -d
uv run python src/main.py
# Code formatting and linting
uv run black src/
uv run isort src/
# Run tests
uv run pytest
uv run pytest tests/test_memory_service.py # Single test file
# Run integration tests (local script mirrors CI)
./run-test.sh # Complete integration test suite
# Environment setup
cp .env.template .env # Configure environment variables
# Reset data (development)
sudo rm -rf backends/advanced/data/All test operations are managed through a simple Makefile interface:
cd tests
# Full test workflow (recommended)
make test # Start containers + run all tests
# Or step by step
make start # Start test containers (with health checks)
make test-all # Run all test suites
make stop # Stop containers (preserves volumes)
# Run specific test suites
make test-endpoints # API endpoint tests (~40 tests, fast)
make test-integration # End-to-end workflows (~15 tests, slower)
make test-infra # Infrastructure resilience (~5 tests)
# Quick iteration (reuse existing containers)
make test-quick # Run tests without restarting containersAll container operations automatically preserve logs before cleanup:
make start # Start test containers
make stop # Stop containers (keep volumes)
make restart # Restart without rebuild
make rebuild # Rebuild images + restart (for code changes)
make containers-clean # SAVES LOGS → removes everything
make status # Show container health
make logs SERVICE=<name> # View specific service logsLog Preservation: All cleanup operations save container logs to tests/logs/YYYY-MM-DD_HH-MM-SS/
Test services use isolated ports and database:
- Ports: Backend (8001), MongoDB (27018), Redis (6380)
- Database:
test_db(separate from production) - Credentials:
test-admin@example.com/test-admin-password-123
For complete test documentation, see tests/README.md
cd app
# Start Expo development server
npm start
# Platform-specific builds
npm run android
npm run ios
npm run web# ASR Services
cd extras/asr-services
docker compose up parakeet-asr # Offline ASR with Parakeet
# Speaker Recognition (with tests)
cd extras/speaker-recognition
docker compose up --build
./run-test.sh # Run speaker recognition integration tests
# HAVPE Relay (ESP32 bridge)
cd extras/havpe-relay
docker compose up --build
# TTS Services (text-to-speech, run ONE provider at a time on port 8770)
cd extras/tts
docker compose up tada-tts -d --build # HumeAI TADA (GPU, voice cloning)
docker compose up fish-tts -d --build # Fish Speech (GPU, 50+ langs, emotion tags)
docker compose up kittentts-tts -d --build # KittenTTS (~25MB CPU ONNX, no GPU)
docker compose up kokoro-tts -d --build # Kokoro-82M (<~1GB VRAM GPU/CPU, preset voices)- Audio Pipeline: Real-time Opus/PCM → Application-level processing → Deepgram transcription → memory extraction
- Wyoming Protocol: WebSocket communication uses Wyoming protocol (JSONL + binary) for structured audio sessions
- Unified Pipeline: Job-based tracking system for all audio processing (WebSocket and file uploads)
- Job Tracker: Tracks pipeline jobs with stage events (audio → transcription → memory) and completion status
- Task Management: BackgroundTaskManager tracks all async tasks to prevent orphaned processes
- Unified Transcription: Deepgram transcription with fallback to offline ASR services
- Memory System: Single agentic Markdown vault — a tool-calling memory agent records conversations and surgically edits Obsidian-style People/Topic/Category notes; a read-only retrieval agent drives ripgrep over the vault to answer queries
- Authentication: Email-based login with MongoDB ObjectId user system
- Client Management: Auto-generated client IDs as
{user_id_suffix}-{device_name}, centralized ClientManager - Data Storage: MongoDB (conversations,
audio_chunks, chat, annotations), disk WAV files, and the Markdown vault (data/conversation_docs/<user_id>/) as the memory source of truth - Web Interface: React-based web dashboard with authentication and real-time monitoring
Required:
- MongoDB: User data and conversations
- Redis: Job queues (RQ workers) and session state
- FastAPI Backend: Core audio processing
- LLM Service: Memory agent (vault read/write) and action items (OpenAI or Ollama)
Recommended:
- Transcription: Deepgram or offline ASR services
Optional:
- Parakeet ASR: Offline transcription service
- Speaker Recognition: Voice identification service
- Caddy: HTTPS reverse proxy (auto-configured when HTTPS enabled)- Audio Ingestion: OMI devices stream audio via WebSocket using Wyoming protocol with JWT auth
- Wyoming Protocol Session Management: Clients send audio-start/audio-stop events for session boundaries
- Application-Level Processing: Global queues and processors handle all audio/transcription/memory tasks
- Speech-Driven Conversation Creation: User-facing conversations only created when speech is detected
- Dual Storage System: Audio sessions always stored in
audio_chunks, conversations created inconversationscollection only with speech - Versioned Processing: Transcript and memory versions tracked with active version pointers
- Memory Processing: A tool-calling memory agent records each conversation and surgically edits People/Topic/Category notes in the Markdown vault
- Memory Storage: Obsidian-style Markdown vault at
data/conversation_docs/<user_id>/— the single source of truth, searched by a read-only retrieval agent via ripgrep - Audio Optimization: Speech segment extraction removes silence automatically
- Task Tracking: BackgroundTaskManager ensures proper cleanup of all async operations
Core Principle: Conversations are only created when speech is detected, eliminating noise-only sessions from user interfaces.
Storage Architecture:
audio_chunksCollection: Always stores audio sessions byaudio_uuid(raw audio capture)conversationsCollection: Only created when speech is detected, identified byconversation_id- Speech Detection: Analyzes transcript content, duration, and meaningfulness before conversation creation
- Automatic Filtering: No user-facing conversations for silence, noise, or brief audio without speech
Benefits:
- Clean user experience with only meaningful conversations displayed
- Reduced noise in conversation lists and memory processing
- Efficient storage utilization for speech-only content
- Automatic quality filtering without manual intervention
- User System: Email-based authentication with MongoDB ObjectId user IDs
- Client Registration: Automatic
{objectid_suffix}-{device_name}format - Data Isolation: All data scoped by user_id with efficient permission checking
- API Security: JWT tokens required for all endpoints and WebSocket connections
- Admin Bootstrap: Automatic admin account creation with ADMIN_EMAIL/ADMIN_PASSWORD
# Authentication
AUTH_SECRET_KEY=your-super-secret-jwt-key-here
ADMIN_PASSWORD=your-secure-admin-password
ADMIN_EMAIL=admin@example.com
# LLM Configuration
LLM_PROVIDER=openai # or ollama
OPENAI_API_KEY=your-openai-key-here
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o-mini
# Speech-to-Text
DEEPGRAM_API_KEY=your-deepgram-key-here
# Optional: PARAKEET_ASR_URL=http://host.docker.internal:8767
# Optional: TRANSCRIPTION_PROVIDER=deepgram
# Memory Provider
MEMORY_PROVIDER=chronicle # agentic Markdown vault (only valid value)
# Database
MONGODB_URI=mongodb://mongo:27017
# Database name: chronicle
# Network Configuration
HOST_IP=localhost
BACKEND_PUBLIC_PORT=8000
WEBUI_PORT=5173 # Vite dev server (the only webui; fronted by Caddy for HTTPS)
CORS_ORIGINS=http://localhost:5173,http://localhost:8000Chronicle has a single memory provider, chronicle: an agentic Markdown vault. The vault (Obsidian-style notes at data/conversation_docs/<user_id>/) is the single source of truth. A tool-calling memory agent records each conversation and surgically edits People/Topic/Category notes; a read-only retrieval agent drives ripgrep over the vault to synthesize answers. There is no provider choice to configure — only an LLM for the agents to use.
# Memory provider (only valid value)
MEMORY_PROVIDER=chronicle
# LLM Configuration for the memory agent
LLM_PROVIDER=openai
OPENAI_API_KEY=your-openai-key-here
OPENAI_MODEL=gpt-4o-miniChronicle supports multiple transcription services:
# Option 1: Deepgram (High quality, recommended)
TRANSCRIPTION_PROVIDER=deepgram
DEEPGRAM_API_KEY=your-deepgram-key-here
# Option 2: Local ASR (Parakeet)
PARAKEET_ASR_URL=http://host.docker.internal:8767# LLM Processing
OLLAMA_BASE_URL=http://ollama:11434
# Speaker Recognition
SPEAKER_SERVICE_URL=http://speaker-recognition:8085Three-File Separation:
-
backends/advanced/.env - Secrets (gitignored)
SMTP_PASSWORD=abcdefghijklmnop OPENAI_API_KEY=sk-proj-...
-
config/plugins.yml - Orchestration (uses env var references)
plugins: email_summarizer: enabled: true smtp_password: ${SMTP_PASSWORD} # Reference, not actual value!
-
plugins/{plugin_id}/config.yml - Non-secret defaults
subject_prefix: "Conversation Summary"
CRITICAL: Never hardcode secrets in config/plugins.yml. Always use ${ENV_VAR} syntax.
- GET /health: Basic application health check
- GET /readiness: Service dependency validation
- WS /ws: Audio streaming endpoint with codec parameter (Wyoming protocol, supports pcm and opus codecs)
- GET /api/conversations: User's conversations with transcripts
- GET /api/memories/search: Agentic vault search (retrieval agent over the Markdown vault)
- POST /auth/jwt/login: Email-based login (returns JWT token)
# 1. Get auth token
curl -s -X POST \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "username=admin@example.com&password=your-password-here" \
http://localhost:8000/auth/jwt/login
# 2. Use token in API calls
curl -s -H "Authorization: Bearer YOUR_TOKEN" \
http://localhost:8000/api/conversations- Get token first: Always authenticate in a separate Bash call, store the token, then use it in subsequent calls. Never chain login + API call in one command.
- Read .env with Read tool: Use the Read tool to get values from
.envfiles. Don't usegrep | sed | cutin Bash to extract env values. - Keep Bash simple: Each Bash call should do one thing. Don't string together complex piped commands for backend queries.
# Reset all data (development only)
cd backends/advanced
sudo rm -rf data/
# Reset Docker volumes
docker compose down -v
docker compose up --build -dThe system supports processing existing audio files through the file upload API. This allows you to import and process pre-recorded conversations without requiring a live WebSocket connection.
Upload and Process WAV Files:
export USER_TOKEN="your-jwt-token"
# Upload single WAV file
curl -X POST "http://localhost:8000/api/audio/upload" \
-H "Authorization: Bearer $USER_TOKEN" \
-F "files=@/path/to/audio.wav" \
-F "device_name=file_upload"
# Upload multiple WAV files
curl -X POST "http://localhost:8000/api/audio/upload" \
-H "Authorization: Bearer $USER_TOKEN" \
-F "files=@/path/to/recording1.wav" \
-F "files=@/path/to/recording2.wav" \
-F "device_name=import_batch"Response Example:
{
"message": "Successfully processed 2 audio files",
"processed_files": [
{
"filename": "recording1.wav",
"sample_rate": 16000,
"channels": 1,
"duration_seconds": 120.5,
"size_bytes": 3856000
},
{
"filename": "recording2.wav",
"sample_rate": 44100,
"channels": 2,
"duration_seconds": 85.2,
"size_bytes": 7532800
}
],
"client_id": "user01-import_batch"
}For ESP32 audio streaming using the HAVPE relay (extras/havpe-relay/):
# Environment variables for HAVPE relay
export AUTH_USERNAME="user@example.com" # Email address
export AUTH_PASSWORD="your-password"
export DEVICE_NAME="havpe" # Device identifier
# Run the relay
cd extras/havpe-relay
uv run python main.py --backend-url http://your-server:8000 --backend-ws-url ws://your-server:8000The relay will automatically:
- Authenticate using
AUTH_USERNAME(email address) - Generate client ID as
objectid_suffix-havpe - Forward ESP32 audio to the backend with proper authentication
- Handle token refresh and reconnection
Provider-based text-to-speech (extras/tts/), built on the same provider pattern as extras/asr-services/. Run one provider at a time, all serving on port 8770 (configurable via TTS_PORT).
| Provider | Service | Hardware | Highlights |
|---|---|---|---|
| TADA (HumeAI) | tada-tts |
GPU | Zero-shot voice cloning, 1:1 token alignment (no hallucinations), MIT. tada-1b (English) / tada-3b-ml (9 langs). Needs HF_TOKEN (Llama 3.2 base is gated). |
| Fish Speech (Fish Audio) | fish-tts |
GPU | Dual-AR, 50+ langs, inline emotion/prosody tags ([laugh], [whispers]), streaming. s2-pro (default) / openaudio-s1-mini / fish-speech-1.5. Optional torch.compile. |
| KittenTTS (KittenML) | kittentts-tts |
CPU | Ultra-light (~25MB) ONNX, no GPU/API key, preset voices, English only. Uses dedicated KITTEN_TTS_* env vars. |
| Kokoro (hexgrad) | kokoro-tts |
GPU/CPU | Lightweight (~82M, <~1GB VRAM) StyleTTS2, preset voices, 8 langs, Apache-2.0. Quality-per-VRAM sweet spot. Uses dedicated KOKORO_TTS_* env vars. |
cd extras/tts
# Configure (selects provider, model, CUDA version)
uv run --with-requirements ../../setup-requirements.txt python init.py
# Start ONE provider
docker compose up tada-tts -d --build # or fish-tts / kittentts-tts
# Test
curl http://localhost:8770/health
curl -X POST http://localhost:8770/synthesize -F "text=Hello world." -o output.wav| Endpoint | Method | Description |
|---|---|---|
/health |
GET | Service health (healthy / initializing) |
/info |
GET | Model id, provider, capabilities, supported languages |
/synthesize |
POST | Generate speech (multipart form) |
POST /synthesize — text (required); optional reference_audio (WAV) + reference_text for voice cloning; optional generation params (temperature, top_p, repetition_penalty, seed, max_new_tokens). Returns WAV bytes with X-Sample-Rate, X-Provider, X-Model headers.
Notes:
- Not registered in
services.py— manage withdocker composedirectly (like the HAVPE relay). - GPU providers require CUDA 12.6+ (
PYTORCH_CUDA_VERSION=cu126/cu128);cu121is unsupported (torch>=2.7). - Add a provider by creating
extras/tts/providers/{name}/withservice.py,synthesizer.py, andDockerfile(subclassBaseTTSService). - An optional
edge-agentsidecar (--profile edge) advertises the service on the Tailnet.
Single Machine (Default):
# Everything on one machine
docker compose up --build -dDistributed Setup (GPU + Backend separation):
# Start GPU-accelerated services
cd extras/asr-services
docker compose up moonshine -d
cd extras/speaker-recognition
docker compose up --build -d
# Ollama with GPU support
docker run -d --gpus=all -p 11434:11434 \
-v ollama:/root/.ollama \
ollama/ollama:latest# .env configuration for distributed services
OLLAMA_BASE_URL=http://[gpu-machine-tailscale-ip]:11434
SPEAKER_SERVICE_URL=http://[gpu-machine-tailscale-ip]:8085
PARAKEET_ASR_URL=http://[gpu-machine-tailscale-ip]:8080
# Start lightweight backend services
docker compose up --build -d# Install on each machine
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# Find machine IPs
tailscale ip -4Benefits of Distributed Setup:
- GPU services on dedicated hardware
- Lightweight backend on VPS/Raspberry Pi
- Automatic Tailscale IP support (100.x.x.x) - no CORS configuration needed
- Encrypted inter-service communication
Service Examples:
- GPU machine: LLM inference, ASR, speaker recognition
- Backend machine: FastAPI, WebUI, databases
- Database machine: MongoDB (optional separation)
- Backend: Uses
uvfor Python dependency management (faster than pip) - Mobile: Uses
npmwith React Native and Expo - Containers: The service lifecycle (
services.py/status.py/service-manager) supports both Docker and Podman, selected viacontainer_engine: docker|podmaninconfig/config.yml(or theCONTAINER_ENGINE/COMPOSE_CMDenv vars). The repo's compose files run unmodified under either engine. For Podman it drivespodman-composeand needs CDI for GPU. See @docs/podman.md for rootless/GPU setup and migration notes.
- Makefile-Based: All test operations through simple
makecommands (make test,make start,make stop) - Log Preservation: Container logs always saved before cleanup (never lose debugging info)
- End-to-End Integration: Robot Framework validates complete audio processing pipeline
- Environment Flexibility: Tests work with both local .env files and CI environment variables
- CI/CD Integration: Same test logic locally and in GitHub Actions
- Python: Black formatter with 100-character line length, isort for imports
- TypeScript: Standard React Native conventions
- Import Guidelines:
- NEVER import modules in the middle of functions or files
- ALL imports must be at the top of the file after the docstring
- Use lazy imports sparingly and only when absolutely necessary for circular import issues
- Group imports: standard library, third-party, local imports
- Error Handling Guidelines:
- Always raise errors, never silently ignore: Use explicit error handling with proper exceptions rather than silent failures
- Understand data structures: Research and understand input/response or class structure instead of adding defensive
hasattr()checks
- Default Behavior: Docker automatically detects file changes in Dockerfile COPY/ADD instructions and invalidates cache as needed
- No --no-cache by Default: Only use
--no-cachewhen explicitly needed (e.g., package updates, dependency issues) - Smart Caching: Docker checks file modification times and content hashes to determine when rebuilds are necessary
- Development Efficiency: Trust Docker's cache system - it handles most development scenarios correctly
The system includes comprehensive health checks:
/readiness: Service dependency validation/health: Basic application status- Memory debug system for transcript processing monitoring
- Makefile Interface: Simple
makecommands for all operations (seetests/README.md) - Test Environment:
docker-compose-test.ymlwith isolated services on separate ports - Test Database: Uses
test_dbdatabase (separate from production) - Log Preservation: All cleanup operations save logs to
tests/logs/automatically - CI Compatibility: Same test logic runs locally and in GitHub Actions
For detailed technical documentation, see:
- @docs/README.md: Documentation index
- @docs/overview.md: Architecture overview and technical deep dive
- @docs/init-system.md: Initialization system and service management
- @docs/ssl-certificates.md: HTTPS/SSL setup details
- @docs/podman.md: Running with Podman instead of Docker (engine selection, rootless/GPU setup)
- @docs/audio-pipeline-architecture.md: Audio pipeline design
- @docs/backend/auth.md: Authentication architecture
- @docs/backend/memories.md: Memory system documentation
- @docs/backend/plugin-development-guide.md: Plugin development guide
IMPORTANT: When writing or modifying Robot Framework tests, you MUST follow the testing guidelines.
Before writing any Robot Framework test:
- Read @tests/TESTING_GUIDELINES.md for comprehensive testing patterns and standards
- Check @tests/tags.md for approved tags - ONLY 11 tags are permitted
- SCAN existing resource files for keywords - NEVER write code that duplicates existing keywords
- Follow the Arrange-Act-Assert pattern with inline verifications (not abstracted to keywords)
Key Testing Rules:
- Check Existing Keywords FIRST: Before writing ANY test code, scan relevant resource files (
websocket_keywords.robot,queue_keywords.robot,conversation_keywords.robot, etc.) for existing keywords - Tags: ONLY use the 11 approved tags from tags.md, tab-separated (e.g.,
[Tags] infra audio-streaming) - Verifications: Write assertions directly in tests, not in resource keywords
- Keywords: Only create keywords for reusable setup/action operations AFTER confirming no existing keyword exists
- Resources: Always check existing resource files before creating new keywords or duplicating logic
- Naming: Use descriptive names that explain business purpose, not technical implementation
DO NOT:
- Write inline code without checking if a keyword already exists for that operation
- Create custom tags (use only the 11 approved tags)
- Abstract verifications into keywords (keep them inline in tests)
- Use space-separated tags (must be tab-separated)
- Skip reading the guidelines before writing tests
Check if the src/ is volume mounted. If not, do compose build so that code changes are reflected. Do not simply run docker compose restart as it will not rebuild the image.
Check docs/backend/ for up-to-date information on the advanced backend.
All docker projects have .dockerignore following the exclude pattern. That means files need to be included for them to be visible to docker.
The uv package manager is used for all python projects. Wherever you'd call python3 main.py you'd call uv run python main.py
Container Engine (Docker or Podman):
- The project supports both Docker and Podman. The active engine is set by
container_engineinconfig/config.yml(defaultdocker); prefer the lifecycle scripts (./start.sh/./stop.sh/./restart.sh) which route through the selected engine. For one-off manual commands under Podman usepodman-compose(notdocker compose). See @docs/podman.md.
Docker Build Guidelines:
-
Use
docker compose build(orpodman-compose build) without--no-cacheby default for faster builds -
Only use
--no-cachewhen explicitly needed (e.g., if cached layers are causing issues or when troubleshooting build problems) -
The build cache is efficient and saves significant time during development
-
Remember that whenever there's a python command, you should use uv run python3 instead