Skip to content

feat: Add OCR text extraction and hybrid search #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
VedankPurohit merged 9 commits into from
Jan 11, 2026
Merged

feat: Add OCR text extraction and hybrid search #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
2570597
feat: Add OCR and hybrid search capabilities
VedankPurohit Jan 11, 2026
71ce921
fix: Thread safety, OCR viewer, and batch embedding optimizations
VedankPurohit Jan 11, 2026
80c4c72
fix: Setup page model status detection and add OCR benchmark
VedankPurohit Jan 11, 2026
ca0b495
fix: CI lint and test failures
VedankPurohit Jan 11, 2026
9ba2529
lint fix
VedankPurohit Jan 11, 2026
9cea0f3
fix: Address CodeRabbit review suggestions
VedankPurohit Jan 11, 2026
02e07f9
refactor: Use module-level singleton for Apple Vision OCR
VedankPurohit Jan 11, 2026
5d05f2d
fix: Correct FTS result key name (relevance_score not fts_rank)
VedankPurohit Jan 11, 2026
7290dd9
fix: Type safety and thread-safety improvements
VedankPurohit Jan 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,29 @@ uv run python scripts/build_release.py --quick # Skip web rebuild
### Key Components

- **core/capture.py** - CaptureService: Screenshot capture using mss, SSIM-based change detection
- **core/database.py** - SQLite + sqlite-vec for vector similarity search
- **core/database.py** - SQLite + sqlite-vec for vector similarity search (thread-safe with Lock)
- **core/embeddings.py** - CLIP model with lazy loading and 5-minute auto-unload
- **core/processor.py** - Background sync service for processing screenshots
- **core/text_embeddings.py** - Text embedding model for OCR semantic search
- **core/ocr.py** - OCR service with pluggable providers (Apple Vision / Tesseract)
- **core/chunking.py** - Dual-size text chunking (small 512 + large 2048 tokens)
- **core/processor.py** - Background sync service for processing screenshots (CLIP + OCR + text embeddings)
- **tray/app.py** - TrayApp: System tray menu, spawns backend via subprocess
- **tray/backend.py** - Manages FastAPI server as subprocess
- **api/main.py** - FastAPI app that serves REST API + static Next.js build from web/out/

### OCR & Text Search Architecture

The sync pipeline processes screenshots through:
1. CLIP image embedding → `screenshot_embeddings` table (512-dim vectors)
2. OCR text extraction → `screenshot_ocr` table (full text + confidence)
3. Text chunking → `ocr_text_chunks` table (dual sizes for precision vs context)
4. Text embeddings → `ocr_text_embeddings` table (384-dim vectors)

Comment on lines +48 to +55

@coderabbitai coderabbitai Bot Jan 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

rg -n --hidden -S "screenshot_embeddings|ocr_text_embeddings|vec0|sqlite-vec|CREATE VIRTUAL TABLE" --type-list

Repository: VedankPurohit/LiveRecall

Length of output: 457

🏁 Script executed:

rg -n "screenshot_embeddings|ocr_text_embeddings|vec0|sqlite-vec|CREATE VIRTUAL TABLE" -A 3 -B 1

Repository: VedankPurohit/LiveRecall

Length of output: 10094

🏁 Script executed:

find . -name "embeddings.py" -o -name "*.sql" -o -name "schema*" | head -20

Repository: VedankPurohit/LiveRecall

Length of output: 105

🏁 Script executed:

cat -n CLAUDE.md | sed -n '48,55p; 98,105p'

Repository: VedankPurohit/LiveRecall

Length of output: 1025

🏁 Script executed:

rg -n "EMBEDDING_DIM|TEXT_EMBEDDING_DIM" core/database.py | head -20

Repository: VedankPurohit/LiveRecall

Length of output: 846

🏁 Script executed:

head -100 core/database.py | grep -n "EMBEDDING_DIM\|TEXT_EMBEDDING_DIM\|import"

Repository: VedankPurohit/LiveRecall

Length of output: 319

🏁 Script executed:

cat core/database.py | head -40

Repository: VedankPurohit/LiveRecall

Length of output: 1139

🏁 Script executed:

cat core/embeddings.py

Repository: VedankPurohit/LiveRecall

Length of output: 8138

Update documented embedding dimensions: CLIP ViT-L-14 produces 768-dim vectors, not 512-dim.

The code in core/database.py defines EMBEDDING_DIM = 768 with a comment confirming "CLIP ViT-L-14 produces 768-dimensional embeddings." The schema and validation logic consistently enforce this constraint. Update CLAUDE.md lines 51 and 100 to replace "512-dim vectors" with "768-dim vectors." The 384-dim claim for text embeddings (lines 54, 103) is correct and uses BGE-small-en-v1.5.

🤖 Prompt for AI Agents 
In @CLAUDE.md around lines 48 - 55, Update the documented image embedding
dimension in CLAUDE.md from "512-dim vectors" to "768-dim vectors" to match the
code's EMBEDDING_DIM = 768 (CLIP ViT-L-14); specifically replace the phrase at
the screenshot_embeddings description (around the first occurrence) and the
second occurrence later in the file, leaving the "384-dim" text embedding
references for BGE-small-en-v1.5 unchanged.

Search uses hybrid Reciprocal Rank Fusion combining:
- Image semantic search (CLIP)
- Text fuzzy search (FTS5 trigram)
- Text semantic search (small + large chunks)

### API Routes Structure

Routes in `api/routes/` with prefixes: `/recording`, `/sync`, `/search`, `/screenshots`, `/compression`, `/incognito`, `/setup`
Expand Down Expand Up @@ -79,6 +95,14 @@ async def get_something():

macOS: `~/Library/Application Support/LiveRecall/` (SQLite database + screenshots folder)

### Database Tables
- `screenshots` - Screenshot metadata (id, path, timestamp, has_embedding, has_ocr)
- `screenshot_embeddings` - CLIP image vectors (512-dim, sqlite-vec)
- `screenshot_ocr` - OCR extracted text (full_text, confidence, word_count)
- `ocr_text_chunks` - Dual-size chunks (small 512 tokens, large 2048 tokens)
- `ocr_text_embeddings` - Text embedding vectors (384-dim, sqlite-vec)
- `ocr_text_fts` - FTS5 full-text search index (trigram tokenizer)

## API Documentation

When running: http://localhost:8742/docs (Swagger) or http://localhost:8742/redoc
36 changes: 31 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
## Features

- **Semantic Search** - Find screenshots by describing what you're looking for
- **OCR Text Extraction** - Automatically extracts text from screenshots for text-based search
- **Hybrid Search** - Combines image similarity and text matching for best results
- **Smart Capture** - Only saves when screen content changes
- **System Tray App** - Runs quietly in your menu bar
- **Web Interface** - Beautiful timeline and search UI
Expand All @@ -20,7 +22,9 @@
3. Launch LiveRecall from Applications
4. Grant **Screen Recording** permission when prompted (System Settings > Privacy & Security > Screen Recording)

**Note:** On first launch, the CLIP model (~400MB) will be downloaded automatically.
**Note:** On first launch, required AI models will be downloaded automatically:
- Image embedding model (~400MB) for visual search
- Text embedding model (~130MB) for OCR text search

### Windows / Intel Mac

Expand Down Expand Up @@ -121,8 +125,27 @@ LiveRecall/

1. **Capture**: Screenshots are taken at regular intervals when screen content changes
2. **Storage**: Images saved to `~/Library/Application Support/LiveRecall/` (macOS)
3. **Sync**: CLIP model generates embeddings for semantic search (runs on-demand)
4. **Search**: Natural language queries matched against embeddings
3. **Sync**: Processing pipeline generates embeddings and extracts text (runs on-demand)
4. **Search**: Natural language queries matched against image and text content

### Search Modes

LiveRecall supports multiple search modes:

- **Auto (Hybrid)** - Default. Combines all search methods using Reciprocal Rank Fusion for best results
- **Image** - Finds screenshots where the visual content matches your query (e.g., "cat on a keyboard")
- **Text Fuzzy** - Fast text matching with typo tolerance using trigram search
- **Text Semantic** - Finds screenshots containing text with similar meaning to your query

### OCR Processing

When sync runs, each screenshot goes through:
1. **Image embedding** - Visual features extracted for semantic image search
2. **OCR extraction** - Text extracted using platform-native OCR (Vision on macOS, Tesseract on Windows)
3. **Text chunking** - Extracted text split into searchable chunks
4. **Text embedding** - Semantic embeddings generated for text search

You can view extracted text for any screenshot using the "View Text" button in the web UI.

## API Endpoints

Expand All @@ -131,9 +154,10 @@ LiveRecall/
| `/api/v1/status` | GET | System status |
| `/api/v1/recording/start` | POST | Start capture |
| `/api/v1/recording/stop` | POST | Stop capture |
| `/api/v1/sync/start` | POST | Start embedding sync |
| `/api/v1/search` | POST | Semantic search |
| `/api/v1/sync/start` | POST | Start embedding/OCR sync |
| `/api/v1/search` | POST | Hybrid search (supports search_mode parameter) |
| `/api/v1/screenshots` | GET | List screenshots |
| `/api/v1/screenshots/{id}/ocr` | GET | Get OCR text for a screenshot |

Full API docs at http://localhost:8742/docs

Expand All @@ -146,6 +170,8 @@ Settings available in the web UI or via API:
- **Quality**: 50-100%
- **Safe Mode**: Filter sensitive content from search
- **Auto-compress**: Compress old screenshots
- **OCR Enabled**: Enable/disable text extraction (enabled by default)
- **Search Mode**: auto (hybrid), image, text_fuzzy, text_semantic

## Privacy & Security

Expand Down
19 changes: 18 additions & 1 deletion api/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from fastapi.responses import FileResponse, RedirectResponse
from fastapi.staticfiles import StaticFiles

from api.routes import compression, incognito, recording, screenshots, search, setup, status, sync
from api.routes import compression, events, incognito, recording, screenshots, search, setup, status, sync
from core.database import db


Expand Down Expand Up @@ -48,6 +48,21 @@ async def lifespan(app: FastAPI):
print(f"📁 Database: {db.db_path}")
print(f"📊 Screenshots: {stats['total_screenshots']} total, {stats['synced']} synced")

# Auto-start OCR migration if needed (for existing screenshots without OCR)
# This does NOT process new screenshots - those still require manual sync
from core.config import config
from core.processor import processor_service

if config.ocr.enabled:
try:
ocr_pending = db.get_ocr_pending_count()
if ocr_pending > 0:
print(f"🔄 Starting OCR migration for {ocr_pending} existing screenshots...")
processor_service.start_ocr_migration()
except Exception as e:
# Don't crash the API if OCR migration fails - log and continue
logging.getLogger(__name__).exception("Failed to start OCR migration: %s", e)

Comment on lines +51 to +65

@coderabbitai coderabbitai Bot Jan 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify whether OCR migration is safe under multi-process or guarded elsewhere.
rg -n --hidden -S "start_ocr_migration\(|_ocr_migration_loop|has_ocr\s*=\s*0" core api | head -n 80

Repository: VedankPurohit/LiveRecall

Length of output: 192

🏁 Script executed:

#!/bin/bash
# Find core/processor.py and examine start_ocr_migration implementation
find . -name "processor.py" -type f

Repository: VedankPurohit/LiveRecall

Length of output: 87

🏁 Script executed:

#!/bin/bash
# Look at api/main.py context around lines 40-70 to see full startup logic and error handling
fd -e py "main.py" api | xargs cat -n | sed -n '40,70p'

Repository: VedankPurohit/LiveRecall

Length of output: 1335

🏁 Script executed:

#!/bin/bash
# Read core/processor.py to understand start_ocr_migration implementation
wc -l core/processor.py

Repository: VedankPurohit/LiveRecall

Length of output: 89

🏁 Script executed:

#!/bin/bash
# Look for start_ocr_migration definition
rg -n "def start_ocr_migration" core/processor.py -A 20

Repository: VedankPurohit/LiveRecall

Length of output: 1022

🏁 Script executed:

#!/bin/bash
# Check for locking or process-safety mechanisms
rg -n "Lock|lock|Semaphore|process_safe|multi.*worker|uvicorn" core/processor.py

Repository: VedankPurohit/LiveRecall

Length of output: 233

🏁 Script executed:

#!/bin/bash
# Check if exception handling exists elsewhere in startup or if crashes block app
rg -n "try:|except|Exception" core/processor.py | head -20

Repository: VedankPurohit/LiveRecall

Length of output: 967

🏁 Script executed:

#!/bin/bash
# Look at db.get_ocr_pending_count() to check for process safety
rg -n "def get_ocr_pending_count" core -A 10

Repository: VedankPurohit/LiveRecall

Length of output: 669

🏁 Script executed:

#!/bin/bash
# Check if there's any inter-process locking or startup synchronization elsewhere
rg -n "flock|fcntl|multiprocessing.*Lock|db.*lock" core api --hidden | head -10

Repository: VedankPurohit/LiveRecall

Length of output: 50

🏁 Script executed:

#!/bin/bash
# Verify if uvicorn/FastAPI has any built-in single-startup mechanism that prevents duplication
rg -n "uvicorn|workers|--workers" . --type py -B2 -A2 | head -20

Repository: VedankPurohit/LiveRecall

Length of output: 722

🏁 Script executed:

#!/bin/bash
# Check if there's a startup guard or singleton pattern elsewhere in api/main.py
cat -n api/main.py | sed -n '1,80p'

Repository: VedankPurohit/LiveRecall

Length of output: 3175

Add exception handling and guard multi-worker OCR migration duplication.

When uvicorn runs with multiple workers, each process independently calls start_ocr_migration() during startup, causing duplicated OCR work unless the processor/db layer is explicitly process-safe. Additionally, any exception during migration startup will currently crash the API.

Proposed fix 
    if config.ocr.enabled:
+        try:
             ocr_pending = db.get_ocr_pending_count()
             if ocr_pending > 0:
+                # NOTE: Ensure this is safe under multi-process deployments (uvicorn workers > 1).
                 print(f"🔄 Starting OCR migration for {ocr_pending} existing screenshots...")
                 processor_service.start_ocr_migration()
+        except Exception as e:
+            # Don't take down the API if OCR migration can't start; surface it in logs.
+            logging.getLogger(__name__).exception("Failed to start OCR migration: %s", e)

yield

# Shutdown
Expand Down Expand Up @@ -122,6 +137,7 @@ async def strip_trailing_slash(request: Request, call_next):
app.include_router(compression.router, prefix="/api/v1")
app.include_router(incognito.router, prefix="/api/v1")
app.include_router(setup.router, prefix="/api/v1")
app.include_router(events.router, prefix="/api/v1")


# Serve static web UI if available
Expand Down Expand Up @@ -195,4 +211,5 @@ async def root():
host="127.0.0.1",
port=8742,
reload=True,
log_level="warning", # Suppress verbose access logs

@coderabbitai coderabbitai Bot Jan 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# First, let's see the context around line 210 in api/main.py
sed -n '200,220p' api/main.py

Repository: VedankPurohit/LiveRecall

Length of output: 343

🏁 Script executed:

# Let's also see more context to understand what object/function uses log_level
sed -n '190,230p' api/main.py

Repository: VedankPurohit/LiveRecall

Length of output: 665

🏁 Script executed:

# Search for uvicorn.access configuration and logger setup
rg -n "uvicorn|access|logger" api/main.py -A 2 -B 2

Repository: VedankPurohit/LiveRecall

Length of output: 556

🏁 Script executed:

# Search for SSE or progress-related logging
rg -n "SSE|progress|stream|info|debug" api/main.py

Repository: VedankPurohit/LiveRecall

Length of output: 118

🏁 Script executed:

# Search for logging configuration in the entire codebase
rg -n "logging\.config|getLogger|uvicorn" --type py

Repository: VedankPurohit/LiveRecall

Length of output: 980

🏁 Script executed:

# Search for SSE, streaming, WebSocket, or progress event implementations
rg -n "sse|streaming|websocket|progress|EventSourceResponse|Server-Sent" --type py -i

Repository: VedankPurohit/LiveRecall

Length of output: 50380

🏁 Script executed:

# Search for any logging calls in the app (especially info level)
rg -n "\.info\(|\.debug\(" --type py api/

Repository: VedankPurohit/LiveRecall

Length of output: 50

🏁 Script executed:

# Check if there are any custom loggers or logging config that might be affected
rg -n "getLogger|StreamingResponse|yield|event" --type py api/main.py -B 1 -A 1

Repository: VedankPurohit/LiveRecall

Length of output: 572

Clarify logging level configuration and confirm it doesn't inadvertently suppress uvicorn's non-access logs.

The explicit logging.getLogger("uvicorn.access").setLevel(logging.WARNING) at line 38 targets only access logs. However, log_level="warning" at line 210 sets uvicorn's root logging level broadly. Depending on uvicorn's logger hierarchy, these may be redundant or redundantly cautious. Since the app tracks progress via SyncProgress callbacks and SSE endpoints rather than logging, the impact is likely minimal, but verify this is intentional and doesn't suppress any other uvicorn INFO-level diagnostics you may need during development.

🤖 Prompt for AI Agents 
In @api/main.py at line 210, The current setup sets uvicorn.access logger to
WARNING via logging.getLogger("uvicorn.access").setLevel(...) but also passes
log_level="warning" into uvicorn.run (symbol: uvicorn.run in main.py), which
sets the root/uvicorn logger level and may unintentionally suppress non-access
INFO logs; decide whether you want to silence only access logs or all uvicorn
logs and update accordingly—either remove or change log_level="warning" to
"info"/None to preserve uvicorn INFO diagnostics, or keep log_level="warning"
and instead explicitly set only "uvicorn.access" to WARNING (and optionally
adjust "uvicorn.error"/root via
logging.getLogger("uvicorn.error"/"uvicorn").setLevel(...)) so behavior matches
your intent (then run a quick dev start to confirm SyncProgress/SSE messages
still appear).

)
Loading