Summary
Add an optional --db <path> CLI flag that persists all search results (and scraped page data when --scrape is used) into a lightweight SQLite database. This enables users to build up a queryable local archive of searches without external dependencies.
Current Behavior
sophon-cli prints search results to stdout and discards them. There is no built-in way to save result metadata, URLs, snippets, or scraped page content for later analysis.
Expected Behavior
When --db results.db is passed:
- Search results are written to
results.db immediately after the API response is received.
- If
--scrape is also enabled, scraped page content and crawl telemetry are appended to the same database.
- The schema is lightweight and self-contained (SQLite, no external server).
Technical Context
- Entrypoint:
src/cli/runner.rs (run_single_provider and run_all_enabled)
- CLI args:
src/cli/args.rs
- Result types:
src/domain/result.rs (SearchResponse, SearchBatchResponse, SearchResult variants)
- Scrape types:
src/cli/scrape.rs (ScrapedSite, ScrapedPage)
- Architecture boundary: DB code belongs in the CLI layer only; domain must remain pure.
Proposed Schema
CREATE TABLE search_runs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
query TEXT NOT NULL,
provider TEXT NOT NULL,
run_at TEXT NOT NULL
);
CREATE TABLE search_results (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id INTEGER NOT NULL,
result_type TEXT NOT NULL,
title TEXT NOT NULL,
url TEXT NOT NULL,
snippet TEXT,
source TEXT,
published_at TEXT,
thumbnail_url TEXT,
duration TEXT,
display_url TEXT
);
CREATE TABLE scraped_sites (
id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id INTEGER NOT NULL,
seed_url TEXT NOT NULL,
duration_ms INTEGER,
page_limit INTEGER,
error TEXT
);
CREATE TABLE scraped_pages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
site_id INTEGER NOT NULL,
url TEXT NOT NULL,
status_code INTEGER,
content TEXT
);
Implementation Notes
- Use
rusqlite with the bundled feature for portability (zero system dependency on libsqlite3).
- DB writes can be synchronous (
rusqlite is sync); wrap in tokio::task::spawn_blocking if called from async context, or keep it simple since writes are tiny.
- For
--provider all, create one search_runs row per successful provider response, then one shared row for scraped data if --scrape is active.
- Consider adding
DbWriter in a new src/cli/db.rs module.
Acceptance Criteria
Related Issues
None yet.
Summary
Add an optional
--db <path>CLI flag that persists all search results (and scraped page data when--scrapeis used) into a lightweight SQLite database. This enables users to build up a queryable local archive of searches without external dependencies.Current Behavior
sophon-cliprints search results to stdout and discards them. There is no built-in way to save result metadata, URLs, snippets, or scraped page content for later analysis.Expected Behavior
When
--db results.dbis passed:results.dbimmediately after the API response is received.--scrapeis also enabled, scraped page content and crawl telemetry are appended to the same database.Technical Context
src/cli/runner.rs(run_single_providerandrun_all_enabled)src/cli/args.rssrc/domain/result.rs(SearchResponse,SearchBatchResponse,SearchResultvariants)src/cli/scrape.rs(ScrapedSite,ScrapedPage)Proposed Schema
Implementation Notes
rusqlitewith thebundledfeature for portability (zero system dependency on libsqlite3).rusqliteis sync); wrap intokio::task::spawn_blockingif called from async context, or keep it simple since writes are tiny.--provider all, create onesearch_runsrow per successful provider response, then one shared row for scraped data if--scrapeis active.DbWriterin a newsrc/cli/db.rsmodule.Acceptance Criteria
cargo run -- "query" --db test.dbcreatestest.dband populatessearch_runs+search_results.cargo run -- "query" --db test.db --scrapealso populatesscraped_sites+scraped_pages.--provider allstores results from each provider under distinct rows.just checkpasses (fmt, clippy, tests, docs build).src/cli/args.rsvalidates--dbparses correctly.rusqliteorcrate::cli::db.Related Issues
None yet.