Skip to content

feat: add optional --db flag to persist search results to SQLite #14

Description

@larock22

Summary

Add an optional --db <path> CLI flag that persists all search results (and scraped page data when --scrape is used) into a lightweight SQLite database. This enables users to build up a queryable local archive of searches without external dependencies.

Current Behavior

sophon-cli prints search results to stdout and discards them. There is no built-in way to save result metadata, URLs, snippets, or scraped page content for later analysis.

Expected Behavior

When --db results.db is passed:

  • Search results are written to results.db immediately after the API response is received.
  • If --scrape is also enabled, scraped page content and crawl telemetry are appended to the same database.
  • The schema is lightweight and self-contained (SQLite, no external server).

Technical Context

  • Entrypoint: src/cli/runner.rs (run_single_provider and run_all_enabled)
  • CLI args: src/cli/args.rs
  • Result types: src/domain/result.rs (SearchResponse, SearchBatchResponse, SearchResult variants)
  • Scrape types: src/cli/scrape.rs (ScrapedSite, ScrapedPage)
  • Architecture boundary: DB code belongs in the CLI layer only; domain must remain pure.

Proposed Schema

CREATE TABLE search_runs (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  query TEXT NOT NULL,
  provider TEXT NOT NULL,
  run_at TEXT NOT NULL
);

CREATE TABLE search_results (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  run_id INTEGER NOT NULL,
  result_type TEXT NOT NULL,
  title TEXT NOT NULL,
  url TEXT NOT NULL,
  snippet TEXT,
  source TEXT,
  published_at TEXT,
  thumbnail_url TEXT,
  duration TEXT,
  display_url TEXT
);

CREATE TABLE scraped_sites (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  run_id INTEGER NOT NULL,
  seed_url TEXT NOT NULL,
  duration_ms INTEGER,
  page_limit INTEGER,
  error TEXT
);

CREATE TABLE scraped_pages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  site_id INTEGER NOT NULL,
  url TEXT NOT NULL,
  status_code INTEGER,
  content TEXT
);

Implementation Notes

  • Use rusqlite with the bundled feature for portability (zero system dependency on libsqlite3).
  • DB writes can be synchronous (rusqlite is sync); wrap in tokio::task::spawn_blocking if called from async context, or keep it simple since writes are tiny.
  • For --provider all, create one search_runs row per successful provider response, then one shared row for scraped data if --scrape is active.
  • Consider adding DbWriter in a new src/cli/db.rs module.

Acceptance Criteria

  • cargo run -- "query" --db test.db creates test.db and populates search_runs + search_results.
  • cargo run -- "query" --db test.db --scrape also populates scraped_sites + scraped_pages.
  • Running with --provider all stores results from each provider under distinct rows.
  • just check passes (fmt, clippy, tests, docs build).
  • Unit test in src/cli/args.rs validates --db parses correctly.
  • No domain-layer imports of rusqlite or crate::cli::db.

Related Issues

None yet.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions