feat(cli): optional --db SQLite persistence and --scrape - #16
Draft
tunahorse wants to merge 52 commits into
Draft
Conversation
Fix HARNESS canonical gate chain
Add Exa as a selectable search provider
Branch: main Changes Summary: Cargo.toml | 6 ++++++ LICENSE | 21 +++++++++++++++++++++ README.md | 51 +++++++++++++++++++++++++++++++++++++++++++++------ src/main.rs | 2 +- 4 files changed, 73 insertions(+), 7 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/Cargo.toml b/Cargo.toml index 27d356b..dbedfb0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,6 +2,12 @@ name = "sophon-cli" version = "0.1.0" edition = "2024" +description = "Provider-agnostic search CLI for Brave Search and Exa" +license = "MIT" +repository = "https://github.com/larock22/sophon" +readme = "README.md" +keywords = ["search", "cli", "brave", "exa"] +categories = ["command-line-utilities"] [dependencies] tokio = { version = "1", features = ["full"] } diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..098c48b --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Sophon Relay + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index a0604c8..660867f 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,63 @@  -A provider-agnostic Rust CLI that queries the Brave Search API and prints normalized text results. +A provider-agnostic Rust CLI that queries Brave Search or Exa and prints normalized text results. -## Quick start +## Install ```bash -# Set your API key -echo "BRAVE_API_KEY=your_key_here" > .env +cargo install sophon-cli +``` -# Run a search +## Quick start + +```bash +# Run locally from the repo cargo run -- "rust programming" -# About +# Choose a provider explicitly +cargo run -- "rust programming" --provider brave +cargo run -- "rust programming" --provider exa + +# Show package info cargo run -- --about # Run all checks just check ``` +## Configuration + +Set the API key for the provider you want to use: + +```bash +# Brave +echo "BRAVE_API_KEY=your_key_here" > .env + +# Exa +echo "EXA_API_KEY=your_key_here" > .env +``` + +You can also export the variables directly in your shell instead of using `.env`. + +## Example usage + +```bash +# Web search with Brave +sophon-cli "rust programming" --provider brave + +# News search with Brave +sophon-cli "open source ai" --provider brave --search-type news --limit 3 + +# Exa search +sophon-cli "vector database benchmarks" --provider exa --limit 5 +``` + +## Supported providers + +- `brave` for web, news, images, and video search +- `exa` for Exa search results mapped into the shared domain model + ## Docs See the [architecture docs](docs/architecture.md) for the typed input-to-output flow and layer boundaries. diff --git a/src/main.rs b/src/main.rs index 88f0577..6ac73aa 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,7 +29,7 @@ async fn main() { println!("across vast distances. This tiny CLI delegates its heavy lifting to"); println!("distant search APIs the same way."); println!(); - println!("Currently supports Brave Search (web, news, images, video)."); + println!("Currently supports Brave Search (web, news, images, video) and Exa."); return; }
Branch: main Changes Summary: Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Detailed Diffs (truncated to 200 lines): diff --git a/Cargo.toml b/Cargo.toml index dbedfb0..0b28afe 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,7 @@ version = "0.1.0" edition = "2024" description = "Provider-agnostic search CLI for Brave Search and Exa" license = "MIT" -repository = "https://github.com/larock22/sophon" +repository = "https://github.com/alchemiststudiosDOTai/sophon" readme = "README.md" keywords = ["search", "cli", "brave", "exa"] categories = ["command-line-utilities"]
- Updated `ExaContentsRequest` to include optional `text`, `highlights`, and `summary` fields. - Introduced `ExaHighlightsRequest` and `ExaSummaryRequest` structs for better request structuring. - Modified `ExaResult` to include a `highlights` vector and adjusted deserialization tests accordingly. - Updated the `ExaProvider` to utilize new request structures and handle highlights and summaries in responses. - Enhanced mapping functions to prioritize summaries over highlights and ensure proper snippet generation for CLI output.
Document Unreleased changes: Exa contents request (highlights/summary), snippet mapping without full-text fallback, and CLI news snippet output. Made-with: Cursor
feat: Enhance Exa provider with optional fields and improved response handling
Add pre-commit hooks, GitHub templates, CODEOWNERS, skill definitions, AGENTS.md validation workflow, .env.example, and label documentation. - .pre-commit-config.yaml: general hygiene + just check gate - .github/workflows/validate-agents.yml: CI that verifies AGENTS.md paths and canonical commands - .github/ISSUE_TEMPLATE/: bug report and feature request templates - .github/pull_request_template.md: structured PR checklist - .github/CODEOWNERS: ownership fallback - .github/labels.yml: priority, type, and area label definitions - .factory/skills/sophon-cli/SKILL.md: agent skill with project context - .env.example: documents required API keys - Remove obsolete PRD.md reference from AGENTS.md and CI validation Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add the bootstrap module skeleton and provider-neutral ProviderId used by the upcoming registry. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Define provider builders, registry lookup, stable availability ordering, and the provider-unavailable error used during service construction. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Register Brave and Exa providers from their typed environment configs, omitting unconfigured providers from the registry. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Map CliProvider into the provider-neutral ProviderId in main without making the bootstrap registry depend on CLI types. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Replace inline provider construction in main with ProviderRegistry::production_from_env and registry-based service construction. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Add an architecture test that documents bootstrap as the composition layer while forbidding CLI imports there. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Cover pure registry registration, service construction, stable availability ordering, and keep env-backed availability separate. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Run the planned formatter, test, clippy, docs, and umbrella gates; update the architecture and harness documentation for the new bootstrap layer. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
feat: add agent legibility infrastructure
…-interface [codex] Add provider registry composition layer
- Add provider registry implementation notes - Document new plan artifacts and interface designs - Update changed section with main.rs refactoring and harness updates
* feat: add structured logging with tracing Add tracing and tracing-subscriber for structured, environment-filtered logs across the CLI, application layer, transport, and provider adapters. Logs are written to stderr so stdout remains clean for CLI results. Key spans instrument startup, search orchestration, provider calls, and HTTP transport. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix: avoid logging auth headers --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* add docs metadata guard * fix docs metadata CI guard
Branch: main Changes Summary: CHANGELOG.md | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ac763b..69872ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,22 +21,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **Bootstrap**: Provider registry (`src/bootstrap/provider_registry.rs`) for compile-time provider registration and metadata discovery. -- **Docs Guard**: Markdown frontmatter validator and Cargo-managed pre-push hook for the canonical `just check` gate. +- Provider registry composition layer for built-in provider registration, provider metadata discovery, and `SearchService` construction. (#7) +- Environment-filtered structured tracing spans for startup, search orchestration, provider adapters, and HTTP transport. Logs are written to stderr so CLI result output stays clean. (#8) +- Markdown frontmatter validation and mdBook frontmatter stripping in the canonical `just check` gate. (#9) +- Cargo-managed pre-push hook that runs `just check`. (#9) +- Repository operating surfaces: `.env.example`, CODEOWNERS, issue and PR templates, label definitions, AGENTS validation CI, and a `sophon-cli` agent skill. (#6) ### Changed -- **Main**: Refactored to use provider registry for provider instantiation instead of direct constructor calls. -- **Architecture Tests**: Updated to allow `bootstrap` module imports from `main.rs`. -- **HARNESS.md**: Updated harness map to reflect current validation chain. -- **Artifacts**: `.artifacts/` is now ignored and no longer tracked in Git. -- **Exa**: Default `/search` `contents` now requests **highlights** (with `maxCharacters` and the user query) and a **query-scoped summary** object instead of full-page **`text`**, so the API is not asked for article bodies for normal CLI usage. -- **Exa**: Normalized `snippet` is derived as **summary** (trimmed, capped) if non-empty, else **joined highlights** (separator ` … `, capped); **`text` is never used** as a snippet fallback, even when present in the response. +- `main.rs` now selects providers through `ProviderId` and `ProviderRegistry` instead of directly constructing Brave and Exa clients. (#7) +- Production startup registers only providers with valid environment configuration, and provider-unavailable errors list configured providers. (#7) +- Architecture tests now include the `bootstrap` composition layer boundary. (#7) +- `HARNESS.md` and architecture docs now reflect the bootstrap layer, docs metadata guard, cargo-husky hook, and current validation chain. (#7, #9) +- Exa `/search` requests use highlights plus a query-scoped summary instead of requesting full-page `text` for normal CLI output. (#5) +- Exa snippet normalization now prefers trimmed summaries, then capped joined highlights; `text` is not used as a snippet fallback. (#5) +- CLI news output now prints `snippet` when present, matching web-result rendering. (#5) ### Fixed -- **Exa**: Web results no longer dump full extracted page markdown into the terminal when `summary` is missing. +- Exa web results no longer dump full extracted page markdown into the terminal when `summary` is missing. (#5) -### Added +### Removed + +- Tracked `.artifacts/` planning and execution files; future local artifact output is ignored by Git. (#9) +- Legacy pre-commit configuration in favor of the Cargo-managed pre-push hook. (#9) + +### Security -- **CLI**: News rows print **`snippet`** when present, matching web results and providers that populate `NewsResult.snippet`. +- Structured logging avoids recording provider authentication headers or API keys. (#8)
- Introduced a new `hygiene` target in the justfile to run checks for unused dependencies, code duplication, technical debt, and large files. - Updated the CI workflow to install `cargo-udeps` and run the new `just hygiene` command alongside existing checks. - Enhanced documentation in HARNESS.md to include hygiene checks and their usage.
feat: add hygiene checks to justfile and CI workflow Adds cargo-udeps, jscpd, tech-debt scanner, and large-file gate. Post-merge fixes applied: - Fix rg error handling in check_tech_debt.sh (distinguish exit 0/1/2+) - Sanitize tech-debt allowlist (skip blank lines and comments) - Run just check on stable before installing nightly toolchain
Add provider-agnostic batch response and failure wrappers with a unit proof for holding successes and provider errors. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add sequential fan-out orchestration over SearchProvider trait objects with tests proving provider order and failure capture. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add build_all_enabled with empty-registry errors and stable provider-order proof for fan-out registry construction. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add all to CLI provider parsing and prove it preserves the brave default; add temporary non-concrete main handling until fan-out wiring replaces it. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add CLI-only fan-out text rendering with summary, per-provider success sections, and failure lines. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Branch main between single-provider search and all-enabled fan-out, with nonzero exit when no providers succeed or none are configured. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Update README and mdBook docs for --provider all, environment-enabled providers, no-provider behavior, and fan-out architecture. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Branch: all-enabled-provider-fanout Changes Summary: src/app/fanout_search_service.rs | 7 +----- src/app/search_service.rs | 7 +----- src/domain/provider.rs | 2 +- src/providers/exa/mapper.rs | 50 +++++++++++++++++++--------------------- 4 files changed, 27 insertions(+), 39 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/src/app/fanout_search_service.rs b/src/app/fanout_search_service.rs index 16ae62e..62416fe 100644 --- a/src/app/fanout_search_service.rs +++ b/src/app/fanout_search_service.rs @@ -63,12 +63,7 @@ mod tests { fn capabilities(&self) -> ProviderCapabilities { ProviderCapabilities { web: true, - news: false, - images: false, - videos: false, - pagination: false, - safe_search: false, - time_range_filter: false, + ..ProviderCapabilities::default() } } diff --git a/src/app/search_service.rs b/src/app/search_service.rs index 91bd6d1..f96d783 100644 --- a/src/app/search_service.rs +++ b/src/app/search_service.rs @@ -43,12 +43,7 @@ mod tests { fn capabilities(&self) -> ProviderCapabilities { ProviderCapabilities { web: true, - news: false, - images: false, - videos: false, - pagination: false, - safe_search: false, - time_range_filter: false, + ..ProviderCapabilities::default() } } diff --git a/src/domain/provider.rs b/src/domain/provider.rs index 88dab2a..f6009d6 100644 --- a/src/domain/provider.rs +++ b/src/domain/provider.rs @@ -3,7 +3,7 @@ use crate::domain::query::SearchQuery; use crate::domain::result::SearchResponse; use async_trait::async_trait; -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Default)] #[allow(dead_code)] pub struct ProviderCapabilities { pub web: bool, diff --git a/src/providers/exa/mapper.rs b/src/providers/exa/mapper.rs index f0c470d..64e53d4 100644 --- a/src/providers/exa/mapper.rs +++ b/src/providers/exa/mapper.rs @@ -7,28 +7,32 @@ const SNIPPET_DISPLAY_MAX_CHARS: usize = 500; const HIGHLIGHT_JOIN: &str = " … "; pub fn map_web_response(query: &str, dto: ExaSearchResponse) -> SearchResponse { - SearchResponse { - query: query.to_string(), - provider: "exa".to_string(), - total_estimated: None, - next_page: None, - results: dto - .results - .into_iter() - .map(|result| { - let snippet = preferred_snippet(&result); - SearchResult::Web(WebResult { - title: result.title.unwrap_or_default(), - url: result.url.unwrap_or_default(), - snippet, - display_url: None, - }) - }) - .collect(), - } + map_response(query, dto, |result, snippet| { + SearchResult::Web(WebResult { + title: result.title.unwrap_or_default(), + url: result.url.unwrap_or_default(), + snippet, + display_url: None, + }) + }) } pub fn map_news_response(query: &str, dto: ExaSearchResponse) -> SearchResponse { + map_response(query, dto, |result, snippet| { + SearchResult::News(NewsResult { + title: result.title.unwrap_or_default(), + url: result.url.unwrap_or_default(), + snippet, + source: result.author, + published_at: result.published_date, + }) + }) +} + +fn map_response<F>(query: &str, dto: ExaSearchResponse, map_result: F) -> SearchResponse +where + F: Fn(ExaResult, Option<String>) -> SearchResult, +{ SearchResponse { query: query.to_string(), provider: "exa".to_string(), @@ -39,13 +43,7 @@ pub fn map_news_response(query: &str, dto: ExaSearchResponse) -> SearchResponse .into_iter() .map(|result| { let snippet = preferred_snippet(&result); - SearchResult::News(NewsResult { - title: result.title.unwrap_or_default(), - url: result.url.unwrap_or_default(), - snippet, - source: result.author, - published_at: result.published_date, - }) + map_result(result, snippet) }) .collect(), }
…r-fanout Add all-enabled provider fan-out
Branch: main Changes Summary: CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) Detailed Diffs (truncated to 200 lines): diff --git a/CHANGELOG.md b/CHANGELOG.md index 69872ac..070741a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `--provider all` fan-out mode that queries every environment-enabled provider and renders per-provider successes and failures. (#11) - Provider registry composition layer for built-in provider registration, provider metadata discovery, and `SearchService` construction. (#7) - Environment-filtered structured tracing spans for startup, search orchestration, provider adapters, and HTTP transport. Logs are written to stderr so CLI result output stays clean. (#8) - Markdown frontmatter validation and mdBook frontmatter stripping in the canonical `just check` gate. (#9)
Adds integration coverage for app services, provider registry behavior, and CLI error/output paths. Hardens CI hygiene dependencies and deterministic CLI test environment handling.
Branch: main Changes Summary: .cargo-husky/hooks/pre-commit | 11 ++ sophon-cli-architecture.html | 304 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 315 insertions(+) Detailed Diffs (truncated to 200 lines): diff --git a/.cargo-husky/hooks/pre-commit b/.cargo-husky/hooks/pre-commit new file mode 100755 index 0000000..9d98b80 --- /dev/null +++ b/.cargo-husky/hooks/pre-commit @@ -0,0 +1,11 @@ +#!/bin/sh +# +# Pre-commit hook installed by cargo-husky. +# Runs fast quality checks before every commit. +set -e + +echo "[pre-commit] Running cargo fmt --check" +cargo fmt --check + +echo "[pre-commit] Running cargo clippy" +cargo clippy -- -D warnings -W clippy::complexity -W clippy::cognitive_complexity diff --git a/sophon-cli-architecture.html b/sophon-cli-architecture.html new file mode 100644 index 0000000..686eec3 --- /dev/null +++ b/sophon-cli-architecture.html @@ -0,0 +1,304 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>sophon-cli Architecture Diagram</title> + <link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet"> + <style> + * { margin: 0; padding: 0; box-sizing: border-box; } + body { + font-family: 'JetBrains Mono', monospace; + background: #020617; + min-height: 100vh; + padding: 2rem; + color: white; + } + .container { max-width: 1200px; margin: 0 auto; } + .header { margin-bottom: 2rem; } + .header-row { display: flex; align-items: center; gap: 1rem; margin-bottom: 0.5rem; } + .pulse-dot { + width: 12px; height: 12px; background: #22d3ee; border-radius: 50%; + animation: pulse 2s infinite; + } + @Keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.5; } } + h1 { font-size: 1.5rem; font-weight: 700; letter-spacing: -0.025em; } + .subtitle { color: #94a3b8; font-size: 0.875rem; margin-left: 1.75rem; } + .diagram-container { + background: rgba(15, 23, 42, 0.5); border-radius: 1rem; border: 1px solid #1e293b; + padding: 1.5rem; overflow-x: auto; + } + svg { width: 100%; min-width: 900px; display: block; } + .cards { + display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); + gap: 1rem; margin-top: 2rem; + } + .card { + background: rgba(15, 23, 42, 0.5); border-radius: 0.75rem; + border: 1px solid #1e293b; padding: 1.25rem; + } + .card-header { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.75rem; } + .card-dot { width: 8px; height: 8px; border-radius: 50%; } + .card-dot.cyan { background: #22d3ee; } + .card-dot.emerald { background: #34d399; } + .card-dot.amber { background: #fbbf24; } + .card h3 { font-size: 0.875rem; font-weight: 600; } + .card ul { list-style: none; color: #94a3b8; font-size: 0.75rem; } + .card li { margin-bottom: 0.375rem; } + .footer { text-align: center; margin-top: 1.5rem; color: #475569; font-size: 0.75rem; } + </style> +</head> +<body> + <div class="container"> + <div class="header"> + <div class="header-row"> + <div class="pulse-dot"></div> + <h1>sophon-cli Architecture</h1> + </div> + <p class="subtitle">Module map and data flow from main.rs through the provider-agnostic search pipeline</p> + </div> + + <div class="diagram-container"> + <svg viewBox="0 0 1000 820"> + <defs> + <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto"> + <polygon points="0 0, 10 3.5, 0 7" fill="#64748b" /> + </marker> + <pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse"> + <path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1e293b" stroke-width="0.5"/> + </pattern> + </defs> + + <!-- Background Grid --> + <rect width="100%" height="100%" fill="url(#grid)" /> + + <!-- ==================== ARROWS ==================== --> + <!-- Terminal -> CliArgs --> + <line x1="160" y1="135" x2="216" y2="135" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="188" y="129" fill="#94a3b8" font-size="9" text-anchor="middle">args</text> + + <!-- CliArgs -> main.rs --> + <line x1="360" y1="135" x2="416" y2="135" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="388" y="129" fill="#94a3b8" font-size="9" text-anchor="middle">parse</text> + + <!-- main.rs -> ProviderRegistry --> + <line x1="520" y1="165" x2="520" y2="208" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="555" y="190" fill="#94a3b8" font-size="8" text-anchor="middle">query + limit</text> + + <!-- ProviderRegistry -> SearchService --> + <line x1="480" y1="260" x2="420" y2="308" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="438" y="290" fill="#94a3b8" font-size="8" text-anchor="middle">build</text> + + <!-- ProviderRegistry -> FanoutSearchService --> + <line x1="560" y1="260" x2="630" y2="308" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="605" y="290" fill="#94a3b8" font-size="8" text-anchor="middle">build</text> + + <!-- SearchService -> Domain --> + <line x1="420" y1="360" x2="420" y2="408" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="445" y="390" fill="#94a3b8" font-size="8" text-anchor="middle">search()</text> + + <!-- FanoutSearchService -> Domain --> + <line x1="630" y1="360" x2="630" y2="408" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="665" y="390" fill="#94a3b8" font-size="8" text-anchor="middle">search_all()</text> + + <!-- Domain -> BraveProvider --> + <line x1="430" y1="475" x2="280" y2="518" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="340" y="500" fill="#94a3b8" font-size="8" text-anchor="middle">trait call</text> + + <!-- Domain -> ExaProvider --> + <line x1="610" y1="475" x2="760" y2="518" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="695" y="500" fill="#94a3b8" font-size="8" text-anchor="middle">trait call</text> + + <!-- BraveProvider -> ReqwestHttpClient --> + <line x1="280" y1="590" x2="460" y2="638" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="350" y="620" fill="#94a3b8" font-size="8" text-anchor="middle">HTTP GET</text> + + <!-- ExaProvider -> ReqwestHttpClient --> + <line x1="760" y1="590" x2="580" y2="638" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="680" y="620" fill="#94a3b8" font-size="8" text-anchor="middle">HTTP POST</text> + + <!-- ReqwestHttpClient -> Brave API --> + <line x1="460" y1="690" x2="320" y2="738" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="380" y="720" fill="#94a3b8" font-size="8" text-anchor="middle">JSON</text> + + <!-- ReqwestHttpClient -> Exa API --> + <line x1="580" y1="690" x2="760" y2="738" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="680" y="720" fill="#94a3b8" font-size="8" text-anchor="middle">JSON</text> + + <!-- main.rs -> output.rs --> + <line x1="620" y1="125" x2="676" y2="125" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="648" y="119" fill="#94a3b8" font-size="8" text-anchor="middle">render</text> + + <!-- main.rs -> tracing --> + <line x1="620" y1="140" x2="848" y2="122" stroke="#fb7185" stroke-width="1.5" stroke-dasharray="4,4" marker-end="url(#arrowhead)"/> + <text x="740" y="125" fill="#fb7185" font-size="8" text-anchor="middle">init</text> + + <!-- main.rs -> dotenvy --> + <line x1="620" y1="155" x2="848" y2="182" stroke="#94a3b8" stroke-width="1.5" stroke-dasharray="4,4" marker-end="url(#arrowhead)"/> + <text x="740" y="175" fill="#94a3b8" font-size="8" text-anchor="middle">load</text> + + <!-- ==================== COMPONENTS ==================== --> + <!-- Terminal --> + <rect x="40" y="110" width="120" height="50" rx="6" fill="#0f172a"/> + <rect x="40" y="110" width="120" height="50" rx="6" fill="rgba(30, 41, 59, 0.5)" stroke="#94a3b8" stroke-width="1.5"/> + <text x="100" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">Terminal</text> + <text x="100" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">stdin / stdout / stderr</text> + + <!-- CliArgs --> + <rect x="220" y="110" width="140" height="50" rx="6" fill="#0f172a"/> + <rect x="220" y="110" width="140" height="50" rx="6" fill="rgba(8, 51, 68, 0.4)" stroke="#22d3ee" stroke-width="1.5"/> + <text x="290" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">CliArgs</text> + <text x="290" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">clap parser</text> + + <!-- main.rs --> + <rect x="420" y="95" width="200" height="70" rx="6" fill="#0f172a"/> + <rect x="420" y="95" width="200" height="70" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="520" y="122" fill="white" font-size="12" font-weight="600" text-anchor="middle">main.rs</text> + <text x="520" y="142" fill="#94a3b8" font-size="9" text-anchor="middle">entrypoint · tokio::main</text> + <text x="520" y="155" fill="#94a3b8" font-size="8" text-anchor="middle">run_single_provider</text> + + <!-- output.rs --> + <rect x="680" y="110" width="140" height="50" rx="6" fill="#0f172a"/> + <rect x="680" y="110" width="140" height="50" rx="6" fill="rgba(8, 51, 68, 0.4)" stroke="#22d3ee" stroke-width="1.5"/> + <text x="750" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">output.rs</text> + <text x="750" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">render · render_fanout</text> + + <!-- ProviderRegistry --> + <rect x="440" y="210" width="160" height="50" rx="6" fill="#0f172a"/> + <rect x="440" y="210" width="160" height="50" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="520" y="235" fill="white" font-size="12" font-weight="600" text-anchor="middle">ProviderRegistry</text> + <text x="520" y="251" fill="#94a3b8" font-size="9" text-anchor="middle">bootstrap from env</text> + + <!-- SearchService --> + <rect x="340" y="310" width="160" height="50" rx="6" fill="#0f172a"/> + <rect x="340" y="310" width="160" height="50" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="420" y="335" fill="white" font-size="12" font-weight="600" text-anchor="middle">SearchService</text> + <text x="420" y="351" fill="#94a3b8" font-size="9" text-anchor="middle">single provider</text> +
Add cli::request::build_search_query with focused field mapping coverage and expose it from the CLI module. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Move CLI runtime orchestration into cli::runner with parsed-args and environment entrypoints, provider selection, rendering, and return-code based errors. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Delegate process execution to cli::runner::run_from_env and remove the binary-private single provider helper. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Add the explicit entrypoint-to-CLI boundary test and reuse it in the ideal dependency direction contract. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Refresh architecture docs and the current dependency map for the CLI runner boundary, and add import organization guidance. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Remove stale architecture documentation and update the canonical dependency-architecture-map.html to reflect the post-refactor module structure. Deleted docs (content was outdated or duplicated by the map itself): - docs/dependency-direction.html + .md — standalone dependency visual - docs/ideal-dependency-architecture-map.html + .md — ideal target map - docs/import-organization.md — import rules now encoded in tests Updated docs/dependency-architecture-map.html: - Rewrote SVG to show cli::runner, cli::request, and cli::output - Added bootstrap::provider_registry as the composition root - Added app::FanoutSearchService alongside app::SearchService - Added exa::client/config/dto and exa::mapper adapter nodes - Updated arrow annotations and accuracy footer for post-refactor state Updated tests/architecture_test.rs: - Renamed test_t005_import_organization_docs_contract to test_t005_current_dependency_map_reflects_refactor - Removed import-organization.md existence and content checks - Kept dependency-architecture-map.html assertions (cli::runner present, single_provider_search absent) Validation: just check passes (fmt, clippy, 59 tests, mdBook build, frontmatter check)
Add rusqlite (bundled) with SearchDbWriter in cli::db for the issue #14 schema. Runner persists after successful search; optional --scrape fetches result URLs via HTTP and stores rows in scraped_sites/pages. For --provider all, each successful provider gets its own search_runs row; scrape telemetry attaches to the first provider run. Domain gains Clone on batch/failure types for async handoff without rusqlite in domain. Co-authored-by: tuna <tunahorse@users.noreply.github.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Remove references to the ideal dependency architecture map and import organization visual from the SUMMARY.md. Simplify the current dependency architecture map documentation by removing comparison notes and footer references to the ideal map. This streamlines the documentation to focus solely on the current architecture.
Update the current dependency architecture map to enhance clarity and accuracy. Removed outdated references and streamlined content to better reflect the current module structure. Adjusted visual elements and annotations for improved understanding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implements issue #14: optional
--db <path>writes search results to SQLite (bundledrusqlite, CLI-onlySearchDbWriterinsrc/cli/db.rs). Optional--scrape(requires--db) fetches up to--scrape-limitresult URLs withreqwestand stores HTTP response bodies inscraped_sites/scraped_pages.For
--provider all, each successful provider gets its ownsearch_runsrow; scrape rows attach to the first provider run so telemetry is shared.SearchBatchResponseandProviderSearchFailureareClonefor moving batch data into blocking DB tasks without pulling SQLite into the domain layer.Testing Done
just checkpasses locally (fmt, clippy, tests, mdbook build)args--db/--scrape,db::persist_response)cargo test)Type of Change
Relevant Context
Closes #14.
--scrapeuses simple HTTP GETs on result URLs because the codebase did not yet includesrc/cli/scrape.rsfrom the issue sketch; behavior matches the acceptance criteria (populate scrape tables when combined with--db).