[codex] Fix scrape content extraction - #13
Open
tunahorse wants to merge 51 commits into
Open
Conversation
Fix HARNESS canonical gate chain
Add Exa as a selectable search provider
Branch: main Changes Summary: Cargo.toml | 6 ++++++ LICENSE | 21 +++++++++++++++++++++ README.md | 51 +++++++++++++++++++++++++++++++++++++++++++++------ src/main.rs | 2 +- 4 files changed, 73 insertions(+), 7 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/Cargo.toml b/Cargo.toml index 27d356b..dbedfb0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -2,6 +2,12 @@ name = "sophon-cli" version = "0.1.0" edition = "2024" +description = "Provider-agnostic search CLI for Brave Search and Exa" +license = "MIT" +repository = "https://github.com/larock22/sophon" +readme = "README.md" +keywords = ["search", "cli", "brave", "exa"] +categories = ["command-line-utilities"] [dependencies] tokio = { version = "1", features = ["full"] } diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..098c48b --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Sophon Relay + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index a0604c8..660867f 100644 --- a/README.md +++ b/README.md @@ -4,24 +4,63 @@  -A provider-agnostic Rust CLI that queries the Brave Search API and prints normalized text results. +A provider-agnostic Rust CLI that queries Brave Search or Exa and prints normalized text results. -## Quick start +## Install ```bash -# Set your API key -echo "BRAVE_API_KEY=your_key_here" > .env +cargo install sophon-cli +``` -# Run a search +## Quick start + +```bash +# Run locally from the repo cargo run -- "rust programming" -# About +# Choose a provider explicitly +cargo run -- "rust programming" --provider brave +cargo run -- "rust programming" --provider exa + +# Show package info cargo run -- --about # Run all checks just check ``` +## Configuration + +Set the API key for the provider you want to use: + +```bash +# Brave +echo "BRAVE_API_KEY=your_key_here" > .env + +# Exa +echo "EXA_API_KEY=your_key_here" > .env +``` + +You can also export the variables directly in your shell instead of using `.env`. + +## Example usage + +```bash +# Web search with Brave +sophon-cli "rust programming" --provider brave + +# News search with Brave +sophon-cli "open source ai" --provider brave --search-type news --limit 3 + +# Exa search +sophon-cli "vector database benchmarks" --provider exa --limit 5 +``` + +## Supported providers + +- `brave` for web, news, images, and video search +- `exa` for Exa search results mapped into the shared domain model + ## Docs See the [architecture docs](docs/architecture.md) for the typed input-to-output flow and layer boundaries. diff --git a/src/main.rs b/src/main.rs index 88f0577..6ac73aa 100644 --- a/src/main.rs +++ b/src/main.rs @@ -29,7 +29,7 @@ async fn main() { println!("across vast distances. This tiny CLI delegates its heavy lifting to"); println!("distant search APIs the same way."); println!(); - println!("Currently supports Brave Search (web, news, images, video)."); + println!("Currently supports Brave Search (web, news, images, video) and Exa."); return; }
Branch: main Changes Summary: Cargo.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Detailed Diffs (truncated to 200 lines): diff --git a/Cargo.toml b/Cargo.toml index dbedfb0..0b28afe 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -4,7 +4,7 @@ version = "0.1.0" edition = "2024" description = "Provider-agnostic search CLI for Brave Search and Exa" license = "MIT" -repository = "https://github.com/larock22/sophon" +repository = "https://github.com/alchemiststudiosDOTai/sophon" readme = "README.md" keywords = ["search", "cli", "brave", "exa"] categories = ["command-line-utilities"]
- Updated `ExaContentsRequest` to include optional `text`, `highlights`, and `summary` fields. - Introduced `ExaHighlightsRequest` and `ExaSummaryRequest` structs for better request structuring. - Modified `ExaResult` to include a `highlights` vector and adjusted deserialization tests accordingly. - Updated the `ExaProvider` to utilize new request structures and handle highlights and summaries in responses. - Enhanced mapping functions to prioritize summaries over highlights and ensure proper snippet generation for CLI output.
Document Unreleased changes: Exa contents request (highlights/summary), snippet mapping without full-text fallback, and CLI news snippet output. Made-with: Cursor
feat: Enhance Exa provider with optional fields and improved response handling
Add pre-commit hooks, GitHub templates, CODEOWNERS, skill definitions, AGENTS.md validation workflow, .env.example, and label documentation. - .pre-commit-config.yaml: general hygiene + just check gate - .github/workflows/validate-agents.yml: CI that verifies AGENTS.md paths and canonical commands - .github/ISSUE_TEMPLATE/: bug report and feature request templates - .github/pull_request_template.md: structured PR checklist - .github/CODEOWNERS: ownership fallback - .github/labels.yml: priority, type, and area label definitions - .factory/skills/sophon-cli/SKILL.md: agent skill with project context - .env.example: documents required API keys - Remove obsolete PRD.md reference from AGENTS.md and CI validation Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add the bootstrap module skeleton and provider-neutral ProviderId used by the upcoming registry. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Define provider builders, registry lookup, stable availability ordering, and the provider-unavailable error used during service construction. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Register Brave and Exa providers from their typed environment configs, omitting unconfigured providers from the registry. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Map CliProvider into the provider-neutral ProviderId in main without making the bootstrap registry depend on CLI types. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Replace inline provider construction in main with ProviderRegistry::production_from_env and registry-based service construction. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Add an architecture test that documents bootstrap as the composition layer while forbidding CLI imports there. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Cover pure registry registration, service construction, stable availability ordering, and keep env-backed availability separate. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
Run the planned formatter, test, clippy, docs, and umbrella gates; update the architecture and harness documentation for the new bootstrap layer. Refs: .artifacts/plan/2026-04-25_13-50-32_provider-registry/PLAN.md
feat: add agent legibility infrastructure
…-interface [codex] Add provider registry composition layer
- Add provider registry implementation notes - Document new plan artifacts and interface designs - Update changed section with main.rs refactoring and harness updates
* feat: add structured logging with tracing Add tracing and tracing-subscriber for structured, environment-filtered logs across the CLI, application layer, transport, and provider adapters. Logs are written to stderr so stdout remains clean for CLI results. Key spans instrument startup, search orchestration, provider calls, and HTTP transport. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> * fix: avoid logging auth headers --------- Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
* add docs metadata guard * fix docs metadata CI guard
Branch: main Changes Summary: CHANGELOG.md | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ac763b..69872ac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,22 +21,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **Bootstrap**: Provider registry (`src/bootstrap/provider_registry.rs`) for compile-time provider registration and metadata discovery. -- **Docs Guard**: Markdown frontmatter validator and Cargo-managed pre-push hook for the canonical `just check` gate. +- Provider registry composition layer for built-in provider registration, provider metadata discovery, and `SearchService` construction. (#7) +- Environment-filtered structured tracing spans for startup, search orchestration, provider adapters, and HTTP transport. Logs are written to stderr so CLI result output stays clean. (#8) +- Markdown frontmatter validation and mdBook frontmatter stripping in the canonical `just check` gate. (#9) +- Cargo-managed pre-push hook that runs `just check`. (#9) +- Repository operating surfaces: `.env.example`, CODEOWNERS, issue and PR templates, label definitions, AGENTS validation CI, and a `sophon-cli` agent skill. (#6) ### Changed -- **Main**: Refactored to use provider registry for provider instantiation instead of direct constructor calls. -- **Architecture Tests**: Updated to allow `bootstrap` module imports from `main.rs`. -- **HARNESS.md**: Updated harness map to reflect current validation chain. -- **Artifacts**: `.artifacts/` is now ignored and no longer tracked in Git. -- **Exa**: Default `/search` `contents` now requests **highlights** (with `maxCharacters` and the user query) and a **query-scoped summary** object instead of full-page **`text`**, so the API is not asked for article bodies for normal CLI usage. -- **Exa**: Normalized `snippet` is derived as **summary** (trimmed, capped) if non-empty, else **joined highlights** (separator ` … `, capped); **`text` is never used** as a snippet fallback, even when present in the response. +- `main.rs` now selects providers through `ProviderId` and `ProviderRegistry` instead of directly constructing Brave and Exa clients. (#7) +- Production startup registers only providers with valid environment configuration, and provider-unavailable errors list configured providers. (#7) +- Architecture tests now include the `bootstrap` composition layer boundary. (#7) +- `HARNESS.md` and architecture docs now reflect the bootstrap layer, docs metadata guard, cargo-husky hook, and current validation chain. (#7, #9) +- Exa `/search` requests use highlights plus a query-scoped summary instead of requesting full-page `text` for normal CLI output. (#5) +- Exa snippet normalization now prefers trimmed summaries, then capped joined highlights; `text` is not used as a snippet fallback. (#5) +- CLI news output now prints `snippet` when present, matching web-result rendering. (#5) ### Fixed -- **Exa**: Web results no longer dump full extracted page markdown into the terminal when `summary` is missing. +- Exa web results no longer dump full extracted page markdown into the terminal when `summary` is missing. (#5) -### Added +### Removed + +- Tracked `.artifacts/` planning and execution files; future local artifact output is ignored by Git. (#9) +- Legacy pre-commit configuration in favor of the Cargo-managed pre-push hook. (#9) + +### Security -- **CLI**: News rows print **`snippet`** when present, matching web results and providers that populate `NewsResult.snippet`. +- Structured logging avoids recording provider authentication headers or API keys. (#8)
- Introduced a new `hygiene` target in the justfile to run checks for unused dependencies, code duplication, technical debt, and large files. - Updated the CI workflow to install `cargo-udeps` and run the new `just hygiene` command alongside existing checks. - Enhanced documentation in HARNESS.md to include hygiene checks and their usage.
feat: add hygiene checks to justfile and CI workflow Adds cargo-udeps, jscpd, tech-debt scanner, and large-file gate. Post-merge fixes applied: - Fix rg error handling in check_tech_debt.sh (distinguish exit 0/1/2+) - Sanitize tech-debt allowlist (skip blank lines and comments) - Run just check on stable before installing nightly toolchain
Add provider-agnostic batch response and failure wrappers with a unit proof for holding successes and provider errors. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add sequential fan-out orchestration over SearchProvider trait objects with tests proving provider order and failure capture. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add build_all_enabled with empty-registry errors and stable provider-order proof for fan-out registry construction. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add all to CLI provider parsing and prove it preserves the brave default; add temporary non-concrete main handling until fan-out wiring replaces it. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Add CLI-only fan-out text rendering with summary, per-provider success sections, and failure lines. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Branch main between single-provider search and all-enabled fan-out, with nonzero exit when no providers succeed or none are configured. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Update README and mdBook docs for --provider all, environment-enabled providers, no-provider behavior, and fan-out architecture. Refs: plan/2026-04-26_15-04-07_all-enabled-provider-fanout/PLAN.md
Branch: all-enabled-provider-fanout Changes Summary: src/app/fanout_search_service.rs | 7 +----- src/app/search_service.rs | 7 +----- src/domain/provider.rs | 2 +- src/providers/exa/mapper.rs | 50 +++++++++++++++++++--------------------- 4 files changed, 27 insertions(+), 39 deletions(-) Detailed Diffs (truncated to 200 lines): diff --git a/src/app/fanout_search_service.rs b/src/app/fanout_search_service.rs index 16ae62e..62416fe 100644 --- a/src/app/fanout_search_service.rs +++ b/src/app/fanout_search_service.rs @@ -63,12 +63,7 @@ mod tests { fn capabilities(&self) -> ProviderCapabilities { ProviderCapabilities { web: true, - news: false, - images: false, - videos: false, - pagination: false, - safe_search: false, - time_range_filter: false, + ..ProviderCapabilities::default() } } diff --git a/src/app/search_service.rs b/src/app/search_service.rs index 91bd6d1..f96d783 100644 --- a/src/app/search_service.rs +++ b/src/app/search_service.rs @@ -43,12 +43,7 @@ mod tests { fn capabilities(&self) -> ProviderCapabilities { ProviderCapabilities { web: true, - news: false, - images: false, - videos: false, - pagination: false, - safe_search: false, - time_range_filter: false, + ..ProviderCapabilities::default() } } diff --git a/src/domain/provider.rs b/src/domain/provider.rs index 88dab2a..f6009d6 100644 --- a/src/domain/provider.rs +++ b/src/domain/provider.rs @@ -3,7 +3,7 @@ use crate::domain::query::SearchQuery; use crate::domain::result::SearchResponse; use async_trait::async_trait; -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Default)] #[allow(dead_code)] pub struct ProviderCapabilities { pub web: bool, diff --git a/src/providers/exa/mapper.rs b/src/providers/exa/mapper.rs index f0c470d..64e53d4 100644 --- a/src/providers/exa/mapper.rs +++ b/src/providers/exa/mapper.rs @@ -7,28 +7,32 @@ const SNIPPET_DISPLAY_MAX_CHARS: usize = 500; const HIGHLIGHT_JOIN: &str = " … "; pub fn map_web_response(query: &str, dto: ExaSearchResponse) -> SearchResponse { - SearchResponse { - query: query.to_string(), - provider: "exa".to_string(), - total_estimated: None, - next_page: None, - results: dto - .results - .into_iter() - .map(|result| { - let snippet = preferred_snippet(&result); - SearchResult::Web(WebResult { - title: result.title.unwrap_or_default(), - url: result.url.unwrap_or_default(), - snippet, - display_url: None, - }) - }) - .collect(), - } + map_response(query, dto, |result, snippet| { + SearchResult::Web(WebResult { + title: result.title.unwrap_or_default(), + url: result.url.unwrap_or_default(), + snippet, + display_url: None, + }) + }) } pub fn map_news_response(query: &str, dto: ExaSearchResponse) -> SearchResponse { + map_response(query, dto, |result, snippet| { + SearchResult::News(NewsResult { + title: result.title.unwrap_or_default(), + url: result.url.unwrap_or_default(), + snippet, + source: result.author, + published_at: result.published_date, + }) + }) +} + +fn map_response<F>(query: &str, dto: ExaSearchResponse, map_result: F) -> SearchResponse +where + F: Fn(ExaResult, Option<String>) -> SearchResult, +{ SearchResponse { query: query.to_string(), provider: "exa".to_string(), @@ -39,13 +43,7 @@ pub fn map_news_response(query: &str, dto: ExaSearchResponse) -> SearchResponse .into_iter() .map(|result| { let snippet = preferred_snippet(&result); - SearchResult::News(NewsResult { - title: result.title.unwrap_or_default(), - url: result.url.unwrap_or_default(), - snippet, - source: result.author, - published_at: result.published_date, - }) + map_result(result, snippet) }) .collect(), }
…r-fanout Add all-enabled provider fan-out
Branch: main Changes Summary: CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) Detailed Diffs (truncated to 200 lines): diff --git a/CHANGELOG.md b/CHANGELOG.md index 69872ac..070741a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `--provider all` fan-out mode that queries every environment-enabled provider and renders per-provider successes and failures. (#11) - Provider registry composition layer for built-in provider registration, provider metadata discovery, and `SearchService` construction. (#7) - Environment-filtered structured tracing spans for startup, search orchestration, provider adapters, and HTTP transport. Logs are written to stderr so CLI result output stays clean. (#8) - Markdown frontmatter validation and mdBook frontmatter stripping in the canonical `just check` gate. (#9)
Adds integration coverage for app services, provider registry behavior, and CLI error/output paths. Hardens CI hygiene dependencies and deterministic CLI test environment handling.
Branch: main Changes Summary: .cargo-husky/hooks/pre-commit | 11 ++ sophon-cli-architecture.html | 304 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 315 insertions(+) Detailed Diffs (truncated to 200 lines): diff --git a/.cargo-husky/hooks/pre-commit b/.cargo-husky/hooks/pre-commit new file mode 100755 index 0000000..9d98b80 --- /dev/null +++ b/.cargo-husky/hooks/pre-commit @@ -0,0 +1,11 @@ +#!/bin/sh +# +# Pre-commit hook installed by cargo-husky. +# Runs fast quality checks before every commit. +set -e + +echo "[pre-commit] Running cargo fmt --check" +cargo fmt --check + +echo "[pre-commit] Running cargo clippy" +cargo clippy -- -D warnings -W clippy::complexity -W clippy::cognitive_complexity diff --git a/sophon-cli-architecture.html b/sophon-cli-architecture.html new file mode 100644 index 0000000..686eec3 --- /dev/null +++ b/sophon-cli-architecture.html @@ -0,0 +1,304 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="UTF-8"> + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + <title>sophon-cli Architecture Diagram</title> + <link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;500;600;700&display=swap" rel="stylesheet"> + <style> + * { margin: 0; padding: 0; box-sizing: border-box; } + body { + font-family: 'JetBrains Mono', monospace; + background: #020617; + min-height: 100vh; + padding: 2rem; + color: white; + } + .container { max-width: 1200px; margin: 0 auto; } + .header { margin-bottom: 2rem; } + .header-row { display: flex; align-items: center; gap: 1rem; margin-bottom: 0.5rem; } + .pulse-dot { + width: 12px; height: 12px; background: #22d3ee; border-radius: 50%; + animation: pulse 2s infinite; + } + @Keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.5; } } + h1 { font-size: 1.5rem; font-weight: 700; letter-spacing: -0.025em; } + .subtitle { color: #94a3b8; font-size: 0.875rem; margin-left: 1.75rem; } + .diagram-container { + background: rgba(15, 23, 42, 0.5); border-radius: 1rem; border: 1px solid #1e293b; + padding: 1.5rem; overflow-x: auto; + } + svg { width: 100%; min-width: 900px; display: block; } + .cards { + display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); + gap: 1rem; margin-top: 2rem; + } + .card { + background: rgba(15, 23, 42, 0.5); border-radius: 0.75rem; + border: 1px solid #1e293b; padding: 1.25rem; + } + .card-header { display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.75rem; } + .card-dot { width: 8px; height: 8px; border-radius: 50%; } + .card-dot.cyan { background: #22d3ee; } + .card-dot.emerald { background: #34d399; } + .card-dot.amber { background: #fbbf24; } + .card h3 { font-size: 0.875rem; font-weight: 600; } + .card ul { list-style: none; color: #94a3b8; font-size: 0.75rem; } + .card li { margin-bottom: 0.375rem; } + .footer { text-align: center; margin-top: 1.5rem; color: #475569; font-size: 0.75rem; } + </style> +</head> +<body> + <div class="container"> + <div class="header"> + <div class="header-row"> + <div class="pulse-dot"></div> + <h1>sophon-cli Architecture</h1> + </div> + <p class="subtitle">Module map and data flow from main.rs through the provider-agnostic search pipeline</p> + </div> + + <div class="diagram-container"> + <svg viewBox="0 0 1000 820"> + <defs> + <marker id="arrowhead" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto"> + <polygon points="0 0, 10 3.5, 0 7" fill="#64748b" /> + </marker> + <pattern id="grid" width="40" height="40" patternUnits="userSpaceOnUse"> + <path d="M 40 0 L 0 0 0 40" fill="none" stroke="#1e293b" stroke-width="0.5"/> + </pattern> + </defs> + + <!-- Background Grid --> + <rect width="100%" height="100%" fill="url(#grid)" /> + + <!-- ==================== ARROWS ==================== --> + <!-- Terminal -> CliArgs --> + <line x1="160" y1="135" x2="216" y2="135" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="188" y="129" fill="#94a3b8" font-size="9" text-anchor="middle">args</text> + + <!-- CliArgs -> main.rs --> + <line x1="360" y1="135" x2="416" y2="135" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="388" y="129" fill="#94a3b8" font-size="9" text-anchor="middle">parse</text> + + <!-- main.rs -> ProviderRegistry --> + <line x1="520" y1="165" x2="520" y2="208" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="555" y="190" fill="#94a3b8" font-size="8" text-anchor="middle">query + limit</text> + + <!-- ProviderRegistry -> SearchService --> + <line x1="480" y1="260" x2="420" y2="308" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="438" y="290" fill="#94a3b8" font-size="8" text-anchor="middle">build</text> + + <!-- ProviderRegistry -> FanoutSearchService --> + <line x1="560" y1="260" x2="630" y2="308" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="605" y="290" fill="#94a3b8" font-size="8" text-anchor="middle">build</text> + + <!-- SearchService -> Domain --> + <line x1="420" y1="360" x2="420" y2="408" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="445" y="390" fill="#94a3b8" font-size="8" text-anchor="middle">search()</text> + + <!-- FanoutSearchService -> Domain --> + <line x1="630" y1="360" x2="630" y2="408" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="665" y="390" fill="#94a3b8" font-size="8" text-anchor="middle">search_all()</text> + + <!-- Domain -> BraveProvider --> + <line x1="430" y1="475" x2="280" y2="518" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="340" y="500" fill="#94a3b8" font-size="8" text-anchor="middle">trait call</text> + + <!-- Domain -> ExaProvider --> + <line x1="610" y1="475" x2="760" y2="518" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="695" y="500" fill="#94a3b8" font-size="8" text-anchor="middle">trait call</text> + + <!-- BraveProvider -> ReqwestHttpClient --> + <line x1="280" y1="590" x2="460" y2="638" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="350" y="620" fill="#94a3b8" font-size="8" text-anchor="middle">HTTP GET</text> + + <!-- ExaProvider -> ReqwestHttpClient --> + <line x1="760" y1="590" x2="580" y2="638" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="680" y="620" fill="#94a3b8" font-size="8" text-anchor="middle">HTTP POST</text> + + <!-- ReqwestHttpClient -> Brave API --> + <line x1="460" y1="690" x2="320" y2="738" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="380" y="720" fill="#94a3b8" font-size="8" text-anchor="middle">JSON</text> + + <!-- ReqwestHttpClient -> Exa API --> + <line x1="580" y1="690" x2="760" y2="738" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="680" y="720" fill="#94a3b8" font-size="8" text-anchor="middle">JSON</text> + + <!-- main.rs -> output.rs --> + <line x1="620" y1="125" x2="676" y2="125" stroke="#94a3b8" stroke-width="1.5" marker-end="url(#arrowhead)"/> + <text x="648" y="119" fill="#94a3b8" font-size="8" text-anchor="middle">render</text> + + <!-- main.rs -> tracing --> + <line x1="620" y1="140" x2="848" y2="122" stroke="#fb7185" stroke-width="1.5" stroke-dasharray="4,4" marker-end="url(#arrowhead)"/> + <text x="740" y="125" fill="#fb7185" font-size="8" text-anchor="middle">init</text> + + <!-- main.rs -> dotenvy --> + <line x1="620" y1="155" x2="848" y2="182" stroke="#94a3b8" stroke-width="1.5" stroke-dasharray="4,4" marker-end="url(#arrowhead)"/> + <text x="740" y="175" fill="#94a3b8" font-size="8" text-anchor="middle">load</text> + + <!-- ==================== COMPONENTS ==================== --> + <!-- Terminal --> + <rect x="40" y="110" width="120" height="50" rx="6" fill="#0f172a"/> + <rect x="40" y="110" width="120" height="50" rx="6" fill="rgba(30, 41, 59, 0.5)" stroke="#94a3b8" stroke-width="1.5"/> + <text x="100" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">Terminal</text> + <text x="100" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">stdin / stdout / stderr</text> + + <!-- CliArgs --> + <rect x="220" y="110" width="140" height="50" rx="6" fill="#0f172a"/> + <rect x="220" y="110" width="140" height="50" rx="6" fill="rgba(8, 51, 68, 0.4)" stroke="#22d3ee" stroke-width="1.5"/> + <text x="290" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">CliArgs</text> + <text x="290" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">clap parser</text> + + <!-- main.rs --> + <rect x="420" y="95" width="200" height="70" rx="6" fill="#0f172a"/> + <rect x="420" y="95" width="200" height="70" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="520" y="122" fill="white" font-size="12" font-weight="600" text-anchor="middle">main.rs</text> + <text x="520" y="142" fill="#94a3b8" font-size="9" text-anchor="middle">entrypoint · tokio::main</text> + <text x="520" y="155" fill="#94a3b8" font-size="8" text-anchor="middle">run_single_provider</text> + + <!-- output.rs --> + <rect x="680" y="110" width="140" height="50" rx="6" fill="#0f172a"/> + <rect x="680" y="110" width="140" height="50" rx="6" fill="rgba(8, 51, 68, 0.4)" stroke="#22d3ee" stroke-width="1.5"/> + <text x="750" y="132" fill="white" font-size="12" font-weight="600" text-anchor="middle">output.rs</text> + <text x="750" y="148" fill="#94a3b8" font-size="9" text-anchor="middle">render · render_fanout</text> + + <!-- ProviderRegistry --> + <rect x="440" y="210" width="160" height="50" rx="6" fill="#0f172a"/> + <rect x="440" y="210" width="160" height="50" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="520" y="235" fill="white" font-size="12" font-weight="600" text-anchor="middle">ProviderRegistry</text> + <text x="520" y="251" fill="#94a3b8" font-size="9" text-anchor="middle">bootstrap from env</text> + + <!-- SearchService --> + <rect x="340" y="310" width="160" height="50" rx="6" fill="#0f172a"/> + <rect x="340" y="310" width="160" height="50" rx="6" fill="rgba(6, 78, 59, 0.4)" stroke="#34d399" stroke-width="1.5"/> + <text x="420" y="335" fill="white" font-size="12" font-weight="600" text-anchor="middle">SearchService</text> + <text x="420" y="351" fill="#94a3b8" font-size="9" text-anchor="middle">single provider</text> +
Add cli::request::build_search_query with focused field mapping coverage and expose it from the CLI module. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Move CLI runtime orchestration into cli::runner with parsed-args and environment entrypoints, provider selection, rendering, and return-code based errors. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Delegate process execution to cli::runner::run_from_env and remove the binary-private single provider helper. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Add the explicit entrypoint-to-CLI boundary test and reuse it in the ideal dependency direction contract. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Refresh architecture docs and the current dependency map for the CLI runner boundary, and add import organization guidance. Refs: plan/2026-04-28_16-23-25_ideal-dependency-organization/PLAN.md
Remove stale architecture documentation and update the canonical dependency-architecture-map.html to reflect the post-refactor module structure. Deleted docs (content was outdated or duplicated by the map itself): - docs/dependency-direction.html + .md — standalone dependency visual - docs/ideal-dependency-architecture-map.html + .md — ideal target map - docs/import-organization.md — import rules now encoded in tests Updated docs/dependency-architecture-map.html: - Rewrote SVG to show cli::runner, cli::request, and cli::output - Added bootstrap::provider_registry as the composition root - Added app::FanoutSearchService alongside app::SearchService - Added exa::client/config/dto and exa::mapper adapter nodes - Updated arrow annotations and accuracy footer for post-refactor state Updated tests/architecture_test.rs: - Renamed test_t005_import_organization_docs_contract to test_t005_current_dependency_map_reflects_refactor - Removed import-organization.md existence and content checks - Kept dependency-architecture-map.html assertions (cli::runner present, single_provider_search absent) Validation: just check passes (fmt, clippy, 59 tests, mdBook build, frontmatter check)
📝 WalkthroughWalkthroughThis PR adds web scraping functionality to the search CLI application. It introduces a new Changes
Sequence DiagramsequenceDiagram
participant User as User/CLI
participant Runner as Search Runner
participant SearchAPI as Search API
participant Scraper as Scraper Module
participant Spider as Spider Crate
participant Output as Output Renderer
User->>Runner: Invoke CLI with --scrape
Runner->>SearchAPI: Execute search query
SearchAPI-->>Runner: Return SearchResponse/BatchResponse
Runner->>Scraper: Extract & deduplicate URLs
Scraper-->>Runner: Return seed URLs (ordered, unique)
Runner->>Scraper: Call scrape_seed_urls with limits/timeout
loop Per seed URL
Scraper->>Spider: Crawl seed with page limit
Spider-->>Scraper: Return pages (URL, status, content)
Scraper->>Scraper: Truncate content to 2K chars
Scraper->>Scraper: Capture visited URLs
end
Scraper-->>Runner: Return ScrapedSite results
Runner->>Output: render_scraped_sites(ScrapedSite[])
Output-->>Runner: Formatted scraped content + telemetry
Runner-->>User: Print search + scraped results
Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsTimed out fetching pipeline failures after 30000ms Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
larock22
marked this pull request as ready for review
April 28, 2026 22:44
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
src/cli/runner.rs (1)
74-86: Extract shared scrape append logic to reduce drift risk.Both branches repeat the same scrape→render append pattern. Pulling this into one helper will keep behavior consistent and simplify future changes.
Also applies to: 116-128
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/cli/runner.rs` around lines 74 - 86, Extract the duplicated "scrape → render → append to printed" logic into a single async helper (e.g., async fn append_scraped_sites(printed: &mut String, response: &Response, args: &Args) ) that runs deduped_urls_from_response, calls scrape_seed_urls with page_limit_for_scrape and timeout_duration_for_scrape, and when scraped is non-empty pushes a '\n' and render_scraped_sites(&scraped) into printed; replace both duplicated blocks (the branch using args.scrape and the other identical block) with a call to this helper and await it so behavior remains identical.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@docs/response-shape.md`:
- Around line 195-214: The docs show an enum variant ProviderError(String) but
the code uses SearchError::Provider (see usage in src/cli/scrape.rs), so update
the documentation to match the actual enum variant name: change references and
examples from ProviderError(...) to Provider(...) and adjust the human-readable
mapping (e.g., `Provider("down")` → `provider error: down`) so the SearchError
enum name and example formatting in the docs align with the code's
SearchError::Provider variant.
In `@src/cli/scrape.rs`:
- Around line 17-21: The function page_limit_for_scrape currently allows a CLI
value of 0 which causes internal crawling but returns no visible pages; change
the behavior by clamping the resolved page limit to a minimum of 1 (or
alternatively return an error for zero). Specifically, update
page_limit_for_scrape to coerce any parsed value less than 1 up to 1 (while
still using DEFAULT_SCRAPE_PAGE_LIMIT when None or invalid) so the crawler and
truncation logic (the places that currently enforce an internal minimum) operate
consistently and a --scrape-limit 0 no longer performs wasted work with empty
output.
---
Nitpick comments:
In `@src/cli/runner.rs`:
- Around line 74-86: Extract the duplicated "scrape → render → append to
printed" logic into a single async helper (e.g., async fn
append_scraped_sites(printed: &mut String, response: &Response, args: &Args) )
that runs deduped_urls_from_response, calls scrape_seed_urls with
page_limit_for_scrape and timeout_duration_for_scrape, and when scraped is
non-empty pushes a '\n' and render_scraped_sites(&scraped) into printed; replace
both duplicated blocks (the branch using args.scrape and the other identical
block) with a call to this helper and await it so behavior remains identical.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ef6492d2-7b71-487f-ba43-119eba1ceb6d
⛔ Files ignored due to path filters (1)
Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (14)
AGENTS.mdCargo.tomlREADME.mddocs/SUMMARY.mddocs/dependency-architecture-map.htmldocs/dependency-architecture-map.mddocs/response-shape.mdsrc/cli/args.rssrc/cli/mod.rssrc/cli/output.rssrc/cli/request.rssrc/cli/runner.rssrc/cli/scrape.rstests/integration/cli_test.rs
Comment on lines
+195
to
+214
| pub enum SearchError { | ||
| InvalidQuery(String), | ||
| Unauthorized, | ||
| RateLimited, | ||
| ProviderError(String), | ||
| NetworkError(String), | ||
| SerializationError(String), | ||
| Unexpected(String), | ||
| } | ||
| ``` | ||
|
|
||
| In text output, errors are formatted as lower-case messages prefixed by the variant: | ||
|
|
||
| - `InvalidQuery("foo")` → `invalid query: foo` | ||
| - `Unauthorized` → `unauthorized` | ||
| - `RateLimited` → `rate limited` | ||
| - `ProviderError("down")` → `provider error: down` | ||
| - `NetworkError("timeout")` → `network error: timeout` | ||
| - `SerializationError("bad json")` → `serialization error: bad json` | ||
| - `Unexpected("oops")` → `unexpected error: oops` |
There was a problem hiding this comment.
SearchError variant name appears incorrect in docs.
The documented ProviderError(String) doesn’t match code usage (SearchError::Provider(...), see src/cli/scrape.rs Line [320]). Please align the enum variant name and its example mapping to prevent API confusion.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@docs/response-shape.md` around lines 195 - 214, The docs show an enum variant
ProviderError(String) but the code uses SearchError::Provider (see usage in
src/cli/scrape.rs), so update the documentation to match the actual enum variant
name: change references and examples from ProviderError(...) to Provider(...)
and adjust the human-readable mapping (e.g., `Provider("down")` → `provider
error: down`) so the SearchError enum name and example formatting in the docs
align with the code's SearchError::Provider variant.
Comment on lines
+17
to
+21
| pub fn page_limit_for_scrape(cli: &crate::cli::args::CliArgs) -> u32 { | ||
| cli.scrape_limit | ||
| .and_then(|n| u32::try_from(n).ok()) | ||
| .unwrap_or(DEFAULT_SCRAPE_PAGE_LIMIT) | ||
| } |
There was a problem hiding this comment.
--scrape-limit 0 still crawls pages internally but returns empty output.
Because Line [143] forces an internal minimum crawl while Lines [166] and [170] truncate to the requested limit, a zero limit performs crawl work with zero visible pages. Consider clamping to at least 1 (or explicitly rejecting 0) before crawling.
Suggested minimal fix
pub fn page_limit_for_scrape(cli: &crate::cli::args::CliArgs) -> u32 {
cli.scrape_limit
.and_then(|n| u32::try_from(n).ok())
.unwrap_or(DEFAULT_SCRAPE_PAGE_LIMIT)
+ .max(1)
}Also applies to: 143-170
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/cli/scrape.rs` around lines 17 - 21, The function page_limit_for_scrape
currently allows a CLI value of 0 which causes internal crawling but returns no
visible pages; change the behavior by clamping the resolved page limit to a
minimum of 1 (or alternatively return an error for zero). Specifically, update
page_limit_for_scrape to coerce any parsed value less than 1 up to 1 (while
still using DEFAULT_SCRAPE_PAGE_LIMIT when None or invalid) so the crawler and
truncation logic (the places that currently enforce an internal minimum) operate
consistently and a --scrape-limit 0 no longer performs wasted work with empty
output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--scrapesupport that deduplicates search result URLs, runs Spider/Chrome, and renders scraped content excerpts separately from crawl telemetry.Website::scrape()flow and read captured pages throughget_pages()/get_content().with_limit(1)behavior by using an internal minimum of 2 while truncating rendered pages and visited URLs back to the user's requested limit.--scrape-limitis pages per result URL seed, not top-N search results.Root Cause
The previous implementation used crawl telemetry as a proxy for scraping. It could show visited URLs without proving content was captured, and the manual subscriber path timed out during live verification.
Validation
cargo fmt --checkcargo test scrapejust checkcargo run -- "rust testing" --limit 1 --scrape --scrape-limit 1 --scrape-timeout-seconds 60Extracted content pages: 1,Status: 200, and an HTML content excerpt.Notes
The displayed content is intentionally truncated by the CLI renderer. The full scraped page body is captured before rendering, but this PR does not add full-content artifact export or HTML-to-clean-text conversion.
Summary by CodeRabbit
New Features
--scrapeCLI flag to crawl URLs from search results with configurable page limits and timeouts.Documentation