Skip to content

feat: add Tavily as parallel search engine option#33

Open
tavily-integrations wants to merge 2 commits into
Mrbaeksang:mainfrom
Tavily-FDE:feat/tavily-migration/deepcloak-core-tavily
Open

feat: add Tavily as parallel search engine option#33
tavily-integrations wants to merge 2 commits into
Mrbaeksang:mainfrom
Tavily-FDE:feat/tavily-migration/deepcloak-core-tavily

Conversation

@tavily-integrations

Copy link
Copy Markdown

Summary

Adds Tavily as a third configurable --engine tavily search provider alongside the existing DuckDuckGo (default) and SearXNG options. This is an additive change — no existing providers are modified.

When --engine tavily is used, search results feed through the same StealthRetriever pipeline, so full-page stealth fetch, Evidence Records, and Bot Wall detection work identically to the SearXNG path.

Changes

pyproject.toml

  • Added tavily-python>=0.5 under a new [tavily] optional extra

src/deepcloak/config.py

  • Added 'tavily' to _VALID_ENGINE set
  • Added tavily_api_key: str | None field to Settings dataclass
  • resolve() now reads TAVILY_API_KEY from the environment

src/deepcloak/retriever.py

  • Added tavily_search(api_key, query, max_results) using TavilyClient.search()
  • Refactored build_stealth_retriever() to accept an optional search_fn callable (defaults to searxng_search for backward compatibility)

src/deepcloak/research_core.py

  • Added a tavily branch in _run_ldr() that builds the StealthRetriever with tavily_search bound to the API key, gated on settings.tavily_api_key

src/deepcloak/cli.py

  • Added 'tavily' to --engine choices

Dependency changes

  • Added tavily-python>=0.5 as optional extra [tavily] in pyproject.toml

Environment variable changes

  • Added TAVILY_API_KEY — read by config.resolve(); required when --engine tavily is used

Notes for reviewers

  • The build_stealth_retriever() signature change is backward-compatible: searxng_url is now optional (with default None) and existing callers passing it as a keyword arg continue to work unchanged.
  • tavily-python is opt-in via pip install deepcloak[tavily] — it is not added to core dependencies.

Automated Review

  • Passed after 2 attempt(s)
  • Final review: The Tavily migration unit is correct, complete, and consistent with the codebase. All 6 reported fixes are present: ConfigError on missing key, 'tavily'→'duckduckgo' in to_ldr_overrides(), pre-constructed TavilyClient passed via functools.partial, 3 config tests, 2 retriever tests, and README+pyproject.toml updates. The implementation follows the same patterns as the existing SearXNG integration. One minor concern: the broad except Exception: pass in research_core.py silently swallows ImportError if tavily-python is not installed, falling back to duckduckgo without user feedback — but this is consistent with the existing SearXNG branch and is not a blocker.

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

Someone is attempting to deploy a commit to the Baeksang's projects Team on Vercel.

A member of the Team first needs to authorize it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant