feat: switch production answering to DeepSeek via OpenRouter - #49
Merged
Conversation
Comprehensive record of the ingestion + retrieval investigation this session: what's been explored and shipped (hierarchical chunking, jurisdiction/section-title boosts, diversity cap, Cohere rerank, scorer literalism fixes, DeepEval pilot), what's built but not adopted (query decomposition - both the destructive prompt regression and the append-only redesign's still-inconclusive result are documented in detail), and what's identified but not yet started (state legislation ingestion gap, pure recall misses, supersession checking, evaluation methodology). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…_strong to GPT-5.1 Cheap-tier answering (haiku/sonnet + everything aliased to them) now routes through openrouter/deepseek-v4-flash instead of Anthropic Claude, validated against the 30-question accuracy benchmark used throughout the RAG-quality audit. verify_strong is kept as a deliberate exception on GPT-5.1 (also via OpenRouter) rather than following the alias cascade, preserving a real capability gap for VerifyAgent's escalation tier. Requires LLM_API_KEY/ LLM_API_BASE set to OpenRouter in Doppler to take effect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
haiku/sonnettiers, and everything that aliases to them — draft/verify/rerank/classify) now defaults toopenrouter/deepseek/deepseek-v4-flashinstead of Anthropic Claude Haiku/Sonnet.verify_strong(VerifyAgent's escalation tier for the most severely flagged draft answers) is kept as a deliberate exception onopenrouter/openai/gpt-5.1rather than following the alias cascade — an independent model family from the DeepSeek draft generator, stronger structured-output track record, still meaningfully cheaper than Anthropic Sonnet, and rarely invoked so the cost impact is minimal.docs/model-routing.mdrewritten to document the new default, the required Doppler secrets (LLM_API_KEY,LLM_API_BASE=https://openrouter.ai/api/v1), and a "switching back to Anthropic" recipe.Why
This closes out the RAG-quality audit documented in
docs/rag-quality-audit.md: DeepSeek + the retrieval-precision fixes shipped in the prior two PRs (#46, #47) scored comparably to — and on several individual benchmark questions better than — the previous Anthropic-only baseline, at a fraction of the per-token cost.Verification
verify_strongstays distinct from the cheap tier).resolve_model()returns the correct model string for every tier under the new default.ResearchAgent._generatecode path (not an isolated litellm smoke test) againstprd's OpenRouter credentials — both the DeepSeek (haiku/sonnet) and GPT-5.1 (verify_strong) tiers returned correct, well-cited answers with real token usage.Reviewer notes
MODEL_TIER_MAPinconfig.py); no service code hardcodes a model. Takes effect only onceLLM_API_KEY/LLM_API_BASEare set in Doppler for the target config —prdalready has both set as of this PR.docs/model-routing.md.🤖 Generated with Claude Code