fix: query decomposition compression bug, opt-in Cohere rerank flag, experiment runner updates - #47
Merged
Merged
Conversation
…nner USE_COHERE_RERANK=1 sets RERANK_MODE="cohere" (plus the reranker's OPENROUTER_API_KEY) before the graph module is imported, and saves results under a distinct filename (baseline_deepseek_graph_cohere_rerank) so an A/B run never overwrites the no-rerank baseline - opt-in via env var so the script stays usable for both a baseline and a reranked run. Validated with a full 30-question run: 16/30 -> 20/30 pass, with q14 (the exact thin-capitalisation/Division 820 sibling-section case this feature targets) fully recovering from 2/5 to 5/5. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two independent full 30-question benchmark runs with query
decomposition enabled both regressed below the Cohere-rerank-only
baseline (20/30 -> 17/30, then 20/30 -> 15/30) - not noise, since 4
questions (q07, q15, q19, q27) dropped in BOTH runs.
Root cause found by inspecting the actual rewrites: the old prompt
said "Keep it short (1-2 sentences)", which encouraged the model to
strip natural-language question words and compress into a bare
keyword fragment ("PAYG withholding variation eligibility conditions
employer application process" instead of the original question) -
apparently losing signal plainto_tsquery was actually using, even
though no individual "important" word was dropped.
Redesigned to be strictly append-only: default to returning the
question completely unchanged, and when a genuine sibling-provision
ambiguity exists, APPEND a clarifying phrase rather than rewrite
anything. Verified against all 4 previously-regressed questions (now
consistently unchanged) and the original q24 target case (now
genuinely appends "concessional contributions cap" rather than
replacing the question).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…nt runner
Extends the A/B experiment runner with USE_QUERY_DECOMPOSITION, mirroring
the existing USE_COHERE_RERANK toggle, so the query-decomposition fix
(destructive compression bug, previous commit) can be evaluated against
the DeepSeek/graph baseline without touching production code paths -
resolves via providers.resolve_model("rerank"), already pointed at
DeepSeek/OpenRouter in this runner, so it doesn't need Anthropic credits.
Also ignores .ua/, a code-understanding tool's local scan-cache output
that isn't source and shouldn't be tracked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up commits on this branch since PR #46 merged:
Also adds
.ua/(a code-understanding tool's local scan-cache output) to.gitignore— it isn't source and was showing up as untracked noise.Reviewer notes
This is retrieval/experiment-tooling work, not customer-facing — no dashboard or API surface changes. The two "opt-in flag" commits only affect standalone experiment scripts under
scripts/experiments/, gated behind env vars, so production request paths are unaffected.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com