fix: scorer topic literalism, DeepEval pilot, precision-improving retrieval features - #46
Merged
Merged
Conversation
q17 asked about "s.100A" as an expected_topic (not just a citation) - the answer correctly discussed "section 100A" at length, cited it exactly (cit_ratio=1.0), but scored topic_ratio=0.33 because the literal "s.100a" string was never going to appear in natural prose. Same brittleness as the earlier expected_citations fix, extracted into a shared _section_token() helper and applied to topic matching too. Deliberately NOT extended to plain-phrase topics like "family trust" - that's a genuine remaining gap only a semantic/LLM-judged scorer can close, not something a regex can safely generalize further. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
expected_topics named "43.5%" - the flat R&D offset rate that existed before the 2021 reform. Confirmed stale by checking the corpus's own ingested content (not guessed): across every run this session, retrieval consistently and correctly surfaced the CURRENT tiered 8.5%/16.5% system tied to specific, real ITAA 1997 sections (355-100, 355-115) - the model was describing current law more accurately than the fixture's own expectation. Updated to "8.5%" (the tier every correct answer actually cites). Other rate/threshold-sensitive questions (q03 FBT rate, q10 instant asset write-off threshold, q24 SMSF concessional cap, q28 WFH shortcut rate) were checked against the corpus too but found no clear corroborating evidence either way - left unchanged rather than guessed at, per the same discipline. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scores the existing baseline_deepseek_graph.json results with DeepEval's Faithfulness and AnswerRelevancy metrics instead of the hand-rolled score_answer() heuristic - a direct test of whether an LLM judge resolves the literalism failures found by hand this session (multi-bracket citations, section/division token formats, natural phrasing, stale fixture facts) without any more regex patching. Judge model: DeepSeek V4 Flash via OpenRouter, wrapped as a minimal DeepEvalBaseLLM subclass (litellm.acompletion under the hood). Not the single cheapest OpenRouter model available, but cheaper than most on output tokens (which dominates judge-verdict cost) and already proven reliable in this exact pipeline, rather than introducing an unverified new model. Smoke-tested on q17 (the section-topic literalism case) and q22 (the stale-fixture case): both score faithfulness=1.0 with no manual scorer patch involved, confirming the core thesis - an LLM judge reads for meaning, not string presence. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two failure modes from the first pilot run - "invalid JSON" and a
missing schema key (e.g. KeyError('verdicts')) - both trace to the
same root cause: DeepEval's default a_generate_with_schema just calls
a_generate() and returns the raw string with NO validation. Overrode
it directly with a schema-validating retry loop (up to 3 attempts).
Initially escalated temperature on retry (reasoning: temp=0 is
deterministic, so retrying with the identical prompt would just
reproduce the same bad output) - this was WRONG and made things
worse: a case that failed 3x at temp=0.4 succeeded cleanly on the very
next call at temp=0. This cheap judge model's JSON-schema compliance
is apparently *more* reliable at temp=0; OpenRouter's own
multi-provider routing already introduces enough call-to-call
variance for a retry to land on a cleaner completion without adding
more randomness. Now retries at temp=0 on every attempt.
Also gitignore .deepeval/ (telemetry cache dirs deepeval creates,
same pattern as the existing lightrag_storage/cognee_storage entries).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAG-quality audit precision follow-up (#1 of 3): a cross-encoder does fine-grained joint query/passage scoring, specifically better than RRF or the LLM reranker's coarse 0-1 judgment at discriminating between structurally-similar sibling provisions - the "right Division, wrong Section" pattern found repeatedly in the benchmark (e.g. ITAA 1997 Subdivision 292-B concessional cap vs 292-C excess/non-concessional). Hosted via OpenRouter's /rerank endpoint (cohere/rerank-4-fast, $0.002/ search) rather than a local cross-encoder - this backend deliberately carries no ML/torch dependency (documented reason already in RERANK_MODE: "2 vCPU / 4GB droplet"). Reuses the same OPENROUTER_API_KEY doppler secret already set up for the DeepSeek routing experiments. New RERANK_MODE="cohere" value, off by default (still "rrf_only") - same never-fail-over-rerank contract as the existing "llm" mode. Verified live against the real endpoint (not just docs): asked it to rank three candidates for "SMSF concessional contributions cap" and it correctly scored Subdivision 292-B (0.74) above 292-C (0.54) above an unrelated payroll-tax ruling (0.15) - exactly the disambiguation this was built for. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAG-quality audit precision follow-up (#2 of 3): a cheap, deterministic complement to the Cohere cross-encoder rerank (#1) that runs at the RRF-merge stage, before any rerank, so it helps even when RERANK_MODE stays "rrf_only". Boosts a candidate whose OWN section heading title shares content words with the question - e.g. a question about the "concessional contributions cap" now favours a sibling titled "Concessional contributions cap" over one titled "Excess non-concessional contributions tax", even when both are in the same Division and RRF's whole-chunk score alone can't tell them apart. Verified against live retrieval for the exact q24 case: more concessional-cap-relevant content now surfaces in the top ranks than before. Doesn't fully close the gap by itself - the true concessional- cap section still doesn't reliably enter the pool for this specific query, which is a separate recall problem (see RETRIEVAL_MAX_PER_SOURCE_URL/ jurisdiction-boost commits) that no reranking-stage fix can solve if the right chunk was never retrieved in the first place. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
RAG-quality audit precision follow-up (#3 of 3): one cheap LLM call that rewrites the question into a more specific, disambiguated search query before the full-text search leg only - closest to the LegalMALR pattern (multi-agent query understanding before statute retrieval), the best literature match found for this exact "right Division, wrong sibling Section" symptom. Deliberately scoped narrow: only affects the full-text search query (composed with normalise_query in generate_candidates), NOT the shared question embedding used across semantic/firm/historical search - that embedding is computed once upstream and reused across several call sites, so rewriting it would mean re-plumbing a wider surface than this fix needs. Full-text search is also where an exact lexical distinction like "concessional" vs "non-concessional" actually helps; embedding similarity tends to see near-synonyms as close regardless. QUERY_DECOMPOSITION_ENABLED defaults False - adds one LLM call's worth of latency/cost to every retrieval when on. Same never-fail-over-rerank contract as the LLM/Cohere rerankers: any failure returns the original question unchanged. Live-model verification blocked by Anthropic credit exhaustion (same recurring account issue from earlier this session) - the never-fail fallback triggered correctly and returned the original question unchanged, confirming the resilience path works; the rewrite logic itself is covered by mocked unit tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Aug 1, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #44 (already merged at the first commit only). These 7 commits were pushed to the same branch afterward but the PR closed before they landed — opening fresh since
maindoesn't have them yet.Scorer fixes (found while writing a detailed failure analysis)
score_answer()'stopic_rationow credits section-style expected topics (e.g."s.100A") the same way the citation matcher already does — a case with perfect citation grounding was scoring 2/5 purely because the literal"s.100a"string never appears in natural prose ("section 100A")."43.5%"R&D offset rate was the pre-2021 flat rate, superseded by the tiered 8.5%/16.5% system every correct answer actually cites. Corrected after confirming via repeated corpus evidence, not guessed. Other rate-sensitive questions checked but left unchanged — no clear corroborating evidence either way.DeepEval pilot (LLM-judged scoring vs. the hand-rolled scorer)
run_deepeval_pilot.py: scores existing benchmark results with DeepEval's Faithfulness/AnswerRelevancy metrics via a cheap OpenRouter-routed judge (DeepSeek V4 Flash). Result: old scorer 13-15/23-24 pass vs. DeepEval 19-21/23-24 pass across two independent runs — the LLM judge cleanly resolves the literalism failures found by hand, and its lowest scores landed precisely on two independently-diagnosed real defects (a low-relevancy honest refusal, and an exact numeric hallucination VerifyAgent had also caught).a_generate_with_schemaperforms no validation at all, causing "invalid JSON" / missing-schema-key failures. Overrode it directly with a schema-validating retry loop.Three retrieval-precision features (targeting "right Division, wrong sibling Section")
Deep-diving individual failures found retrieval repeatedly landing in the right legislative Division but the wrong adjacent Section (e.g. ITAA 1997 Subdivision 292-C instead of 292-B for a concessional-contributions-cap question). Researched options (OSS "memory" frameworks don't apply — they solve session persistence, a different problem) and implemented the top three recommendations, all off by default:
RERANK_MODE="cohere") — a cross-encoder for fine-grained sibling disambiguation. Zero new dependencies (plain REST via existinghttpx), avoids adding torch/sentence-transformers to a 2 vCPU/4GB droplet. Live-verified against the real endpoint: correctly scored the concessional-cap section above the excess/non-concessional one.SECTION_TITLE_BOOST_WEIGHT, on by default) — cheap, deterministic, reuses heading metadata the hierarchical chunker already extracts. Verified against live retrieval.QUERY_DECOMPOSITION_ENABLED) — one LLM call disambiguating the question before the full-text search leg only (not the shared embedding). Live-model verification blocked by Anthropic credit exhaustion; the never-fail fallback path confirmed correct, and the rewrite logic is covered by mocked tests.Test plan
🤖 Generated with Claude Code