Summarize CBDR/LCR papers and draft confidence-aware reranking spec#7
Open
pko89403 wants to merge 23 commits into
Open
Summarize CBDR/LCR papers and draft confidence-aware reranking spec#7pko89403 wants to merge 23 commits into
pko89403 wants to merge 23 commits into
Conversation
Reference digests written strictly from the committed PDFs, with page- cited numbers and explicit '논문에 명시되지 않음' markers for gaps (including the CBDR paper's internal inconsistencies: Mid_Layer vs hidden_states[-1], dataset-size and epoch discrepancies). - docs/wiki/references/cbdr_parametric_confidence_rag.md: CBDR requires white-box hidden-state access at train and inference, so only its design pattern transfers to ranksmith; retrieval triggering is out of library scope - docs/wiki/references/llm_confidence_reranker.md: training-free MSCP reranking; conflicts with the temperature-0 JSON-only ModelClient contract are recorded - docs/specs/spec_confidence_aware_reranking.md: Draft spec consuming the runtime-readiness signal contract; five open decisions filed as Q004 in 05_open_questions.md, no implementation before resolution - References index updated; stale metrics path fixed in spec TEMPLATE Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Q004-1 resolved: confidence source = structural + LightGBM scorer (path A), chosen over LCR/MSCP because MSCP needs K+ LLM calls per document. Smoke run (LM Studio qwen3.5-9b, libomp installed) confirmed: - generation pipeline runs end-to-end (15 fixture rows generated) - FrozenAutoEncoder loads and extract_structural_features produces the documented 70-dim structural-v1 vector on real data - training is blocked only by the MIN_TOTAL_SAMPLES=30 guard; the repo fixture yields 15, so producing an artifact needs >=30 real labels Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fetched real BEIR/SciFact data (56 samples: 14 qrels positives + 42 lexical hard negatives), generated judgments via LM Studio, extracted bert features, trained LightGBM + sigmoid calibration, and produced a real scorer.joblib that scores held-out test features. Two findings recorded in Q004: 1. load_lightgbm_scorer misroutes: when metadata_path is passed it assumes a raw Booster file, but the training pipeline exports a joblib dict. The spec's documented from_artifact(path, metadata_path=) call fails on the pipeline's own artifact; from_artifact(path) works. Same root cause as YAGNI finding #1 (4 loader formats, 1 produced). 2. macOS ARM: torch + lightgbm in one process deadlock/segfault via dual OpenMP; artifact was produced by splitting extraction and training into separate processes. Not a ranksmith code bug. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confidence source = path A (structural confidence + LightGBM); form = new Strategy; ranking = signed confidence. Confidence is computed locally (frozen encoder + LightGBM scorer) with zero LLM calls — only one relevance judge() call per document, never LCR-style repeated sampling. - model.py: add ModelClient.judge / AsyncModelClient.judge (relevance verdict), mirroring rank/compare/select - parsing.py: add parse_judgment_response - strategies/confidence.py: ConfidenceRerankStrategy / AsyncConfidenceRerankStrategy. Per doc: judge -> structural confidence -> signed score (+conf if relevant, -conf if not); sort desc, stable on ties. Estimator typed via Protocol so strategies never force a torch import. Async gathers judge calls concurrently, scores sequentially. - Exported from ranksmith and ranksmith.strategies. Verified: 8 unit tests (synthetic judge + fake estimator) + end-to-end against LM Studio judge with the smoke artifact (relevant docs sort to the top, 3/3). 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the earlier signed-judgment reranker with the CBDR-faithful confidence-change approach. CBDR ranks documents by how much each one raises the model's answer confidence: Inc = C(query+doc) - C(query). Within a single query the baseline C(query) is a constant (and answer_confidence cannot score an empty context anyway), so ordering by C(query+doc) is exactly the confidence-change ranking. - model.py: replace judge() with answer(query, context) on both clients, using the same answer prompt as the generation pipeline so training and inference stay consistent - parsing.py: replace parse_judgment_response with parse_answer_response - strategies/confidence.py: AnswerConfidenceRerankStrategy / AsyncAnswerConfidenceRerankStrategy. Per doc: answer -> local answer_confidence -> sort desc. Estimator typed via Protocol; async gathers answer calls, scores sequentially. Confidence uses zero LLM calls; one answer call per document, never LCR-style repeated sampling. Verified: 8 unit tests + end-to-end against LM Studio with an answer_confidence artifact trained on 60 SQuAD samples — the gold context sorts to rank 1 (magnitudes cluster; the smoke artifact discriminates weakly, as expected). 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Held-out gold-context recovery over 15 SQuAD cases (gold + 3 random distractors), answer_confidence trained on LM Studio judgments: samples test_auc rerank_acc@1 MRR 100 0.333 0.000 0.294 500 0.875 0.800 0.878 random 0.5 0.250 0.521 Data scale is the deciding factor: 100 samples overfit (worse than random, gold sinks to the bottom); 500 samples discriminate strongly (gold ranks #1 in 12/15). Honest caveats recorded: distractors are unrelated SQuAD contexts (easy signal via NO_ANSWER), eval set is small, and no README performance claim is made. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ponytail review found the answer prompt duplicated byte-for-byte in model._answer_messages (reranker inference) and confidence_generation.build_answer_prompt (training data generation). They must stay identical for train/inference consistency, but nothing enforced it, and generation's configurable no_answer_value could silently diverge from the reranker's. Make model._answer_messages the single builder (now takes no_answer_value, default '__NO_ANSWER__'); the generation pipeline calls it. Delete ANSWER_SYSTEM_PROMPT and build_answer_prompt from confidence_generation. Tests updated to the shared builder. 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the earlier prompt unification. The reranker hardcoded the NO_ANSWER sentinel while AnswerGenerationConfig still let generation set a different no_answer_value — so a custom-sentinel artifact would silently mismatch the reranker's inference prompt. My earlier 'consistency guaranteed by code' claim was therefore false. Fix the sentinel to the single NO_ANSWER_VALUE constant (model.py): - drop no_answer_value from _answer_messages, AnswerGenerationConfig, and normalized_exact_match; all use the constant - generation labeling/metadata source it from the same constant Verified by reflection that no divergence path remains (no no_answer parameter on any of the three). 439 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same 15 held-out cases, ConfidenceRerank (500-sample artifact) vs ListwiseStrategy: strategy acc@1 MRR calls/query ConfidenceRerank (500) 0.800 0.878 4.0 ListwiseStrategy 1.000 1.000 1.0 Listwise is perfect and 4x cheaper on this easy setting. No setting has yet shown the confidence reranker beating an existing strategy. Recorded honestly; usefulness is not oversold. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
examples/answer_confidence.py mirrors the other fake-provider examples (deterministic KeywordAnswerClient + a keyword estimator standing in for StructuralConfidenceEstimator.from_artifact). test_examples runs it without a live provider and pins the output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Audit found the new strategy was missing from the conventional surfaces that every other strategy covers, and that several docs still said 'confidence is not a reranker' (now false). - Fix the now-false statements: the confidence estimator is a scoring utility; AnswerConfidenceRerankStrategy consumes it and IS a reranking Strategy (guardrails.md, SKILL.md, README x2, 02_architecture.md) - README.md/README.ko.md: add an 'Answer Confidence Reranking (experimental)' section — honestly notes it loses to Listwise (acc@1 0.80 vs 1.00) at 4x cost and needs a trained artifact - advisor method-guide.md: quick-decision row + parameters entry, marked experimental / not-a-default with the Listwise baseline named - snippets.md: reference examples/answer_confidence.py - test_advisor_references: guard AnswerConfidenceRerankStrategy defaults - docs/wiki/00_context.md: list the new strategies 440 tests pass, mypy + ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds 'answer_confidence' as a selectable algorithm in scripts/compare_reranking.py so it compares against the other strategies on the standard qrels benchmark. Opt-in only (excluded from --algorithm all): needs --answer-confidence-artifact pointing at a trained scorer. Estimator is loaded once (lru_cache); nominal call estimate = one answer call per document. Records the special-case caveat (in CLI help and the spec): this IR benchmark has qrels but no gold answers, so it cannot train an answer_confidence scorer — you must supply one trained separately on QA data. Passing an artifact trained on another domain (e.g. SQuAD) measures domain shift, not a fair comparison, and must be labeled as such. The script targets the Azure SDK, so it does not run against LM Studio here; scripts/ is outside the mypy CI gate. 440 tests pass (30 compare_reranking), ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ere reality - compare_reranking.py: RANKSMITH_OPENAI_BASE_URL (+_MODEL/_API_KEY) points the run at any OpenAI-compatible endpoint (LM Studio, vLLM, ...) instead of Azure, for local/test runs. Azure stays the default when unset. - Spec: record that the README AskUbuntu benchmark cannot be reproduced in this dev env (gitignored corpus cache, Azure VNet-blocked, and AskUbuntu has no gold answers to train an answer_confidence artifact). It must be run on an environment that has the AskUbuntu corpus + Azure access. - Spec: record the preliminary LM-Studio/SciFact diagnostic (answer_confidence 0.311 NDCG@5 vs listwise 1.000 at 10x cost) — explicitly NOT for the README table (domain-shifted artifact, easy distractors, small local model). 440 tests pass, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The answer_confidence reranker needs a scorer artifact, and AskUbuntu (the README benchmark) has no gold answers to train one. This adds a deterministic builder that samples SQuAD v1.1 train questions and emits one gold-context row plus BM25 hard-negative-context rows per question, matching the inference distribution where the estimator scores answers from both relevant and non-relevant candidates. Includes a pure-Python Okapi BM25 (Pyserini-default k1/b) in benchmarks/bm25.py. Verified against the real SQuAD download (87,599 questions): 40-question build produced 80 rows, max context 1,934 chars, zero skips. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
scripts/train_answer_confidence.py chains generation (live model answers labeled against gold answers) and training (frozen encoder + LightGBM + calibration) into one command, printing test-split metrics. Fast fails when generation yields a single class or when test roc_auc drops below 0.6 — the spec recorded a 100-sample artifact at 0.333 (worse than random), and this gate stops such an artifact from reaching a benchmark. Model access mirrors compare_reranking.py: RANKSMITH_OPENAI_BASE_URL for OpenAI-compatible endpoints, Azure env vars otherwise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
Multiple --algorithm flags now run several methods on identical cases in
one invocation ('all' stays the default and cannot be combined;
duplicates are rejected). answer_confidence selection is validated
before any live call: the artifact flag must be present and the file
must exist, instead of failing mid-run after paid provider calls.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
benchmark-results/live/*.merged.json were merged by hand until now. scripts/merge_benchmark_reports.py merges per-method runs (the documented optional-method convention) after verifying benchmark identity fields, candidate file name, disjoint algorithms, and exact query-id set equality across every algorithm — so a new answer_confidence run can be merged with the committed v3 baselines without risking an apples-to-oranges table. Divergent unvalidated fields are recorded per input under merged_from. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
docs/benchmarks/answer_confidence_askubuntu.md records the exact procedure to measure answer_confidence against the committed README methods (AskUbuntu 361 queries, BM25 top-20, gpt-5.4-nano): build SQuAD training data, train the artifact with the same deployment, run the single-method benchmark, merge with the committed v3 results, and label the artifact's out-of-domain provenance in any reported number. The spec's run-elsewhere blocker section now points at the tooling, and the bm25_top20 doc lists answer_confidence as an optional method (20 calls/query). The whole path (build -> generation -> training -> benchmark -> merge) was mechanically verified offline with the real SQuAD download, a deterministic fake OpenAI-compatible server, and a tiny local encoder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
Review findings on the confidence strategy, each reproduced before fixing: - AnswerConfidenceScorer/_ConfidenceResult now use read-only property members; the real StructuralConfidenceEstimator (frozen dataclass, Literal task_type) failed the previous attribute-member protocol under mypy strict while tests passed with mutable fakes. - Blank/whitespace-only documents fast-fail with RerankInputError before any paid answer() call; previously all N calls ran and the estimator then raised ConfidenceInputError, which is outside the RerankError hierarchy every other strategy failure uses. - Result metadata gains the "algorithm" key all five existing strategies emit alongside "strategy". - parse_answer_response is re-exported at the package top level like its sibling parsers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
Closes the coverage gaps the review confirmed: ModelClient.answer / AsyncModelClient.answer had zero direct tests (prompt content, usage emission, provider-error wrapping, empty-response rejection, sync/async prompt parity); parse_answer_response only had its invalid-JSON branch exercised (missing key, wrong type, empty/whitespace answer now covered); the async strategy only had a happy-path test (invalid JSON, missing capability, and pre-call blank-document fast fail now covered, asserting zero answer calls are spent); and compare_reranking's answer_confidence branches (call estimate, artifact-missing ValueError) had no tests. Also asserts the new strategy/algorithm metadata keys and negative top_k rejection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
Review confirmed the docs contradicted the code this branch ships: - 05_open_questions.md: the Q004 entry sat inside the '## 형식' template code fence, so it rendered as example code instead of a real section. It is now a real, resolved entry recording the final decisions (new Strategy, raw-confidence descending sort, QA/SQuAD training data) and the fate of the smoke-run blockers (loader bug fixed via PR #8). - spec_confidence_aware_reranking.md: §2/§3/§5/§6 presented the draft judgment_confidence + LCR binning/gate design as current requirements; they are now explicitly marked superseded with the shipped deltas, §7 marks all four Q004 decisions resolved, and the task checklist matches reality (implementation done, README-table run remaining). - README.md / README.ko.md: dropped the acc@1/MRR numbers the spec forbids putting in the README and the false "committed evaluation" provenance (no evidence artifact exists in the repo); the honest qualitative warning and pointers remain. - 02_architecture.md: strategy list, ModelClient contracts, file tree, and JSON response contracts now include the answer path; same provenance correction as the README. - spec_confidence_generation_pipeline.md: no_answer_value is documented as the shared ranksmith.model.NO_ANSWER_VALUE constant instead of the removed config field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011d5n8TDf8vg22SZYggjnBb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prepares the ground for the confidence-aware reranking (CBDR-direction) work by converting the two pending confidence papers into repo reference knowledge, per
docs/wiki/03_reference_processing.md.Fact discipline: both digests were written strictly from the committed PDF texts. Every number is page-cited in the extraction notes; anything the papers do not state is marked "논문에 명시되지 않음" instead of being filled in. The CBDR paper's own internal inconsistencies (Mid_Layer vs
hidden_states[-1], 1k/300/500 vs 2,000/1,000/500 dataset sizes, 30 vs 100 epochs) are recorded verbatim rather than reconciled.cbdr_parametric_confidence_rag.md— CBDR (anonymous ACL submission). Key finding: the method needs white-box hidden-state access at both training (74,204 target-LLM forward passes) and inference, so it cannot be ported to ranksmith's closed-model premise; only the "confidence signal drives document preference" pattern transfers. CBDR itself is a retrieval trigger, not a reranker — out of library scope.llm_confidence_reranker.md— LCR (arXiv:2602.13571). Training-free, genuinely black-box (sampled text only), so it fits ranksmith better; recorded conflicts: needs temperature-1 sampling K=10 vs our temperature-0 JSON-only ModelClient contract, and the paper never discloses its tuned thresholds or call-cost totals.spec_confidence_aware_reranking.md— Draft spec that consumes the existing runtime-readiness signal contract (order-preservingscore_batch+ caller-side candidate zip).src/ranksmith/_metrics.pypath in spec TEMPLATE fixed tobenchmarks/metrics.py.Update: implementation (commits after the original summary)
The Q004 design decisions were resolved in-session and the strategy was implemented on this branch:
AnswerConfidenceRerankStrategy/AsyncAnswerConfidenceRerankStrategy(src/ranksmith/strategies/confidence.py),ModelClient.answer+parse_answer_response,confidence_generationprompt unification,examples/answer_confidence.py, opt-inanswer_confidenceintegration inscripts/compare_reranking.py, and tests. This PR is not docs-only anymore.Update (2026-07-16): benchmark comparison tooling
Turnkey tooling to measure
answer_confidenceagainst the existing README methods (AskUbuntu 361 queries, BM25 top-20,gpt-5.4-nano) without rerunning the committed baselines:scripts/build_answer_confidence_training_data.py— deterministic SQuAD v1.1 builder: per question one gold-context row + BM25 hard-negative rows (matches the inference distribution); pure-Python Okapi BM25 inbenchmarks/bm25.py(Pyserini-default k1/b). AskUbuntu has no gold answers, so the artifact must be trained on QA data — this fills the SQuAD/QA adapter + CLI gap the generation/training specs explicitly deferred (Phase 2B).scripts/train_answer_confidence.py— generation (live model) → labeling → training → artifact in one command; fast-fails on single-class labels or test roc_auc < 0.6 (the spec's recorded 100-sample artifact scored 0.333).scripts/compare_reranking.py—--algorithmis now repeatable (several methods on identical cases in one run);answer_confidencevalidates the artifact path before any live call.scripts/merge_benchmark_reports.py— merges per-method runs with the committedv3.merged.jsonafter verifying benchmark identity and exact per-algorithm query-id equality (replaces the manual merge).docs/benchmarks/answer_confidence_askubuntu.md— the runbook: build → train (same deployment as the benchmark) → single-method run (7,220 calls) → merge → report with the artifact's out-of-domain (SQuAD) label. Spec updated to point at it.The whole path (build → generation → training → benchmark → merge) was mechanically verified offline with the real SQuAD download, a deterministic fake OpenAI-compatible chat server, and a tiny local encoder; 470 tests + ruff + mypy green. Numbers that depend on LLM quality still need an environment with the AskUbuntu cache and Azure access.
🤖 Generated with Claude Code