Calibrate RAG eval gate and router lifecycle (Batch 73) - #78
Merged
Merged
Conversation
Batch 73 adds the operational layer for the adaptive RAG router: judge calibration, baseline-derived thresholds, nightly faithfulness monitoring, and a router data lifecycle for capture, deduplication, triage, retraining, and measurement. New labs: - labs/38-calibrating-the-eval-gate - labs/39-router-data-lifecycle Updated gate: - labs/37-rag-eval-gates/eval_gate.py now supports --thresholds New workflow: - .github/workflows/rag-faithfulness-nightly.yml Lab 38 covers judge-vs-human agreement, Cohen's kappa, threshold derivation, gate_thresholds.json, and a scheduled non-blocking faithfulness monitor. Lab 39 covers captured query handling, deduplication, confidence triage, second-round training data, and measuring whether retraining improves router behavior on messy queries. Updated indexes: - learning-paths/02-agentic-rag/README.md - labs/README.md - CHANGELOG.md Pre-flight all green: ruff clean, notebook JSON valid, notebook outputs clean, eval gate self-test passes, threshold derivation self-test passes, nightly monitor self-test passes, and no new broken links introduced. Batch 73.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Batch 73 adds the operational layer for the adaptive RAG router: judge calibration, baseline-derived thresholds, nightly faithfulness monitoring, and router training-data lifecycle.
Files added:
Files updated:
Pre-flight:
Batch 73.