rag-eval-gate

Retrieval evaluation that tells you what it can't resolve.

Recall@5 = 1.00 on 10 queries is really [0.69, 1.00]. That +0.30 lift over BM25 has a p-value floor of 0.25 — no outcome of that comparison could ever have been significant. Both of those are computable from the published table alone.

→ Check a number right now, nothing to install

Type in the figure and the query count. It returns the interval that figure supports and, for a claimed improvement, whether any outcome of that comparison could have reached significance. The arithmetic runs in the page; nothing is sent anywhere.

Or from a terminal:

pip install retrieval-eval-gate      # the PyPI name; `rag-eval-gate` there is someone else's

rag-eval-gate audit -n 10 -v 1.00 -b 0.70     # audit a table you are reading
rag-eval-gate power gold.jsonl                # what your own gold set can resolve

Zero dependencies. Works with any retriever in any language. Runs as a GitHub Action. The command is rag-eval-gate; the distribution is retrieval-eval-gate because the shorter name on PyPI belongs to an unrelated project. Both spellings work as the command.

---

Three things your eval is probably not telling you

A perfect score is not a perfect score. Ten queries, one relevant document each, Recall@5 = 1.00 — that is ten successes out of ten, and the exact binomial interval bottoms out at (α/2)^(1/n) = 0.69. The headline is 31 points wide.

Some comparisons cannot be won. Your hybrid retriever beats BM25 by +0.30 Recall@5. On ten queries that is three queries flipping, seven tying — and a paired randomization test on three differing pairs has a smallest attainable two-sided p of 2/2³ = 0.25. Not "not significant": unable to be, whatever the result.

Your interval breaks when you look at it. A 95% interval promises that one look at a finished sample contains the truth 95% of the time. Watching a CI log is not that. Over 800 simulated evaluation streams, inspection starting only at n = 30 where the normal approximation is defensible:

metric	queries	fixed-N interval	confidence sequence
Recall@5 (0/1)	300	34.0%	1.9%
reciprocal rank (discrete, skewed)	300	32.8%	2.6%
nDCG@10 (graded)	300	35.1%	1.6%
groundedness judge (skewed high)	300	30.0%	0.5%

Runs where the true mean escapes a nominal 95% interval at some point while it is watched. The fixed-N column grows with stream length — the longer you look, the more chances to be wrong. That is the law of the iterated logarithm, and it is why the right-hand column exists.

Audit a table in five seconds

No gold set, no retriever, no data — the reported means and the query count are enough:

$ rag-eval-gate audit -n 10 -m recall@5 -v 1.00 -b 0.70

recall@5 = 1.000 on 10 queries
  that is 10/10 successes

  95% interval   [0.692, 1.000]   width 0.308   (exact binomial)
  The reported figure is 31 points wide. Quoting it to two decimal places implies a
  precision 10 queries cannot deliver.

Claimed lift +0.300 over 0.700
  note: the two means determine it exactly: 3 of 10 queries differ

  smallest attainable p   0.2500

  UNRESOLVABLE. Under every possible overlap of the two systems' successes,
  a paired test on this many differing queries has a floor above 0.05, so no
  outcome of that comparison could have reached significance. This is a fact
  about the sample size, not about the systems.

It exits non-zero when the comparison could not have been significant, so it works as a check and not only as something to read. The interval is exact for a 0/1 metric; for a graded one the per-query spread is unknown, so it reports the widest interval the mean allows — a [0,1] variable with mean m has variance at most m(1−m).

The same thing runs in a browser at jinwovo.github.io/rag-eval-gate, with a shareable link for the result.

Quickstart

pip install retrieval-eval-gate

# 1. What can this gold set even measure?  (no retriever needed)
rag-eval-gate power gold.jsonl

# 2. Score it, with intervals and significance
rag-eval-gate eval gold.jsonl --modes bm25,vector,hybrid

# 3. Gate a pull request
rag-eval-gate eval gold.jsonl --gate --min-mrr10 0.85

A gold set is JSONL:

{"query": "how do I rotate a signing key", "relevant_doc_ids": ["kms-rotation"]}

…or the TREC qrels you already have (--gold qrels/test.tsv --queries-file queries.jsonl).

Works with whatever you built

Three ways in, so the language and shape of your retriever are irrelevant.

HTTP — a URL template and a small field map. No assumption about the route or schema:

rag-eval-gate eval gold.jsonl \
  --url-template 'https://search.internal/v2/query?text={query}&strategy={mode}' \
  --results-path 'data.hits' --id-field 'document_id' --score-field 'relevance'

A command — anything that prints JSON to stdout. Go, Rust, a notebook, a shell script:

rag-eval-gate eval gold.jsonl \
  --backend command --command './retrieve --q {query} --mode {mode} --json'

A TREC run file — rank the corpus once, offline, on whatever hardware; evaluate as often as you like with nothing running:

rag-eval-gate eval qrels/test.tsv --backend run-file --run-file runs/bm25.trec

Four gate policies

rag-eval-gate eval gold.jsonl --gate --gate-policy sequential

policy	fails when	use it when
point (default)	the mean falls below the threshold	always — the absolute floor
ci-lower	the 95% lower bound falls below it	your gold set is big enough that the interval is narrower than the safety margin
regression	also on a significant paired drop against a recorded run	you have a green baseline to compare against
sequential	the anytime-valid verdict is fail, or the budget runs out undecided	queries cost money or minutes

regression is the sensitive one. An absolute threshold only notices a regression once the mean crosses a line someone guessed; a paired test compares the same queries before and after, so a real drop on two of them is caught while the mean still clears the line.

sequential is the cheap one. Because a confidence sequence is valid at every sample size, stopping as soon as the verdict is settled is a decision rather than a peek:

the system's true Recall@5	verdict	queries scored (of 300)	saved
0.98	pass	54	82%
0.95	pass	89	70%
0.85 (exactly at the line)	undecided	293	2%
0.60	fail	31	90%

undecided fails. The budget ran out before the evidence arrived, which is a fact about the gold set rather than a clean bill of health — and a gate that treats "we could not tell" as a pass has quietly stopped gating.

As a GitHub Action

- uses: jinwovo/rag-eval-gate@v0
  with:
    gold-file: eval/gold.jsonl
    url-template: 'http://localhost:8080/api/search?q={query}&mode={mode}'
    gate-policy: sequential
    min-mrr10: "0.85"

The step summary gets the interval table, the significance verdicts, what the gold set can resolve, and a ✅/❌.

The action installs its own checkout, so the ref you pin is the tool version: @v0 tracks the latest 0.x, @v0.2.1 pins exactly. It is v0 rather than the customary v1 because the inputs may still change, and a v1 would promise a stability this has not earned yet.

What comes out

queries=300  depth=10  intervals=95%

mode                      recall@5             recall@10                mrr@10               ndcg@10
--------------------------------------------------------------------------------------------------
bm25              0.700 [0.35,0.93]    0.800 [0.44,0.97]    0.670 [0.35,0.90]    0.702 [0.38,0.90]
hybrid            1.000 [0.69,1.00]    1.000 [0.69,1.00]    0.950 [0.70,1.00]    0.963 [0.78,1.00]

paired randomization test vs bm25 (Holm-corrected within each metric)
  recall@5   hybrid    delta=+0.300 p=0.2500 holm=0.2500  unresolvable (3/10 queries differ; floor p = 0.250)
  mrr@10     hybrid    delta=+0.280 p=0.1250 holm=0.2500  unresolvable (4/10 queries differ; floor p = 0.125)

resolution of this gold set (hybrid vs bm25, mrr@10):
  n = 10 queries, per-query difference sd = 0.388
  smallest detectable effect at 80% power: +0.344
  queries needed to resolve +0.020: 2,957

unresolvable is the word this tool exists to print. It is not a softer way of saying "not significant" — it means the queries that differ are too few for any outcome to reach p < 0.05, which is a property of your labels and no amount of rerunning will change it.

The statistics, and why each one

	what	why not the obvious thing
Exact binomial (Clopper–Pearson)	intervals for Recall@k when each query has one relevant document	a bootstrap resamples a constant at 10/10 and returns [1.00, 1.00], which reads as certainty
BCa bootstrap	intervals for graded metrics	percentile intervals under-cover on skewed samples, and reciprocal rank is very skewed
Paired randomization test Smucker, Allan & Carterette, CIKM 2007	mode comparisons	reciprocal rank is discrete, bounded and skewed; a t-test at n = 10 is not measuring what it claims
Holm–Bonferroni	correction across a mode sweep	five challengers against one baseline is five chances at a false win; uncorrected 0.05 is about 0.23
Betting confidence sequences Waudby-Smith & Ramdas, JRSS-B 2024	the sequential gate	alpha spending needs the looks planned in advance; a CI log is inspected continuously
Conformal prediction, adaptive sets Romano, Sesia & Candès, NeurIPS 2020	calibrate: how many passages to send an LLM	a fixed top-K over-spends on easy queries and truncates the hard ones
Risk-controlling prediction sets Bates et al., JACM 2021	calibrate: when to abstain	a hand-picked threshold implies a claim about hallucination rate that was never checked
Prediction-powered inference Angelopoulos et al., Science 2023	ppi_mean: a valid interval when only a biased LLM judge scored your data	averaging the judge gives a tight interval around the judge's belief, which covers the truth ~0% of the time

Every one is seeded, so a gate cannot flap because a resample drew differently.

Certified thresholds

rag-eval-gate calibrate gold.jsonl --alpha 0.10 --risk-alpha 0.05

Turns the two constants that decide what a RAG system costs and how often it can make things up into thresholds with finite-sample, distribution-free guarantees:

coverage   P(the passages sent to the LLM contain a relevant document)  >= 1 - alpha
risk       P( P(answering from context with no relevant document) <= risk-alpha ) >= 1 - delta

Measured on 800 queries against a stand-in cross-encoder: 5.0 passages against a fixed 9 carrying the identical promise — 44% fewer prompt tokens, because the budget moves to where it is needed (3.2 passages when the answer ranked first, 8.4 when it ranked fourth or lower). Three splits — one picks the softmax temperature, one fits the quantile, one scores both having informed neither — because choosing the temperature on the data the quantile is fitted on breaks exchangeability, and the resulting under-coverage is invisible in every in-sample number.

A certificate that abstains on everything is valid, useless, and labelled degenerate. A threshold search that stalls names the calibration size that would settle it.

The certificate above expires

The guarantee from calibrate holds while calibration and serving queries are draws from the same distribution. A live corpus stops satisfying that in a week — documents get ingested, query mix moves, someone swaps the embedding model — and the failure is silent: config unchanged, dashboards green, promise no longer true.

from rag_eval_gate import AdaptiveConformal

controller = AdaptiveConformal(alpha=0.10, gamma=0.05, window=600)
for score in calibration_scores:
    controller.scores.append(score)          # seed the window, no feedback yet

# serving
cutoff = controller.threshold()              # apply this
controller.observe(score)                    # then close the loop
print(controller.report())                   # offset, whether the bound held, compensating

The level is driven by realised miscoverage (Gibbs & Candès, NeurIPS 2021). Summing the update telescopes to an exact identity and the level provably cannot escape [-γ, 1+γ], so long-run coverage converges to target under no distributional assumption at all — not exchangeability, not stationarity, not even that the shift is random rather than adversarial. DtACI (JMLR 2024) runs several step sizes as experts if you would rather not pick γ.

200 streams, 3,000 queries each, promise 90%:

scenario	frozen threshold	rolling window	adaptive
location shift (+0.30)	0.459	0.876	0.900
score saturation	0.306	0.939	0.903

The middle column is the honest part. Against a plain location shift, rolling recalibration does essentially all the work by itself — across shifts of 0.00, 0.20 and 0.30 the widest excursion of the level from target is the same 0.100, so the shift never reaches the controller. If that is your only worry, recalibrate on recent traffic and skip this module. What the controller buys is the residual (the window is systematically off even when it works, 87.6% against a promised 90%) and the case no rolling quantile can track: when most candidates pile onto one tied top score, the window overcovers at 93.9% — not safety, just prompt tokens spent on sets larger than the guarantee needs.

Which is why the signal is compensating, not drifting. It stays False through a shift the window absorbs (0/60 streams) and fires on every saturating one (60/60). Naming it a drift detector would have been wrong, and the number that first looked like drift detection turned out to be an artefact of a test generator clamping scores at 1.0.

When only an LLM has judged your data

Your groundedness score is a judge model's opinion of the system it belongs to. Your relevance labels, at any real scale, were written by a model too. Both are cheap and neither is trustworthy on its own, and averaging thousands of them does not produce a 95% interval — it produces a very narrow interval around whatever the model believes.

ppi_mean implements prediction-powered inference (Angelopoulos et al., Science 2023, power-tuned as PPI++): hand-label a small sample, let the model predict everything, and subtract the bias measured on the labelled part.

from rag_eval_gate import ppi_mean

estimate = ppi_mean(hand_labels, judge_on_those, judge_on_everything_else)
print(estimate.statement())
# The judge reports 0.886. Measured against 50 hand labels it is optimistic by 0.081,
# so the true value is 0.805 [0.781, 0.829] — an interval worth about 474 hand labels,
# from 50.

Over 1,500 repetitions of the whole label-and-estimate cycle, 50 hand labels against 2,050 judge-scored items:

judge behaviour	coverage: judge only	hand labels only	PPI	interval width	effective labels (from 50)
flattering, accurate	0.0%	94.0%	94.3%	−67%	474
flattering, noisy	0.0%	94.2%	93.5%	−22%	84
unbiased, noisy	14.5%	94.7%	93.7%	−22%	85
uninformative	0.0%	94.8%	94.3%	−1%	51

Validity never depends on the judge being good — the bias is measured, not assumed. And a useless judge costs nothing: λ is tuned to minimise variance, so noise gets λ ≈ 0 and the estimator falls back to the hand-label mean. That last row is the safety property, not a failure.

Why zero dependencies

This runs inside a CI job that should not need a resolver, a wheel build or a lockfile to gate a pull request. The machinery it needs — a regularized incomplete beta, a BCa bootstrap, a martingale — is a few hundred readable lines, and pip install retrieval-eval-gate installs exactly one thing. The test suite cross-checks the hand-rolled special functions against SciPy, and skips those checks when SciPy is not installed.

Tests

pip install -e ".[dev]" && pytest

220 tests. Against closed forms ((α/2)^(1/n) at k = n, 2^(1-k) for a uniform improvement, R's p.adjust worked example, the (n+1) conformal quantile correction), against an independent brute-force permutation reference written inside the test, and — for the guarantees — by simulating the whole calibrate-then-deploy cycle and counting how often each promise actually breaks.

Provenance

Extracted from jinwovo/recall, a hybrid-search and grounded-RAG system, after its own eval harness was pointed at its own README and three of the headline claims failed. The design notes live there as ADRs 0011–0014.

License

MIT.