qaeval

A proof-of-work evaluation harness for a retrieval-augmented question answering system that cites its sources: an agent that answers from grounded passages, points at the exact snippets it relied on, and abstains (fails closed) when it can't ground an answer. The harness makes the answer to whether that system works a number, not a vibe: every claim is checked against a source the agent cited, on a fixed set of hand-curated questions, gated in CI, and watched over time.

It runs end to end out of the box against a bundled in-process QA hub, with zero third-party dependencies (Python 3.10+ standard library; pytest is the only dev dependency).

At a glance

• Surface: cited, multi-source question answering over frozen per-case fixtures.
• Trust rule: answer only from retrieved context, cite the exact snippet, abstain when nothing grounds the answer, and never spend a model call on empty context.
• Eval layer: deterministic trust checks, retrieval metrics, a calibrated model-as-judge, trajectory scoring, drift alarms, and a layered CI gate.
• Headline metric: faithfulness_rate, the share of answers where every claim is entailed by the cited source, read next to abstention_recall and over_refusal_rate so neither failure direction can hide.
• SUT boundary: adapters receive a CaseView (an opaque id, the question, the frozen fixture), so the system under test structurally cannot see the gold answer, the gold sources, the expected behavior, or even a label-bearing case name.
• Honesty boundary: the bundled hub and judge are deliberate, documented stand-ins, and the scoreboard says so out loud: the hub's answer_correctness is 0.250, it fails every multi-step case, and calibration refuses to trust the bundled judge on the axis where it is weak.

What it looks like

Run the suite against the bundled hub and you get the scoreboard:

$ qaeval run --data data/golden

Cases: 10   judge: on

  deterministic_pass_rate     1.000
  faithfulness_rate           1.000
  answer_correctness          0.250
  citation_precision          1.000
  citation_coverage           1.000
  abstention_recall           1.000
  over_refusal_rate           0.000
  retrieval_recall_at_k       1.000
  rerank_mrr                  1.000
  multi_step_task_success     0.000

Read that scoreboard the way a reviewer would. The bundled hub answers by quoting its best retrieved snippet verbatim, so faithfulness is 1.000 by construction (a verbatim quote is always entailed by its source) and it never hallucinates a citation. The same scoreboard then shows what quoting can't do: answer_correctness lands at 0.250 because a verbatim passage is rarely the crisp gold answer, and multi_step_task_success is 0.000 because stitching one snippet is not reasoning across two. The harness reports the weakness instead of hiding it, which is the entire point.

The demo of the harness doing its job is data/golden_buggy/ plus qaeval/buggy_hub.py, a deliberately flawed wrapper kept out of the reference path: it contaminates a prompt from a stale cache, cites a snippet it never retrieved, and answers a question it should have refused (the fourth case catches the clean hub itself lexically over-matching a time-sensitive trap). The gate catches every one, names the check that fired, and exits non-zero:

$ qaeval baseline --data data/golden --out baseline.json
$ qaeval gate --data data/golden_buggy --adapter qaeval.buggy_hub:make_adapter --baseline baseline.json

Cases: 4   judge: on
  ...
GATE: BLOCK
  [x] Tier 1 FAILED: deterministic checks not 100% green (0.000). Offending cases: ['leakbug-0201', 'overcite-0202', 'greedyanswer-0203', 'trap-timesensitive-0204']
  [x] Tier 2 FAILED: citation_precision dropped -0.500 (baseline 1.000 -> 0.500, tolerance 0.010).
  [x] Tier 2 FAILED: abstention_recall dropped -1.000 (baseline 1.000 -> 0.000, tolerance 0.000).
  [x] Tier 2 FAILED: new golden case(s) do not pass: ['leakbug-0201', 'overcite-0202', 'greedyanswer-0203', 'trap-timesensitive-0204']

$ echo $?
1

And the judge itself has to earn trust before its scores mean anything. The bundled mechanical judge doesn't, and the calibration layer says so:

$ qaeval calibrate --data data/golden --labels data/calibration/human_labels.jsonl

judge-vs-human calibration  (n=16)
  overall exact agreement: 0.750 (min required 0.700, applied per axis) -> RETUNE JUDGE
  faithfulness           exact=1.000 within1=1.000 mae=0.000 (n=4) [ok]
  answer_correctness     exact=0.500 within1=0.875 mae=0.625 (n=8) [BELOW BAR]
  citation_correctness   exact=1.000 within1=1.000 mae=0.000 (n=4) [ok]

$ echo $?
2

The overall number clears the bar; the answer_correctness axis does not (token overlap under-credits verbose-but-right answers), so the verdict is RETUNE JUDGE and the exit code is non-zero. A calibration gate that only looked at the overall average would have trusted this judge. This one doesn't, which is what calibration is for.

How it works

The harness never talks to the system under test directly. Everything goes through one SystemAdapter, and the adapter is handed a CaseView: an opaque id, the question, and the frozen source fixture, nothing else. The gold answer, the gold source ids, the expected behavior, and even the readable case id (names like abstain-0006 are labels too) never cross the SUT boundary, so a right answer can only come from retrieval. The same adapter contract and the same scoring run whether the system behind it is the bundled fake or your production hub; swap in your real system and the gate measures the upgrade on identical terms.

_{Diagram source: assets/architecture.mmd}

Why this exists

A demo shows a system can succeed once. It says nothing about how often it fails, or how it fails when it does. For a QA system that cites sources, the question that matters is concrete: when it answers, is every claim actually supported by the passage it pointed at, and when it can't know, does it say so? This repo answers that with numbers on every dimension that matters, gated before merge, logged over time.

The discipline underneath is fail-closed grounding: the prompt must carry the retrieved passages and must never carry the gold answer (the one string the evaluator knows would fake competence), every citation must resolve to something actually retrieved, and an empty context means abstain without spending a model call. The deterministic layer checks those properties mechanically on every commit. The judge layer scores what mechanics can't (faithfulness, correctness, whether each citation supports its specific claim), and the judge is trusted only while it agrees with hand labels. The gate then refuses any change that makes the numbers worse.

Quickstart

git clone https://github.com/alemadian/qa-eval-harness.git
cd qa-eval-harness
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# 1) run the suite against the bundled hub (deterministic + retrieval + judge)
qaeval run --data data/golden

# 2) save the current scores as the green baseline
qaeval baseline --data data/golden --out baseline.json

# 3) run the CI gate against the buggy split; it BLOCKS and exits non-zero
qaeval gate --data data/golden_buggy --baseline baseline.json; echo $?   # -> 1

# 4) judge-vs-human calibration on the hand-labeled subset
qaeval calibrate --data data/golden --labels data/calibration/human_labels.jsonl

# 5) drift: re-run the frozen split, append to a time series, alarm on drift
qaeval drift --data data/golden --timeseries history.jsonl

# the test suite
pytest -q

How it scores: three layers, cheapest first

A build fails fast before any model-judge spend.

1. Deterministic trust checks (qaeval/deterministic.py), no model, sub-second, on every commit:

   • grounding isolation: the prompt the agent built contains the question and the retrieved snippet text (the actual passage bodies, not just their ids), and the gold answer appears nowhere in it unless a retrieved snippet legitimately carries it. A leak detector for the known oracle string, not a whole-prompt whitelist, and the docstring says exactly that.
   • citation resolvability: every citation resolves to a snippet that was actually retrieved. A citation to a non-retrieved source is a hard fail.
   • citation coverage: no load-bearing answer sentence is left uncited.
   • abstention / fail-closed: on the must-abstain split the agent abstains, and when context is empty it made no model call at all (no spend before grounding).

2. Retrieval evaluation (qaeval/retrieval.py): recall@k, precision@k, MRR, and nDCG against the gold source ids, isolated from generation so a regression points at the right layer.

3. Model-as-judge (qaeval/judges/), versioned rubric, anchored 0/1/2 scale: faithfulness, answer correctness, and citation correctness. The judge must quote the supporting span before it scores. For citation correctness each pairing is re-scored against the hardest available decoy (the most lexically similar non-cited snippet), and the axis is set by the weakest citation, so one plausible-but-wrong pointer can't hide behind several good ones. The judge model id, temperature, seed, and prompt version are pinned in JudgeConfig and recorded in every verdict, because the judge is itself a dependency that drifts.

On top of those: trajectory scoring for multi-hop and aggregation cases (task success requires the correct end state and every reasoning checkpoint, so a lucky right answer through wrong reasoning reads as the failure it is), judge calibration against hand labels, and drift detection over the frozen split with a band per metric. Higher-is-better metrics alarm only on a drop; over_refusal_rate alarms in either direction, because a system going quiet to stay safe is also a failure.

Wiring up your real system

Write one adapter that calls your service over the frozen per-case fixture and returns a SUTOutput. The case your adapter sees carries only the id, the question, and the sources:

from qaeval import SystemAdapter, SUTOutput, CaseView, Citation

class MyHubAdapter(SystemAdapter):
    name = "prod-qa-hub"
    def answer(self, case: CaseView) -> SUTOutput:
        result = my_hub.answer(case.question, corpus=case.sources)  # your call
        return SUTOutput(
            answer=result.text,
            citations=[Citation(snippet_id=c.id, claim=c.claim) for c in result.cites],
            retrieved_ids=result.retrieved_ids,
            abstained=result.abstained,
            model_was_called=result.model_was_called,   # for fail-closed accounting
            prompt_to_model=result.prompt,              # for grounding isolation
            computed_aggregate=result.aggregate,        # for aggregation checkpoints
        )

def make_adapter():            # a zero-arg factory the CLI can import
    return MyHubAdapter()

qaeval gate --adapter mypkg.eval:make_adapter --baseline baseline.json

A real judge is the same move: subclass JudgeClient, call your provider at temperature 0 with a fixed seed, return the JSON verdict, and pass --judge mypkg.eval:make_judge. See examples/real_judge_adapter.py for both skeletons, and examples/github-actions.yml for wiring the gate into your own repo's CI.

The golden dataset

One JSON object per line in data/golden/*.jsonl. Loading is strict: a malformed case, a duplicate id, a gold source id that doesn't exist in the fixture, an answerable case missing its gold answer, and even an unknown field name are all load errors that block the build. Nothing is silently skipped, because a silently-dropped behavior label is exactly the kind of quiet failure the harness exists to prevent.

Each case carries the question, the frozen source snapshot it should retrieve over, the gold answer, the gold supporting source ids, an expected-behavior label (including unanswerable_must_abstain), a difficulty (single-hop, multi-hop, aggregation), an optional adversarial tag, and optional trajectory checkpoints. See qaeval/schema.py for the documented schema.

Two splits matter:

• frozen: never changes; the drift baseline.
• living: grows by one permanent regression case every time a real production miss surfaces, so the set gets harder exactly where the system has already proven it can fail.

The CI gate

qaeval/gating.py implements a layered merge gate, and this repo runs it on itself: .github/workflows/ci.yml tests on Python 3.10 / 3.12 / 3.14, gates every push against the committed ci-baseline.json, and then proves the gate has teeth by gating the buggy split and failing the build if that ever passes.

• Tier 1 (hard): deterministic checks must be 100% green. Any failed check or adapter error blocks the merge, full stop.
• Tier 2 (delta vs baseline): judge metrics are gated on movement against the last green baseline, not an absolute floor. A change may not drop faithfulness, citation precision, abstention recall, or multi-step task success beyond a small tolerance (zero tolerance for abstention recall and task success), and over_refusal_rate may not rise, so a system can't buy safe-looking metrics by refusing answerable questions. No single previously-passing case may regress (a case that starts wrongly abstaining or missing its trajectory counts as regressed), and any new case must pass.

Updating the baseline is a deliberate act (qaeval baseline --data data/golden --out ci-baseline.json) that shows up as a reviewable diff, never a side effect of the run. A gate run with a missing baseline file fails hard rather than silently skipping Tier 2.

Layout

qaeval/
  schema.py        dataclasses + strict loaders for cases and SUT output
  dataset.py       JSONL loading, validation, slicing
  adapter.py       SystemAdapter interface + the bundled reference hub
  buggy_hub.py     the demo saboteur; used only to prove the gate blocks
  deterministic.py layer 1 trust checks (no model)
  retrieval.py     layer 2 recall@k / MRR / nDCG
  judges/
    client.py      JudgeClient interface + offline FakeJudgeClient + JudgeConfig
    model_judge.py layer 3 rubric judging (span quotes, hardest-decoy swap)
    calibration.py judge-vs-human agreement tracker
  trajectory.py    multi-step checkpoint scoring
  runner.py        drives the adapter over a suite, collects all layers
  metrics.py       aggregation -> headline + per-slice + per-case scoreboard
  gating.py        layered CI regression gate
  drift.py         time-series log + drift alarms
  report.py        console + markdown report
  cli.py           run / gate / baseline / drift / calibrate
data/
  golden/          frozen + living golden cases
  golden_buggy/    cases that demonstrate what the gate catches
  calibration/     hand-labeled judge-calibration subset
tests/             pytest suite covering every layer end to end
examples/          real-adapter + real-judge skeletons, CI wiring for your repo

What this is not

• Not a QA system. It's the harness that scores one. The bundled hub exists so everything runs out of the box; the deliberately broken behaviors live in a separate demo adapter, never in the reference path.
• Not a benchmark. The golden set here is a working demo of the format and the failure taxonomy, not a claim about any model's ability. Your golden set is the one that matters, and the loader holds it to the same strict rules.
• Not an oracle. A green gate means no measured regression against your baseline on your cases. It says nothing about failure modes your cases don't cover yet, which is exactly why the living split grows from production misses.
• Not a security boundary. Grounding isolation proves the gold answer wasn't handed to the model; it doesn't defend against adversarial content inside the retrieved passages themselves.

Honest design notes

• The bundled hub answers by quoting, and the scoreboard is read accordingly. Its faithfulness_rate of 1.000 is a property of extraction (verbatim quotes are entailed by definition), not a capability claim; the honest tells are answer_correctness 0.250 and multi_step_task_success 0.000, printed right next to it. A system that paraphrases or reasons will move all three, and the harness will show exactly which way.
• The bundled judge is mechanical (token overlap against the cited span) and the calibration layer correctly refuses to trust it: answer_correctness agreement with hand labels is 0.500, below the 0.700 bar, so calibrate exits non-zero with RETUNE JUDGE. A production judge swaps in behind JudgeClient and has to pass the same bar before its verdicts count.
• The bundled retriever is lexical with a relevance floor. The floor is what makes fail-closed reachable: an out-of-corpus question retrieves nothing, and the hub must abstain without a model call. An embedding retriever drops in behind the same adapter.
• faithfulness_rate counts only fully-faithful answers rather than averaging partial credit, because a partially-faithful answer still contains an unsupported claim.
• abstention_recall and over_refusal_rate are reported as a pair AND both are gated (recall can't drop, refusal can't rise), so a system that goes quiet to protect its faithfulness number is caught, not rewarded.
• An adapter crash is scored as a failed case (Tier 1 block), never a harness crash, so one bad deploy can't take the scoreboard down with it.

Known limitations (tracked, not hidden)

• The demo dataset is small on purpose: 10 golden cases plus 4 buggy ones, sized to demonstrate the format, the slicing, and every check with output you can read in one screen. It is not a statistically meaningful sample; point the harness at your own few hundred cases for that.
• The judge-calibration demo runs on 16 hand labels, enough to exercise the agreement math end to end (and to catch the bundled judge, which it does), not enough to certify a real judge. Certifying one takes a labeled set sized to your risk tolerance, and the per-axis bar applies either way.
• Claim splitting (split_claims) is a sentence-level heuristic. A compound sentence that bundles a supported clause with an unsupported one is judged as a unit; decomposing into atomic claims before scoring is the obvious next hardening and belongs to a model, not a regex.
• The citation-coverage check verifies every load-bearing sentence carries a resolvable citation, not that the cited snippet semantically supports it. That's deliberate layering: semantic support is the judge's citation-correctness axis, so the deterministic layer stays deterministic.
• Drift detection compares against a trailing window in a local JSONL time series. Real deployments should ship that series somewhere durable and page through their own alerting; the notify hook is where that wiring goes.