feat(report): emit a machine-readable roster summary (rates, gap, ranks, Kendall tau)

[dchaudhari7177]

Closes #37.

What it produces

Run against the three committed roster results:

leakgauge report results/*.json --summary-json roster.json --summary-md roster.md

roster.md:

model	n cases	hijack-ASR [95% CI]	leakage-verified ASR [95% CI]	utility-under-attack [95% CI]	gap	h-rank	l-rank
openai:gpt-4o	37	0.081 [0.027, 0.146]	0.077 [0.027, 0.135]	0.658 [0.520, 0.788]	+0.005	1	1
openrouter:meta-llama/llama-3.3-70b-instruct	37	0.023 [0.009, 0.038]	0.018 [0.005, 0.034]	0.435 [0.300, 0.572]	+0.005	2	2
openai:gpt-4o-mini	37	0.005 [0.000, 0.014]	0.005 [0.000, 0.014]	0.784 [0.658, 0.896]	+0.000	3	3

Kendall τ (hijack-rate vs leakage-rate): 1.000 — 0 of 3 model(s) change rank when scored on verified leakage.

roster.json carries the same content unrounded, keyed for machine consumption:

{
  "schema_version": 1,
  "n_models": 3,
  "kendall_tau": 1.0,
  "rank_crossings": 0,
  "models": [
    {
      "model": "openai:gpt-4o",
      "n_cases": 37,
      "hijack_asr": { "point": 0.081…, "lo": 0.027…, "hi": 0.146… },
      "leakage_asr": { "point": 0.076…, "lo": 0.027…, "hi": 0.135… },
      "utility_under_attack": { "point": 0.657…, "lo": 0.520…, "hi": 0.788… },
      "gap": 0.004505,
      "hijack_rank": 1,
      "leakage_rank": 1
    }
  ]
}

(That table is real output from the committed results/, not an illustration.)

Implementation

• suite.roster_summary(summaries, reorder) builds the object; suite.format_roster_markdown(roster) renders the same object as the table, so the two artifacts cannot drift.
• leakgauge report gains --summary-json PATH and --summary-md PATH. Both are optional and create their parent directory; with neither flag the command behaves exactly as before.
• Rows are sorted worst-first by hijack rank, matching the existing reorder table.

No fabricated rows

• Rates are passed through as loaded, {point, lo, hi} intact — nothing is recomputed, so the summary cannot disagree with the source files.
• utility_under_attack is None unless the run supplied a utility check, and stays None through the JSON round trip / renders as — in markdown. It never becomes 0.0: "not measured" and "measured as zero" are different claims, and on a robustness metric conflating them flatters the model.
• kendall_tau, rank_crossings and both rank columns are null for a single-model roster, since a reorder is undefined below two models.
• gap is the only value computed here, and it is rounded to 6 places. Unrounded, float subtraction gives 0.0045045045045045 on one run and 0.004504504504504505 on another for identical inputs, which would show up as a spurious diff — and diffability is the point of the artifact.

Reusing _crossings

The issue asks to reuse the existing _crossings logic. Rather than import a private name across modules, it moves from leaderboard.py to scoring.py as the public crossings(), next to the RankReorder it operates on; leaderboard.py imports it from there and its single call site is updated. No behaviour change, and suite.py gets it without depending on the renderer.

Tests

Six in tests/test_suite.py. _fake_summary grew an optional utility argument (default None, so existing callers are untouched and still exercise the absent-metric path).

• test_roster_summary_matches_the_computed_report — three models where A hijacks most but leaks least, so the orderings disagree and the gap is positive for one model and negative for another. Asserts every rate dict is identical to the input aggregate, n_cases matches, gap equals the difference, both ranks match reorder, kendall_tau matches, rank_crossings == crossings(reorder) == 2, and row order equals reorder.models_by_hijack.
• test_roster_markdown_matches_the_roster_summary — exactly one row per model (no invented rows), CIs present in the row rather than just the point estimate, signed gap, τ and the crossing sentence.
• test_roster_summary_leaves_an_unmeasured_metric_blank_not_zero — absent utility survives a json.dumps/loads round trip as null, the markdown shows —, and 0.000 appears nowhere in the table.
• test_roster_summary_single_model_has_no_ranks — τ, crossings and ranks all null; markdown says it needs ≥2 summaries.
• test_report_writes_the_roster_summary_artifacts — end to end through main(["report", …]), including creating a missing parent directory; the written JSON is compared against roster_summary(...) recomputed from the same files, and the markdown against format_roster_markdown(...).
• test_report_without_summary_flags_writes_nothing — the no-flags path is unchanged and leaves no files behind.

Docs

README gains a short leakgauge report flag block under the CLI section explaining what the two artifacts contain and the null/— rule.

Checks

ruff check ., mypy src, pytest -q all pass (414 passed, 4 xfailed).

ruff format --check src tests is clean. ruff format --check . flags CONTRIBUTING.md and docs/CONTRACTS.md on a clean tree here only because my local ruff 0.16.1 formats fenced code blocks in Markdown and the uv.lock version does not; neither file is touched by this PR.