Phase P2 of `docs/V1-ULTRAPLAN.adoc`. The scoreboard's credibility. Closes G6. ## WS-2.1 — Run harness (the missing corpus) - [ ] Versioned runner: 14-probe suite + scenario sets × N repeats, seeds where supported, timestamps → `vexometer-trace-v1` bundles - [ ] Aggregation through `ISA_Calculator` + `Compare_Models` (Welch + bootstrap CI already in-tree post-#109) - [ ] ≥3 local models (Ollama) + ≥2 API models (explicit consent) each producing a re-derivable public run bundle - [ ] `validate`-style recomputation accepts every published bundle ## WS-2.2 — Benchmarks ("proven tests and benches") - [ ] Scoring throughput bench, harness wall-time bench, JSON report-generation bench (join existing Ada + Criterion suites) - [ ] Regression gate: warn on >10% regressions vs bench history - [ ] Numbers land in the state file with producing commands ## WS-2.3 — Doc == tool for every printed number - [ ] Generalise the #109 table test: one module recomputes every figure the site renders (ISA, grades, deltas) from raw aggregates - [ ] Editing a rendered number without recomputation fails `just test` ## WS-2.4 — Grade bands + rank rules (D-2 ruled: hybrid) - [ ] Fixed bands canonical: A ≤0.15, B ≤0.30, C ≤0.45, D ≤0.60, F >0.60 (0–1 lower-is-better) — confirm cut points against WS-2.1 corpus before freezing - [ ] Percentile letter line (rank-relative per epoch, labelled relative) - [ ] `docs/GRADES.adoc` (or METRICS.adoc section): both systems pure functions, exhaustive boundary tests **Acceptance:** every leaderboard figure re-derives in tests; grade maths deterministic at boundaries.
Activity