Skip to content

P2 — Measurement v2: run harness, benches, doc==tool, grade bands #112

Open

Description

@hyperpolymath

Phase P2 of docs/V1-ULTRAPLAN.adoc. The scoreboard's credibility. Closes G6.

WS-2.1 — Run harness (the missing corpus)

  • Versioned runner: 14-probe suite + scenario sets × N repeats, seeds where supported, timestamps → vexometer-trace-v1 bundles
  • Aggregation through ISA_Calculator + Compare_Models (Welch + bootstrap CI already in-tree post-fix: scoring correctness for probe regexes, RCI, comparison, and the README table #109)
  • ≥3 local models (Ollama) + ≥2 API models (explicit consent) each producing a re-derivable public run bundle
  • validate-style recomputation accepts every published bundle

WS-2.2 — Benchmarks ("proven tests and benches")

  • Scoring throughput bench, harness wall-time bench, JSON report-generation bench (join existing Ada + Criterion suites)
  • Regression gate: warn on >10% regressions vs bench history
  • Numbers land in the state file with producing commands

WS-2.3 — Doc == tool for every printed number

WS-2.4 — Grade bands + rank rules (D-2 ruled: hybrid)

  • Fixed bands canonical: A ≤0.15, B ≤0.30, C ≤0.45, D ≤0.60, F >0.60 (0–1 lower-is-better) — confirm cut points against WS-2.1 corpus before freezing
  • Percentile letter line (rank-relative per epoch, labelled relative)
  • docs/GRADES.adoc (or METRICS.adoc section): both systems pure functions, exhaustive boundary tests

Acceptance: every leaderboard figure re-derives in tests; grade maths deterministic at boundaries.

Activity

  1. added
    testingTests, benchmarks, fuzzing, property checks, coverage
    on Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingTests, benchmarks, fuzzing, property checks, coverage

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions