Define evaluation contract v2 and acceptance milestones

CONTRIBUTING.md

@@ -0,0 +1,24 @@
+# Contributing
+
+Start with an issue describing the user problem, expected behavior, evidence and the smallest reviewable change. Never include credentials, private cases, employer material or client data in issues, fixtures or logs.
+
+## Local checks
+
+```sh
+uv sync --extra dev
+uv run pytest
+```
+
+Use a branch and submit a pull request explaining what changed, why, tests run and known limitations. Include a small original or rights-cleared fixture for behavioral changes. Document breaking schema, scoring or dataset changes and version them; never silently change historical results.
+
+## Review checklist
+
+| Change | Required evidence |
+|---|---|
+| Dataset/source | Public source, revision/hash, license, transformation, split, lineage and reconstruction check |
+| Task/scorer | Versioned contract, label rubric, baseline, denominator, invalid-output policy and uncertainty |
+| Runner/API | Failure and retry behavior, secret redaction, reproducible configuration |
+| Arena access/publication | Tenant isolation, role checks, exact-result approvals and managed-run provenance |
+| Documentation | Copyable commands and an explicit distinction between implemented and planned behavior |
+
+Report data or scoring problems with source/case IDs and a reproducible explanation using public material. Keep proposed corrections separate from immutable historical releases. Report security issues privately using the repository's security policy where available; never post secrets publicly.

README.md

@@ -1,6 +1,8 @@
 # Najd Benchmark
 
-Source repository for task definitions, scoring contracts, baselines, and reproducible reports. The benchmark consumes versioned, approved datasets from `najdresearch/datasets`; the Arena application handles submissions, execution, review, and presentation.
+[Evaluation contract v2](docs/evaluation-contract-v2.md) · [Contributing](CONTRIBUTING.md)
+
+Source repository for task definitions, scoring contracts, baselines, and reproducible reports. The benchmark consumes versioned, approved datasets from `najdresearch/datasets`; the Arena application handles submissions, managed worker orchestration, review, and presentation. Shared endpoint execution belongs in the benchmark package under the v2 contract.
 
 The [platform contract](docs/platform-contract.md) records the first text score, repository boundaries, submission policy, and release gates. Classification, tool decisions, computer use, embeddings, reranking, OCR, STT, and TTS remain separate scorecards with their own scoring methods.
 

docs/evaluation-contract-v2.md

@@ -0,0 +1,65 @@
+# Evaluation contract v2 — implementation target
+
+This contract records the agreed destination. It does not claim that the endpoint runner or Arena integration is implemented. It supersedes repository ownership and publication policy in the v1 platform contract; historical scores retain their original protocol.
+
+## Repository boundaries
+
+```mermaid
+flowchart LR
+  D[Datasets: sources and generation] --> H[Versioned Hugging Face release]
+  H --> B[Benchmark: task execution and scoring]
+  B --> C[Local CLI reports]
+  B --> W[Arena managed workers]
+  W --> P[Private report and audit]
+  P --> A[Publication approvals]
+  A --> U[Public Arena results]
+```
+
+| Owner | Responsibility |
+|---|---|
+| datasets | Collection, cleaning, normalization, synthesis, provenance and pinned releases |
+| benchmark | Versioned task packs, endpoint runner, scoring saved responses, uncertainty and reports |
+| Arena | Identity, quota reservation, endpoints, orchestration, evidence storage, approvals and public views |
+
+Local and remote candidate models use OpenAI-compatible endpoints. Keep endpoint credentials out of reports and logs. Local CLI results are useful for development but cannot be uploaded as Arena-publishable results. Only website-managed runs may enter the Arena publication workflow. Arena workers must import a pinned benchmark distribution and record its artifact hash; they must not fork scoring logic.
+
+## Required immutable records
+
+| Record | Required fields |
+|---|---|
+| Task pack | ID/version, dataset revision and hashes, task IO schema, case/split manifest, metric/rubric version, baseline, stop rule |
+| Run | ID, origin, organization, task pack digest, benchmark version/hash, model claimed ID, endpoint identity, prompt/settings, harness version, budgets, start/end times |
+| Case attempt | Run and case IDs, attempt number, request configuration hash, response artifact hash, timing, usage source, status, error class |
+| Result revision | Run ID, immutable revision/hash, expected/completed/failed unique case counts, metric numerator/denominator, scorer/judge settings, uncertainty, evidence references |
+| Approval | Exact result hash, actor ID and role, organization or Najd scope, decision/time/reason |
+| Publication | Result hash, independent or organization-submitted basis, required approvals, published_at, withdrawal/correction history |
+
+Server-issued origin is authoritative. A client-provided `arena_managed` field never establishes eligibility. All requested cases remain accounted for, including failures and invalid outputs. Retries retain evidence but cannot inflate the unique-case denominator. Regrading creates a new result revision and invalidates old publication approvals for the new revision.
+
+## Scorecards that explain the numbers
+
+| Metric | Meaning | Required companion |
+|---|---|---|
+| Task success | Successful cases / all assigned eligible cases | Rubric, failures, invalid outputs and confidence interval |
+| Classification | Accuracy and per-label precision/recall; macro F1 where appropriate | Label counts, confusion matrix, abstention handling |
+| Support policy compliance | Cases satisfying policy rubric / assigned cases | Unsupported commitments and escalation failures |
+| Grounding | Claims supported by provided evidence | Citation/claim annotation policy and missing evidence |
+| Tool decisions | Correct call, clarification or abstention | Argument validity and unsafe-action rate |
+| Latency | End-to-end elapsed time distribution | p50/p95, sample counts, concurrency, timeout and retry policy |
+| Cost | Measured or estimated spend | Token accounting source, price date, missing-usage rate |
+
+Report unavailable metrics as unavailable, not zero. Do not blend different prompts, thinking levels, harnesses or datasets. Model-only and Pi coding-agent results use separate views. A historical acceptable-output rate is not automatically accuracy or policy compliance.
+
+Every report answers: what problem was tested, what improved, how much uncertainty remains, which failures matter, and what change to try next. Example: “Version B reduced unsupported refund promises from 8/100 to 3/100 on the same cases; investigate the five changed cases and paired uncertainty before concluding the prompt generalizes.” This is a fictional reporting example, not a Najd result.
+
+## Delivery sequence and gates
+
+| Milestone | Evidence | Dependency / stop rule |
+|---|---|---|
+| 1. Public-source audit and contract | All historical rows inventoried; current gaps visible | Do not call declared URLs a successful rebuild |
+| 2. Support task pack and runner | Fake-server integration tests for successful, invalid, timeout and retry responses; deterministic saved-response scoring | Reviewed task contract and baseline first |
+| 3. Worker extraction | Same frozen inputs produce identical CLI and managed-run scores | Pin package; remove duplicate Arena scoring only after parity |
+| 4. Managed pilot | Two internal support versions run from platform with complete private evidence | Quotas, isolation and queue recovery tested |
+| 5. Publication | Role, result-hash approval and withdrawal tests | No publication from CLI/imported origin |
+
+For the first support pilot, freeze scenarios, policies and rubric before comparing versions. Use scenario-level splits and paired analysis. Stop if evaluation cases influenced prompt tuning or reference labels are unresolved.

docs/platform-contract.md

@@ -1,3 +1,5 @@
+> Repository ownership and publication policy are superseded by [evaluation contract v2](evaluation-contract-v2.md). Historical scoring definitions below remain protocol-specific.
+
 # Najd benchmarking platform contract
 
 ## Outcome and current state