Define and implement the Phase 1 human-relative performance gate

[NeoLorenzo]

Audit Priority: P1
Audit Type: VALIDATE

Problem / Opportunity

PolyVision defers current-interface multi-seed Phase 1 training until the relevant model or candidate, on average, outperforms the canonical human benchmark. That sequencing rule is not yet operational because the repository does not define exactly what "on average outperforms" means in machine-readable terms or provide a reproducible pass/fail evaluation from saved human/model artifacts.

Once complete human first-attempt results and a protocol-conforming model evaluation exist, the gate should be computable without replaying or modifying canonical evidence.

Why This Matters

The human gate currently exists as an intention rather than an executable research rule. Without a frozen comparison definition, the threshold could be interpreted after seeing the results, weakening the validity of the gate and leaving the roadmap ambiguous about whether multi-seed Phase 1 training is actually allowed to proceed.

Evidence

• README.md states that current-interface multi-seed Phase 1 work is deferred until the canonical human-benchmark gate is established and the applicable model/candidate passes it.
• docs/human-benchmark.md describes the 17-map first-completed-attempt sample as the canonical human benchmark, but currently provides only descriptive statistics and an anecdotal n=1 human/model comparison.
• outputs/human_benchmark/summary.json exposes canonical first-attempt map identities and SPT values, while attempt files preserve stable map hashes and environment/interface provenance.
• tools/evaluate_phase1.py emits machine-readable per-map model results and paired policy comparisons, providing a natural model-side artifact once Evaluate the active Phase 1 reference on the human-benchmark pool #13/Add canonical human-benchmark evaluation provenance #14 are complete.
• Superseded issue Superseded: Establish the current Phase 1 human benchmark gate #3 was decomposed specifically because human collection, model evaluation, provenance engineering, and gate definition have independent completion conditions; this issue owns only the final human-relative rule and computation.

Proposed Outcome

Define the statistical comparison rule before interpreting the completed 17-map paired sample, implement a reproducible gate computation over saved immutable human/model artifacts, and emit an explicit machine-readable pass/fail result.

The rule should be simple enough to remain interpretable for a 17-map paired benchmark and must specify the primary metric, aggregation level, comparison direction, tie behavior, and any uncertainty requirement without post-hoc adjustment to observed results.

Definition of Done

• The repository records an explicit machine-readable rule for what "model on average outperforms the human benchmark" means before the final paired result is interpreted.
• The rule specifies the primary metric (final Turn-10 SPT unless deliberately changed with documented rationale), map-level pairing, aggregation level, comparison direction, tie handling, and any uncertainty/pass-margin requirement.
• A paired per-map human-vs-model artifact is produced by joining results using stable canonical map identity/provenance rather than filenames alone.
• The paired artifact reports human and model mean/median Turn-10 SPT, per-map deltas, win/tie/loss counts, and uncertainty appropriate for the 17-map paired sample.
• The artifact reports an explicit machine-readable pass/fail gate result and the exact rule/version used to produce it.
• Gate evaluation consumes saved immutable canonical human results and saved canonical human-benchmark model-evaluation artifacts; it does not replay or mutate canonical attempts.
• Invalid, incomplete, duplicate, mismatched, wrong-interface, or noncanonical input artifacts fail closed rather than silently producing a gate result.
• Regression tests cover at least one known pass fixture, one fail fixture, and malformed/mismatched input cases without consuming live benchmark evidence.
• Documentation explains the gate rule, its statistical interpretation, its dependency on Complete the 17-map canonical human benchmark #12–Add canonical human-benchmark evaluation provenance #14, and the fact that it applies only to the constrained Bardur Turn-10 Phase 1 task.
• Documentation/roadmap can determine mechanically whether multi-seed Phase 1 training is permitted without ad hoc reinterpretation.

Constraints / Non-Goals

This issue does not collect the human sample (#12), run the active reference (#13), or define evaluator canonicality/provenance (#14). It should not modify completed human first attempts or rerun model episodes merely to compute the comparison. It does not claim full-game Polytopia capability.

Validation

Construct synthetic 17-map paired fixtures with stable identities and known deltas. Verify the gate implementation produces the expected pass/fail result, rejects incomplete or mismatched evidence, and records enough rule/version metadata that the decision is reproducible from the saved input artifacts alone.