Audit Priority: P1
Audit Type: VALIDATE
Problem / Opportunity
The canonical Phase 1 human benchmark is incomplete. The committed registry currently contains only 1 completed canonical first attempt across the 17 reserved human-benchmark maps, leaving 16 maps without a canonical human score.
The canonical statistic for each map is the first completed human attempt. Completed canonical attempts are immutable; aborted/error attempts remain in history but do not count as the completed first-attempt score.
Why This Matters
Current-interface multi-seed Phase 1 work is explicitly deferred until the human-relative gate is established and passed. That gate cannot be evaluated on a complete paired sample until all 17 reserved maps have canonical human first-attempt evidence.
Evidence
outputs/human_benchmark/summary.json records benchmark_map_count: 17, completed_first_attempt_maps: 1, and remaining_first_attempt_maps: 16.tools/human_benchmark.py selects only maps without a completed first human attempt during ordinary use and marks the first completed human result per map as canonical.tools/human_benchmark.py refuses to overwrite completed attempt records and keeps aborted/error attempts without treating them as completed evidence.docs/human-benchmark.md reports the benchmark as 1/17 complete and describes first completed attempts as the canonical statistic.- README roadmap text defers current-interface multi-seed Phase 1 work until the human benchmark gate is established and passed.
Proposed Outcome
Complete the remaining canonical first attempts using the existing human-benchmark workflow and preserve the resulting evidence without changing the benchmark protocol.
This issue is intentionally limited to human data collection. It does not own model evaluation (#13), evaluator provenance/classification (#14), or the definition of the human-relative pass/fail gate (#15).
Definition of Done
Constraints / Non-Goals
Do not change the map pool, first-attempt rule, policy-visible interface, episode horizon, or evidence immutability semantics while completing the sample. This issue does not define the statistical human-vs-model gate.
Existing Workflow / Validation
Run from the repository root:
python tools/human_benchmark.py
After the final completion, run:
python tools/human_benchmark.py --summary
Verify that the summary reports 17 completed first-attempt maps, that each map has exactly one canonical first-completed result, and that all attempt files remain readable under the current benchmark schema.
Audit Priority: P1
Audit Type: VALIDATE
Problem / Opportunity
The canonical Phase 1 human benchmark is incomplete. The committed registry currently contains only 1 completed canonical first attempt across the 17 reserved human-benchmark maps, leaving 16 maps without a canonical human score.
The canonical statistic for each map is the first completed human attempt. Completed canonical attempts are immutable; aborted/error attempts remain in history but do not count as the completed first-attempt score.
Why This Matters
Current-interface multi-seed Phase 1 work is explicitly deferred until the human-relative gate is established and passed. That gate cannot be evaluated on a complete paired sample until all 17 reserved maps have canonical human first-attempt evidence.
Evidence
outputs/human_benchmark/summary.jsonrecordsbenchmark_map_count: 17,completed_first_attempt_maps: 1, andremaining_first_attempt_maps: 16.tools/human_benchmark.pyselects only maps without a completed first human attempt during ordinary use and marks the first completed human result per map as canonical.tools/human_benchmark.pyrefuses to overwrite completed attempt records and keeps aborted/error attempts without treating them as completed evidence.docs/human-benchmark.mdreports the benchmark as 1/17 complete and describes first completed attempts as the canonical statistic.Proposed Outcome
Complete the remaining canonical first attempts using the existing human-benchmark workflow and preserve the resulting evidence without changing the benchmark protocol.
This issue is intentionally limited to human data collection. It does not own model evaluation (#13), evaluator provenance/classification (#14), or the definition of the human-relative pass/fail gate (#15).
Definition of Done
outputs/human_benchmark/summary.jsonreports 17/17 maps complete and zero remaining first-attempt maps.Constraints / Non-Goals
Do not change the map pool, first-attempt rule, policy-visible interface, episode horizon, or evidence immutability semantics while completing the sample. This issue does not define the statistical human-vs-model gate.
Existing Workflow / Validation
Run from the repository root:
python tools/human_benchmark.pyAfter the final completion, run:
Verify that the summary reports 17 completed first-attempt maps, that each map has exactly one canonical first-completed result, and that all attempt files remain readable under the current benchmark schema.