Regression-test the canonical Phase 1 evaluation protocol and statistics

[NeoLorenzo]

Audit Priority: P1
Audit Type: VALIDATE

Problem / Opportunity

PolyVision’s canonical Phase 1 evidence is produced by tools/evaluate_phase1.py and tools/phase1_eval_core.py, but the maintained CI gate does not currently exercise the evaluator’s protocol logic. The code that constructs per-policy schedules and seeds, aggregates stochastic replicates to map-level headline statistics, computes paired comparisons, validates map identities, and serializes canonical artifacts is therefore trusted largely through prior successful runs rather than automated regression coverage.

Why This Matters

The evaluator is part of the project’s scientific measurement system. A subtle regression in schedule construction, replicate handling, pairing, confidence-interval input level, or output provenance could change reported Phase 1 conclusions while the environment/parity CI still passes. Because current reference claims and the future human-relative/multi-seed gates depend on these artifacts, evaluator correctness deserves a cheap deterministic test layer independent of full JVM-backed 3,000-episode runs.

Evidence

• tools/evaluate_phase1.py is documented as the maintained canonical batch evaluator and produces the reference validation/test artifacts used throughout docs/evaluation.md.
• tools/phase1_eval_core.py owns load_verified_pool, build_schedule, validate_schedule, aggregate_results, paired_stats, and serialization helpers that define important parts of the evaluation protocol.
• Stochastic policies receive five replicates per map in the canonical suite, while deterministic policies receive one; headline statistics are intentionally computed from per-map means rather than treating all stochastic episodes as independent observations.
• Paired comparisons depend on joining policies by canonical map identity and comparing map-level results.
• .github/workflows/core-contract-ci.yml runs tools/run_core_ci_checks.py.
• tools/run_core_ci_checks.py currently runs only test_environment_contract.py, test_parity_001_city_state.py, and test_parity_002_human_information_parity.py; it does not include evaluator-core regression tests.
• Open issue Make canonical Phase 1 evaluations resumable without changing the experiment schedule #6 covers resumable evaluation state and requires a resume-specific deterministic integration test; it does not validate the existing evaluator protocol as a whole.
• Open issue Fail closed when canonical baselines see an incompatible legal-action feature schema #9 covers fail-closed compatibility for the visible-greedy legal-feature schema; it does not cover scheduling, statistics, pairing, or provenance semantics.

Proposed Outcome

Add a deterministic, cheap regression suite for the canonical evaluator core and include it in the maintained CI gate. The tests should exercise the protocol-defining pure-Python behavior with synthetic map/episode fixtures so they do not consume held-out benchmark evidence or require a full canonical JVM-backed run.

Definition of Done

• Tests verify canonical schedule cardinality and replicate rules: one episode per map for deterministic policies and the configured repeat count for stochastic policies.
• Tests verify stable episode/policy seeding and that equivalent protocol inputs reproduce the same schedule identities.
• Tests verify malformed schedules are rejected, including missing replicates, unexpected map identities, and duplicate/incorrect schedule membership where applicable.
• Tests verify stochastic headline statistics are computed from per-map replicate means rather than treating all episode replicates as independent map observations.
• Tests verify paired comparisons join by canonical map identity and produce correct win/tie/loss counts and deltas on a known synthetic fixture.
• Tests verify pool loading rejects hash mismatches, duplicate canonical identities, and train-identity leakage into evaluation pools using isolated fixtures rather than the real held-out datasets.
• Tests cover the required serialized episode/artifact contract sufficiently to catch missing protocol-critical fields or incompatible output assumptions.
• The evaluator-core regression suite runs in tools/run_core_ci_checks.py / the existing GitHub Actions gate without executing a full canonical evaluation or consuming test/human benchmark evidence.
• Existing canonical evaluation behavior and saved reference artifacts remain semantically unchanged.

Constraints / Non-Goals

This issue does not make evaluations resumable (#6), change the statistical protocol, redesign baselines, or solve baseline feature-schema compatibility (#9). It should validate the currently documented protocol rather than introduce a new one. It must not access or consume the pristine/final-evidence benchmark proposed in #2.

Validation

Run the maintained core CI gate locally and in GitHub Actions. The evaluator tests should complete using synthetic fixtures and deterministic in-memory episode records, with no canonical evaluation output directory or held-out benchmark access required.