Fail closed when canonical baselines see an incompatible legal-action feature schema

[NeoLorenzo]

Audit Priority: P2
Audit Type: FIX

Problem / Opportunity

The canonical visible_greedy baseline interprets legal_action_features_padded through a hard-coded numeric FEATURE index table in tools/phase1_eval_core.py, but the evaluator does not explicitly bind that table to the environment's legal-action feature version or ordered feature names before using it.

The active wrapper currently exposes LEGAL_ACTION_FEATURE_VERSION = "v1_4_parity_spatial_and_cost" with a 47-field ordered schema, so the baseline is correct today. However, recent Phase 1 development has changed this feature contract repeatedly. A future insertion, reorder, rename, or semantic change could make the canonical baseline silently read the wrong columns while still receiving a tensor of compatible shape.

Why This Matters

visible_greedy is part of the canonical 3,000-episode validation/test protocol and its paired comparison against PPO is reported as scientific evidence. Silent schema drift would not necessarily crash evaluation; it could instead change baseline behavior and invalidate longitudinal or human-visible comparisons while producing apparently valid result artifacts.

PolyVision already treats model/environment interface mismatches as fail-closed through checkpoint metadata. The policy-visible scripted baseline should receive equivalent protection for the feature semantics it depends on.

Evidence

• tools/phase1_eval_core.py defines a module-level FEATURE dictionary with fixed numeric positions such as is_move: 0, is_capture: 13, is_train: 14, is_resource: 16, and is_level_up: 17.
• VisibleGreedyPolicy.choose_action() directly indexes the runtime legal_action_features_padded tensor with those numeric constants.
• pol_env/Tribes/py/register_env.py separately defines the authoritative ordered LEGAL_ACTION_FEATURE_NAMES tuple and LEGAL_ACTION_FEATURE_VERSION = "v1_4_parity_spatial_and_cost".
• tools/evaluate_phase1.py validates checkpoint compatibility for PPO models, but baseline construction through make_policy() does not establish an explicit compatibility check between VisibleGreedyPolicy and the runtime legal-action feature schema.
• docs/evaluation.md treats visible greedy as a canonical policy-visible baseline and includes it in all current reference comparisons.
• Recent PARITY001/PARITY002 development changed the policy-visible observation/action-feature contract substantially, making future schema evolution a realistic rather than hypothetical maintenance path.
• Open issues Establish a genuinely pristine Phase 1 test benchmark #2, Superseded: Establish the current Phase 1 human benchmark gate #3, Add resumable full-state checkpoints for long PPO training runs #4, and Make canonical Phase 1 evaluations resumable without changing the experiment schedule #6 cover benchmark evidence, the human gate, training resume, and evaluation resume respectively; none owns baseline feature-schema compatibility.

Proposed Outcome

Make the canonical baseline declare and verify the exact legal-action feature semantics it consumes before an evaluation proceeds. Prefer binding by authoritative feature names/version rather than relying solely on tensor width or duplicated numeric constants. If the runtime schema is incompatible, evaluation should fail before any canonical results are produced.

Definition of Done

• The visible-greedy baseline has an explicit contract for the legal-action feature names/semantics it requires.
• Canonical evaluation validates that contract against the runtime environment before scheduled episodes begin.
• A reordered, missing, renamed, or otherwise incompatible required feature causes a clear fail-closed error rather than silently changing baseline behavior.
• Compatible additive schema changes are either supported deliberately by name-based lookup or rejected explicitly according to a documented rule.
• Regression tests cover the active v1_4_parity_spatial_and_cost schema and at least one incompatible-schema case.
• The maintained CI gate runs the cheap regression coverage without launching canonical evaluation or consuming held-out evidence.
• Existing v5 canonical visible-greedy behavior remains unchanged under the current 47-feature contract.

Constraints / Non-Goals

This issue does not redesign the visible-greedy heuristic, change its ranking logic, or add a stronger baseline. It is limited to protecting the existing baseline from silent feature-schema drift. It also does not replace the broader evaluation-resume work in #6.

Validation

Run the baseline contract tests against the current wrapper schema and verify they pass. Then simulate a feature reorder/removal/version mismatch and verify evaluation refuses to start before creating a canonical result directory.