fix: calibrate FASB default to avoid timeout collapse

[Ian747-tw]

Summary

• Adds calibration configs for the FASB timeout-collapse investigation.
• Fixes MixedFailureSampler so canonical failure-buffer replay only emits seeds inside the active MetaDrive training range.
• Documents the original Axis 1 FASB collapse and the failed 100k-dev-selected v2 candidate.

Original Axis 1 FASB collapse

• Original FASB-PPO final heldout: success_rate=0.00, timeout_rate=1.00, route_completion_mean=0.0143.
• It achieved collision_rate=0.00, offroad_rate=0.00, episode_cost_mean=0.00 by freezing, not by preserving useful driving progress.
• Fixed-budget also collapsed into timeout; naive retained progress and success.

Dev calibration candidates

Calibration used dev only: start_seed=4500, num_scenarios=100, horizon=500, traffic_density=0.1. The final heldout range was not used to select v2.

• fasb_soft_010: success=0.43, timeout=0.57, route=0.6881, safety_eff=-0.585
• fasb_soft_030: success=0.47, timeout=0.53, route=0.7033, safety_eff=-0.525
• fasb_medium_030: success=0.44, timeout=0.56, route=0.6880, safety_eff=-0.470
• fasb_medium_050: rejected, success=0.00, timeout=1.00, route=0.0141

The 100k dev screen selected fasb_medium_030, but it did not beat naive on dev.

Final heldout v2 result

• The 300k FASB v2 candidate failed final heldout: success_rate=0.00, collision_rate=0.37, offroad_rate=0.40, timeout_rate=1.00, route_completion_mean=0.0121, episode_cost_mean=596.30, safety_efficiency_score=-1.27.
• A 300k dev diagnostic also collapsed: success_rate=0.00, timeout_rate=1.00, route_completion_mean=0.0122.
• Therefore this PR does not claim a fixed accepted FASB default; it preserves the failed v2 candidate config for reproducibility and documents the blocker.

Does v2 beat baselines?

• v2 does not beat naive.
• v2 does not meaningfully beat fixed-budget; both collapse, and v2 has high final collision/offroad/cost.
• Original FASB remains a pure hesitation-collapse reference; v2 is a timeout plus collision/offroad/cost collapse after 300k.

Source checks run

python -m compileall fasb tests scripts examples -q
python -m pytest tests/test_component_validation.py tests/test_failure_buffer.py tests/test_metrics.py tests/test_training_stability_patch.py tests/test_plugin_loading.py tests/test_run_dir.py tests/test_research_plugins.py tests/test_aggregate_results.py tests/test_base_checkpoint_quality.py tests/test_failure_buffer_quality.py tests/test_research_v1_ready.py tests/test_research_v1_artifact_scripts.py -q
python scripts/validate_components.py --config configs/research_v1/axis1_fasb_v2_final.yaml

Result: 54 tests passed; component validation passed.

[Ian747-tw]

@codex review this PR. Focus on whether the FASB v2 default avoids timeout collapse without overfitting to final heldout, whether calibration uses dev seeds only, and whether Axis 2/3 remain valid ablations around the calibrated default.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee9f3c49e6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use dev seed window in calibration configs

These new calibration configs are documented as the basis for a "dev-only" 100k selection (start_seed=4500, num_scenarios=100), but this file still trains on the main fine-tune slice (start_seed=2000, num_scenarios=500). Re-running calibration from these checked-in configs will therefore calibrate on the training distribution instead of the claimed dev window, which makes the anti-overfitting claim non-reproducible and can invalidate Axis 2/3 comparisons that are supposed to be anchored to dev-screened defaults.

Useful? React with 👍 / 👎.

[Ian747-tw]

Closing as superseded by PR #8. The v2 calibration in this PR looked acceptable at 100k dev but collapsed at 300k, so it should remain diagnostic only and not become the default. PR #8 replaces it with a 300k dev-calibrated stable FASB protocol and fair stable-protocol Axis 1 baselines.