Preregister iter225: cross-model generalization (gpt-5.5 on iter223's frozen cohort)#23
Merged
Merged
Conversation
…3-target cohort Single-variable replication of iter223: same frozen cohort (sha 9f3078c3), same witnessing generator (gpt-5.6-terra) and both blind judges, changing only the solver model to gpt-5.5. Tests whether certified-yet-wrong is a property of one model or of the SWE-bench certification process. Reported standalone vs iter223's 4/29; not pooled into 5/68. Includes lean solver/scenario scripts (reusing the frozen iter200 solve helpers), the safety-aware validator with a machine-enforced gpt-5.5 solver-model invariant, the sharded execution workflow, and CI wiring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-registers iter225, the mission's cross-model generalization test. It re-solves iter223's identical frozen 53-target cohort (sha
9f3078c3) with a different solver model —gpt-5.5in place ofgpt-5.6-terra— holding the witnessing generator, official-harness certification, and both blind judges byte-identical. Only the solver changes.Question: is a certified-yet-wrong patch a property of one model, or of the SWE-bench certification process itself? The single-model pool (
5/68) cannot answer this.Design guardrails
validate_iter225_scenario_safety.py) machine-enforces the single-changed-variable invariant: it fails closed unless every solve row recordssolver_model = gpt-5.5.gpt-5.6-terraso the adjudication instrument is identical to iter223's.4/29; not pooled into5/68(non-disjoint, different model). Adding it to the pool is a pre-registered falsifier.This PR contains only the pre-registration, frozen cohort, pipeline code, execution workflow, and CI wiring — no solve, scenario, spec, or execution output. Data and RESULT follow in a separate PR after execution. Full CI closure (266 guards) passes locally under python3.11.