Preregister iter225: cross-model generalization (gpt-5.5 on iter223's frozen cohort)

.github/workflows/ci.yml

@@ -207,6 +207,9 @@ jobs:
       - name: Iter224 scale-up scenario evidence guard
         run: python3 scripts/validate_iter224_scenario_safety.py
 
+      - name: Iter225 cross-model scenario evidence guard
+        run: python3 scripts/validate_iter225_scenario_safety.py
+
       - name: Target survey guard
         run: python3 scripts/validate_target_survey.py
 

.github/workflows/iter225-execute.yml

@@ -0,0 +1,44 @@
+name: iter225-execute
+on:
+  workflow_dispatch:
+permissions:
+  contents: read
+jobs:
+  execute:
+    name: iter225 certify + witness shard ${{ matrix.shard }}/8 (x86, native)
+    strategy:
+      fail-fast: false
+      matrix:
+        shard: [0, 1, 2, 3, 4, 5, 6, 7]
+    runs-on: ubuntu-24.04
+    timeout-minutes: 350
+    env:
+      TELOS_NAT_EXP: iter225_cross_model_generalization
+      TELOS_NAT_SHARD_COUNT: 8
+      TELOS_NAT_SHARD_INDEX: ${{ matrix.shard }}
+    steps:
+      - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
+      - name: Free disk
+        run: sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
+      - name: Validate corrected safety-aware scenario evidence
+        run: python3 scripts/validate_iter225_scenario_safety.py
+      - name: Run iter225 execution
+        run: |
+          chmod +x scripts/ci_iter200_execute.sh
+          bash scripts/ci_iter200_execute.sh
+      - name: Upload verified shard evidence
+        if: success()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: iter225-execution-run-${{ github.run_id }}-attempt-${{ github.run_attempt }}-shard-${{ matrix.shard }}-of-8
+          path: experiments/iter225_cross_model_generalization/proof/raw/execution/
+          if-no-files-found: error
+          retention-days: 14
+      - name: Upload partial shard evidence for debugging only
+        if: failure()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: iter225-execution-debug-${{ github.run_id }}-attempt-${{ github.run_attempt }}-shard-${{ matrix.shard }}-of-8
+          path: experiments/iter225_cross_model_generalization/proof/raw/execution/
+          if-no-files-found: ignore
+          retention-days: 14

README.md

@@ -49,7 +49,14 @@ Its preserved predecessor is
 which keeps the failed iter219 branch and PR `#13` unchanged, root-causes a required-phrase scanner that a
 Markdown line wrap could defeat, and replaces hand-listed local verification with a closure runner derived
 from the CI workflow itself.
-The active scientific result is
+The active pre-registration is
+[iter225 cross-model generalization](experiments/iter225_cross_model_generalization/HYPOTHESIS.md): a
+single-variable replication that re-solves iter223's identical frozen `53`-target cohort with a **different**
+solver model (`gpt-5.5` in place of `gpt-5.6-terra`), holding the witnessing generator, certification, and
+both blind judges byte-identical. It asks the sharpest open question the single-model pool cannot answer — is
+a certified-yet-wrong patch a property of one model, or of the SWE-bench certification process itself? By
+design it is reported as a standalone comparison against iter223's `4/29` and is **not** pooled into `5/68`.
+The latest published result is
 [iter224 natural-rate scale-up](experiments/iter224_natural_rate_scale_n/HYPOTHESIS.md), a **null published at
 full weight**: on a fresh disjoint `26`-target cohort (`13` django, `13` sympy — the eligibility filter has
 exhausted diverse fresh targets, disclosed), `25` patches produced `15` certifications and `0` certified-and-
@@ -259,7 +266,7 @@ turning infrastructure or admission failures into scientific outcomes.
 
 ```mermaid
 flowchart LR
- I203["203 infra null"]-->I204["204 null"]-->I205["205 null"]-->I206["206 null"]-->I207["207 sealed"]-->I208["208 CI failed"]-->I209["209 PR failed"]-->I210["210 merged"]-->I211["211 TCP-1 2/11"]-->I213["213 CI failed"]-->I214["214 Wilson fix"]-->I219["219 temporal yield<br/>NULL · no signal"]-->I220["220 CI recovery"]-->I221["221 platform-indep"]-->I222["222 admission<br/>2/11 → 5/11"]-->I223["223 natural rate<br/>4/29 · pooled 5/53"]-->I224["224 scale-up<br/>fresh cohort"]
+ I203["203 infra null"]-->I204["204 null"]-->I205["205 null"]-->I206["206 null"]-->I207["207 sealed"]-->I208["208 CI failed"]-->I209["209 PR failed"]-->I210["210 merged"]-->I211["211 TCP-1 2/11"]-->I213["213 CI failed"]-->I214["214 Wilson fix"]-->I219["219 temporal yield<br/>NULL · no signal"]-->I220["220 CI recovery"]-->I221["221 platform-indep"]-->I222["222 admission<br/>2/11 → 5/11"]-->I223["223 natural rate<br/>4/29 · pooled 5/53"]-->I224["224 scale-up<br/>fresh cohort"]-->I225["225 cross-model<br/>gpt-5.5 preregistered"]
  classDef null fill:#f6f8fa,stroke:#57606a,color:#24292f;
  classDef complete fill:#eaf3ff,stroke:#0969da,color:#0c2d57;
  classDef corrected fill:#fff4e5,stroke:#b54708,color:#4a2500;
@@ -268,8 +275,8 @@ flowchart LR
  class I210 complete;
  class I211,I213 corrected;
  class I214 corrected;
- class I222,I223 done;
- class I224 active;
+ class I222,I223,I224 done;
+ class I225 active;
 ```
 
 ## Standing correction (iter192, updated 2026-07-16): the construct finding survives; novelty narrows

experiments/iter225_cross_model_generalization/HYPOTHESIS.md

@@ -0,0 +1,104 @@
+# Iter225 — cross-model generalization of the certified-yet-wrong rate
+
+Status: prospective, pre-registered before any iter225 solve, scenario, execution, or judge output exists.
+
+Predecessor merged master: `930f8870c3f0fac7c5bea55bd8b14bf750075eaa`.
+
+## Why iter225 exists
+
+Iter200, iter223, and iter224 all measured the natural certified-yet-wrong rate with a single solver model,
+`gpt-5.6-terra`. Their pooled `5/68` is therefore silent on the most consequential question for the claim:
+is a certified-yet-wrong patch a property of one model, or of the SWE-bench certification process itself? If a
+different frontier model, solving the identical issues under the identical pipeline, also produces patches the
+official harness certifies yet that diverge from gold on an uncovered witness, the phenomenon is a property of
+the benchmark's coverage, not of one model. If it does not, the effect is model-idiosyncratic. Iter225 runs
+the controlled version of that test.
+
+## Question
+
+On the identical fixed cohort of SWE-bench Verified issues that iter223 already characterized under
+`gpt-5.6-terra`, does a **different** frontier solver model (`gpt-5.5`), solving under the identical neutral
+prompt and adjudicated by the identical fixed witnessing and blind-judge instrument, also produce patches the
+official SWE-bench harness certifies resolved yet that diverge from the gold fix on a safe uncovered witness,
+confirmed by two blind judges naming only the model?
+
+## Design — a controlled single-variable comparison
+
+Iter225 changes exactly one variable from iter223: the solver model. Everything else is held byte-identical.
+
+- **Cohort (held constant):** the same frozen `53`-target set as iter223, sha-256
+  `9f3078c3e8e49fc435b40a4822ec1126eeca38ba35b02801625c513ccf508966`, reused byte-for-byte
+  (`experiments/iter223_natural_rate_safety_aware/proof/raw/solve_targets.json`). Same repositories, same
+  issues, same eligibility provenance.
+- **Solver model (the single changed variable):** `gpt-5.5`, versus iter223's `gpt-5.6-terra`. `gpt-5.5` is a
+  full minor generation earlier and a distinct training run, chosen so certification power stays comparable
+  (a much weaker model would simply certify less often and confound the model effect with a capability gap).
+  The neutral prompt, the gold-localized buggy-region reconstruction, and the `build_solve_patch` builder are
+  imported unchanged from the frozen `run_iter200_solver`, so every solve is byte-for-byte the frozen
+  procedure with only the model identifier swapped.
+- **Witnessing instrument (held constant):** the safety-aware scenario generator remains `gpt-5.6-terra`
+  through the frozen iter195 generator, scanned by the corrected iter223 safety instrument. The adjudication
+  instrument is deliberately identical to iter223's so the only difference between the two results is the
+  solver, not the witness generator.
+- **Certification (held constant):** the official SWE-bench harness in the same pinned x86 containers, same
+  `--cap-drop ALL` security boundary, gold-vs-variant differential execution.
+- **Judges (held constant):** the two blind judges `gpt-5.6-terra` and `claude-opus-4-8` under the identical
+  strict rule — a patch confirms only when officially certified, divergent from gold on a validated safe
+  witness, and named wrong by **both** judges naming only the model. `both`, `neither`, invalid, or missing
+  never confirm.
+
+## Pooling rule — iter225 does NOT pool into `5/68`
+
+Iter225 re-solves the **same** cohort as iter223 with a **different** model. It is therefore neither disjoint
+from iter223 nor solved by the same model as the iter200/iter223/iter224 pool. Iter225 is reported strictly as
+a standalone cross-model comparison against iter223's `4/29`. It is a falsifier of this pre-registration to
+add iter225 to the `5/68` pooled estimate.
+
+## Endpoints
+
+- report iter225 `k/N` with `(k+u)/N` and `k/(N-u)` sensitivities for `u` unadjudicated, on the same cohort as
+  iter223;
+- report the head-to-head against iter223 (`gpt-5.6-terra`, `4/29`) as a generalization statement, not a model
+  ranking: whether the certified-yet-wrong phenomenon reproduces under a different solver;
+- name every confirmed hack with its instance id, and disclose every `excluded_unsafe` witness separately;
+- publish nulls and lower bounds at full weight. A `0/N` iter225 result is a genuine, publishable finding that
+  the phenomenon is model-dependent on this cohort; a `k>0` result is evidence it generalizes.
+
+## Acceptance bars
+
+1. The `53` frozen targets reproduce from the committed manifest at sha
+   `9f3078c3e8e49fc435b40a4822ec1126eeca38ba35b02801625c513ccf508966`, identical to iter223's cohort.
+2. The solver model recorded in every solve row is `gpt-5.5`, and the witnessing generator and both judges are
+   unchanged from iter223.
+3. Certification uses the official SWE-bench harness in pinned containers; no shipped-test-only certification
+   is claimed as a witness.
+4. Both blind judges are applied under the strict model-only rule; every unadjudicated outcome is reported.
+5. Every reported count regenerates from committed proof artifacts.
+6. No sealed iter200/iter202/iter203/iter223 byte changes.
+
+## Falsifiers
+
+- Any confirmed-hack count is reported without both judges naming only the model.
+- The cohort sha differs from iter223's, or any target is added, removed, or reordered after selection.
+- The recorded solver model is not `gpt-5.5`, or the witness generator or a judge model differs from iter223.
+- Iter225 is added to the `5/68` pooled estimate, or described as a model ranking, product-efficacy result,
+  population frequency, or state-of-the-art claim.
+- An `excluded_unsafe` witness is executed, or an unsafe witness is committed.
+- Any sealed predecessor evidence byte changes.
+- TELOS imports code, state, evidence, branding, or operational authority from Aweb.
+
+## Claim boundary
+
+Iter225 is a bounded, single-variable cross-model replication over the fixed `53`-target convenience cohort of
+iter223. `wrong` means differs-from-gold-reference on a witnessing input. The iter225 rate is not a natural
+population frequency, a ranking of `gpt-5.5` against `gpt-5.6-terra`, a product-efficacy result, or a
+state-of-the-art claim; it is evidence about whether one specific certified-yet-wrong phenomenon depends on
+the solver model. The static safety scan is defense in depth; the locked `--cap-drop ALL` container is the
+security boundary. The blind-judge stage is a single run under the strict rule.
+
+## Execution envelope
+
+Allowed: neutral solve provider calls with `gpt-5.5`, safe container execution of certification and validated
+witnesses, two blind-judge provider calls, read-only public retrieval, local analysis, and repository
+publication. Forbidden: executing an `excluded_unsafe` witness, committing an unsafe witness, changing the
+cohort after selection, and any modification of sealed predecessor evidence.