Add opt-in layout reruns for failed Moli cases

docs/MOLI_COHORT.md

@@ -6,14 +6,16 @@ Run on the **same prepared host** for both Moli versions. The runner records hos
 
 The tested macOS arm64 setup uses Python 3.11, Node 24, the pinned harness drivers, and ChromeDriver 150.0.7871.49. Resource profiling and host telemetry are disabled for functional comparisons. The general resource benchmark still requires Linux with cgroup v2.
 
-The official automation cohort explicitly passes `--moli-layout on`, which starts Moli with `serve --layout`. Real coordinate input and hit testing require that flag in current Moli releases. The runner's general default remains Moli's lightweight mock-layout mode; it deliberately rejects coordinate mouse and touch dispatch. Optional visual/media resource fetching is separately task-scoped through `launch_profile=all_resources`. Compare version candidates with the same launch flags and frozen run profile; a default-mode run and a layout-enabled run test different runtime configurations.
+To reproduce the all-layout automation cohort, explicitly pass `--moli-layout on`, which starts Moli with `serve --layout`. Real coordinate input and hit testing require that flag in current Moli releases. The runner's general default remains Moli's lightweight mock-layout mode; it deliberately rejects coordinate mouse and touch dispatch. Optional visual/media resource fetching is separately task-scoped through `launch_profile=all_resources`. Compare version candidates with the same launch flags and frozen run profile; a default-mode run and a layout-enabled run test different runtime configurations.
 
 From a Python environment satisfying the repository's dependencies:
 
 ```sh
-python tools/run_moli_cohort.py /absolute/path/to/moli-v1 moli_v1_qualified370
-python tools/run_moli_cohort.py /absolute/path/to/moli-v2 moli_v2_qualified370
+python tools/run_moli_cohort.py /absolute/path/to/moli-v1 moli_v1_qualified370 --moli-layout on
+python tools/run_moli_cohort.py /absolute/path/to/moli-v2 moli_v2_qualified370 --moli-layout on
 python tools/compare_moli_cohort.py moli_v1_qualified370 moli_v2_qualified370
 ```
 
 Each run produces 1,110 result rows under the ignored `runs/` directory and a sibling `<run-id>.conditions.json` receipt. The run tool refuses to overwrite an existing run, checks the frozen cohort and manifest before launch, records the Moli and ChromeDriver binary hashes, and verifies completion. The comparator checks every task and attempt plus all recorded non-Moli conditions. Report pass counts from `results.jsonl`, keeping `infra` and `unsupported` separate from task failure. A Moli upgrade may legitimately change task outcomes, timings, and its binary hash and version.
+
+Add `--try-layout` to an off-mode cohort to rerun its failed cases with layout enabled for all three attempts. Only cases passing all three rerun attempts replace their original results. Use the same retry policy for both version cohorts; original and rerun evidence are retained separately.

docs/MOLI_LAYOUT_POLICY.md

@@ -0,0 +1,17 @@
+# Moli layout rerun
+
+Default runs keep layout off and execute every case for the configured `k` attempts (normally three). `--moli-layout on` enables layout from the start.
+
+Add `--try-layout` to rerun failed cases after the complete normal run:
+
+1. Run every case with layout off for all `k` attempts. All attempts must pass for a case to pass.
+2. For each failed Moli case, enable layout and run all `k` attempts again. Do not stop early.
+3. Only if all `k` rerun attempts pass, replace that case's original results with the complete rerun. Otherwise keep the original failed case unchanged. Never combine successful attempts from the two batches.
+
+Successful original cases are not rerun. `--try-layout` has no extra effect with `--moli-layout on`. A mandatory Chrome baseline rejection does not trigger a Moli rerun. There is no `auto` mode or preliminary classification.
+
+Each rerun uses a fresh browser process and a separate fixture session, with the same task, attempt ordinals and seeds. The final `results.jsonl` has the original matrix size, one authoritative row per engine/task/attempt. Pass rates count cases, not physical executions.
+
+The original matrix is retained as `initial_results.jsonl`; the full rerun is retained as `layout_retry_results.jsonl`. Both artifact sets and their hashes remain available. Only after all reruns finish is the final matrix replaced atomically. An interrupted run remains incomplete and does not yield a publishable final score.
+
+Primary duration and resource fields describe the selected final executions. Failed recovery batches remain in the retry matrix, and the manifest retains total extra execution count and duration. `layout_retry.total_execution_duration_ms` on each replaced row includes its original and rerun durations; resource evidence for both remains in the artifacts. The manifest declares `failed_case_layout_rerun_v1`, records retried/recovered cases and hashes of all three matrices. Reports must disclose this recovery policy separately from fixed-layout runs.

docs/RUNNING.md

@@ -131,3 +131,7 @@ Round B compares its task-duration distribution against round A to quantify how
 - `doctor` reports a pin mismatch: the binary under `build_artifacts/` is not the pinned build. Activate the right set or update `active-set.json` deliberately.
 - Rows come back as `infra`: the identity gate failed, meaning the client did not reach the engine it was supposed to reach. This is an environment or routing problem, never a compatibility score.
 - A compiled adapter is missing: rebuild with the Go/Rust commands above; `doctor` prints the exact command it expects.
+
+## Moli layout rerun
+
+Layout is off by default. Add `--try-layout` to finish the normal three attempts first, then rerun failed cases with layout on for three attempts. Replace a case only when all three rerun attempts pass; otherwise keep its original failed result. `--moli-layout on` enables layout from the start. See [Moli layout rerun](MOLI_LAYOUT_POLICY.md).

runner/layout_retry.py

@@ -0,0 +1,90 @@
+"""Retry failed cases once as a complete k-attempt layout-on batch."""
+from __future__ import annotations
+import collections
+import copy
+import hashlib
+from pathlib import Path
+
+POLICY_ID = "failed_case_layout_rerun_v1"
+
+def policy(mode: str, try_layout: bool = False) -> dict:
+    if mode not in {"off", "on"}:
+        raise ValueError("layout mode must be off or on")
+    return {"policy_id": POLICY_ID, "initial_layout": mode, "try_layout": try_layout,
+            "retry_layout": "on" if mode == "off" and try_layout else None,
+            "retry_scope": "failed_cases_after_complete_run", "retry_attempts": "same_k",
+            "final_result": "all_pass_rerun_replaces_original_case"}
+
+def groups(rows):
+    result = collections.defaultdict(list)
+    for row in rows:
+        if row["engine"] == "moli":
+            result[row["task_id"]].append(row)
+    return result
+
+def failed_cases(rows, k):
+    selected = set()
+    for task_id, attempts in groups(rows).items():
+        if len(attempts) != k or {row["attempt"] for row in attempts} != set(range(1,k+1)):
+            raise ValueError("incomplete original case")
+        # A mandatory Chrome gate rejection is not an executed Moli failure.
+        if any(row["status"] == "chrome_gate_fail" for row in attempts):
+            continue
+        if not all(row["status"] == "pass" for row in attempts):
+            selected.add(task_id)
+    return selected
+
+def pass_count(rows):
+    return sum(all(row["status"] == "pass" for row in attempts) for attempts in groups(rows).values())
+
+def replace_cases(initial, retries, k, run_dir: Path, run_id: str):
+    selected = failed_cases(initial,k)
+    expected = {(task_id, attempt) for task_id in selected for attempt in range(1,k+1)}
+    replacement = {(row["task_id"],row["attempt"]):row for row in retries}
+    if len(replacement) != len(retries) or set(replacement) != expected:
+        raise ValueError("incomplete or unexpected layout rerun")
+    originals={(row["task_id"],row["attempt"]):row for row in initial if row["engine"]=="moli"}
+    for key,new in replacement.items():
+        if new["engine"] != "moli" or new["seed"] != originals[key]["seed"] or new.get("engine_provenance",{}).get("layout_enabled") is not True:
+            raise ValueError("layout rerun input or launch mismatch")
+    recovered={task_id for task_id,rows in groups(retries).items() if all(row["status"]=="pass" for row in rows)}
+    result=[]
+    for old in initial:
+        key=(old["task_id"],old["attempt"])
+        if old["engine"] != "moli" or old["task_id"] not in recovered:
+            result.append(old)
+            continue
+        new=copy.deepcopy(replacement[key])
+        if new["engine"] != "moli" or new["seed"] != old["seed"] or new.get("engine_provenance",{}).get("layout_enabled") is not True:
+            raise ValueError("layout rerun input or launch mismatch")
+        new["run_id"]=run_id
+        new["layout_retry"]={
+            "policy_id":POLICY_ID,
+            "original_artifact_dir":old["artifact_dir"],
+            "original_run_sha256":hashlib.sha256((run_dir/old["artifact_dir"]/"run.json").read_bytes()).hexdigest(),
+            "retry_run_sha256":hashlib.sha256((run_dir/new["artifact_dir"]/"run.json").read_bytes()).hexdigest(),
+            "original_status":old["status"],
+            "total_execution_duration_ms":old["duration_ms"]+new["duration_ms"],
+        }
+        result.append(new)
+    return result
+
+
+def verify(run_dir: Path, manifest: dict, final_rows: list) -> None:
+    if (manifest.get("moli_layout_policy") or {}).get("retry_layout") == "on" and manifest.get("completion_status") != "completed":
+        raise ValueError("layout recovery run is incomplete")
+    receipt=manifest.get("layout_retry")
+    if not receipt:
+        return
+    import json
+    matrices=[]
+    for label in ("initial", "retry"):
+        name=receipt[label+"_results"]
+        path=run_dir/name
+        if Path(name).name != name or path.is_symlink() or hashlib.sha256(path.read_bytes()).hexdigest()!=receipt[label+"_results_sha256"]:
+            raise ValueError("layout evidence matrix hash mismatch")
+        matrices.append([json.loads(line) for line in path.read_text().splitlines()])
+    if hashlib.sha256((run_dir/'results.jsonl').read_bytes()).hexdigest()!=receipt['final_results_sha256']:
+        raise ValueError("final matrix hash mismatch")
+    if replace_cases(*matrices,int(manifest['k_runs']),run_dir,manifest['run_id']) != final_rows:
+        raise ValueError("final matrix does not match complete successful reruns")