diff --git a/docs/MOLI_COHORT.md b/docs/MOLI_COHORT.md index 5eed058..3ce5356 100644 --- a/docs/MOLI_COHORT.md +++ b/docs/MOLI_COHORT.md @@ -6,14 +6,16 @@ Run on the **same prepared host** for both Moli versions. The runner records hos The tested macOS arm64 setup uses Python 3.11, Node 24, the pinned harness drivers, and ChromeDriver 150.0.7871.49. Resource profiling and host telemetry are disabled for functional comparisons. The general resource benchmark still requires Linux with cgroup v2. -The official automation cohort explicitly passes `--moli-layout on`, which starts Moli with `serve --layout`. Real coordinate input and hit testing require that flag in current Moli releases. The runner's general default remains Moli's lightweight mock-layout mode; it deliberately rejects coordinate mouse and touch dispatch. Optional visual/media resource fetching is separately task-scoped through `launch_profile=all_resources`. Compare version candidates with the same launch flags and frozen run profile; a default-mode run and a layout-enabled run test different runtime configurations. +To reproduce the all-layout automation cohort, explicitly pass `--moli-layout on`, which starts Moli with `serve --layout`. Real coordinate input and hit testing require that flag in current Moli releases. The runner's general default remains Moli's lightweight mock-layout mode; it deliberately rejects coordinate mouse and touch dispatch. Optional visual/media resource fetching is separately task-scoped through `launch_profile=all_resources`. Compare version candidates with the same launch flags and frozen run profile; a default-mode run and a layout-enabled run test different runtime configurations. From a Python environment satisfying the repository's dependencies: ```sh -python tools/run_moli_cohort.py /absolute/path/to/moli-v1 moli_v1_qualified370 -python tools/run_moli_cohort.py /absolute/path/to/moli-v2 moli_v2_qualified370 +python tools/run_moli_cohort.py /absolute/path/to/moli-v1 moli_v1_qualified370 --moli-layout on +python tools/run_moli_cohort.py /absolute/path/to/moli-v2 moli_v2_qualified370 --moli-layout on python tools/compare_moli_cohort.py moli_v1_qualified370 moli_v2_qualified370 ``` Each run produces 1,110 result rows under the ignored `runs/` directory and a sibling `.conditions.json` receipt. The run tool refuses to overwrite an existing run, checks the frozen cohort and manifest before launch, records the Moli and ChromeDriver binary hashes, and verifies completion. The comparator checks every task and attempt plus all recorded non-Moli conditions. Report pass counts from `results.jsonl`, keeping `infra` and `unsupported` separate from task failure. A Moli upgrade may legitimately change task outcomes, timings, and its binary hash and version. + +Add `--try-layout` to an off-mode cohort to rerun its failed cases with layout enabled for all three attempts. Only cases passing all three rerun attempts replace their original results. Use the same retry policy for both version cohorts; original and rerun evidence are retained separately. diff --git a/docs/MOLI_LAYOUT_POLICY.md b/docs/MOLI_LAYOUT_POLICY.md new file mode 100644 index 0000000..95c4b45 --- /dev/null +++ b/docs/MOLI_LAYOUT_POLICY.md @@ -0,0 +1,17 @@ +# Moli layout rerun + +Default runs keep layout off and execute every case for the configured `k` attempts (normally three). `--moli-layout on` enables layout from the start. + +Add `--try-layout` to rerun failed cases after the complete normal run: + +1. Run every case with layout off for all `k` attempts. All attempts must pass for a case to pass. +2. For each failed Moli case, enable layout and run all `k` attempts again. Do not stop early. +3. Only if all `k` rerun attempts pass, replace that case's original results with the complete rerun. Otherwise keep the original failed case unchanged. Never combine successful attempts from the two batches. + +Successful original cases are not rerun. `--try-layout` has no extra effect with `--moli-layout on`. A mandatory Chrome baseline rejection does not trigger a Moli rerun. There is no `auto` mode or preliminary classification. + +Each rerun uses a fresh browser process and a separate fixture session, with the same task, attempt ordinals and seeds. The final `results.jsonl` has the original matrix size, one authoritative row per engine/task/attempt. Pass rates count cases, not physical executions. + +The original matrix is retained as `initial_results.jsonl`; the full rerun is retained as `layout_retry_results.jsonl`. Both artifact sets and their hashes remain available. Only after all reruns finish is the final matrix replaced atomically. An interrupted run remains incomplete and does not yield a publishable final score. + +Primary duration and resource fields describe the selected final executions. Failed recovery batches remain in the retry matrix, and the manifest retains total extra execution count and duration. `layout_retry.total_execution_duration_ms` on each replaced row includes its original and rerun durations; resource evidence for both remains in the artifacts. The manifest declares `failed_case_layout_rerun_v1`, records retried/recovered cases and hashes of all three matrices. Reports must disclose this recovery policy separately from fixed-layout runs. diff --git a/docs/RUNNING.md b/docs/RUNNING.md index 7470cff..b07964f 100644 --- a/docs/RUNNING.md +++ b/docs/RUNNING.md @@ -131,3 +131,7 @@ Round B compares its task-duration distribution against round A to quantify how - `doctor` reports a pin mismatch: the binary under `build_artifacts/` is not the pinned build. Activate the right set or update `active-set.json` deliberately. - Rows come back as `infra`: the identity gate failed, meaning the client did not reach the engine it was supposed to reach. This is an environment or routing problem, never a compatibility score. - A compiled adapter is missing: rebuild with the Go/Rust commands above; `doctor` prints the exact command it expects. + +## Moli layout rerun + +Layout is off by default. Add `--try-layout` to finish the normal three attempts first, then rerun failed cases with layout on for three attempts. Replace a case only when all three rerun attempts pass; otherwise keep its original failed result. `--moli-layout on` enables layout from the start. See [Moli layout rerun](MOLI_LAYOUT_POLICY.md). diff --git a/runner/layout_retry.py b/runner/layout_retry.py new file mode 100644 index 0000000..795f81f --- /dev/null +++ b/runner/layout_retry.py @@ -0,0 +1,90 @@ +"""Retry failed cases once as a complete k-attempt layout-on batch.""" +from __future__ import annotations +import collections +import copy +import hashlib +from pathlib import Path + +POLICY_ID = "failed_case_layout_rerun_v1" + +def policy(mode: str, try_layout: bool = False) -> dict: + if mode not in {"off", "on"}: + raise ValueError("layout mode must be off or on") + return {"policy_id": POLICY_ID, "initial_layout": mode, "try_layout": try_layout, + "retry_layout": "on" if mode == "off" and try_layout else None, + "retry_scope": "failed_cases_after_complete_run", "retry_attempts": "same_k", + "final_result": "all_pass_rerun_replaces_original_case"} + +def groups(rows): + result = collections.defaultdict(list) + for row in rows: + if row["engine"] == "moli": + result[row["task_id"]].append(row) + return result + +def failed_cases(rows, k): + selected = set() + for task_id, attempts in groups(rows).items(): + if len(attempts) != k or {row["attempt"] for row in attempts} != set(range(1,k+1)): + raise ValueError("incomplete original case") + # A mandatory Chrome gate rejection is not an executed Moli failure. + if any(row["status"] == "chrome_gate_fail" for row in attempts): + continue + if not all(row["status"] == "pass" for row in attempts): + selected.add(task_id) + return selected + +def pass_count(rows): + return sum(all(row["status"] == "pass" for row in attempts) for attempts in groups(rows).values()) + +def replace_cases(initial, retries, k, run_dir: Path, run_id: str): + selected = failed_cases(initial,k) + expected = {(task_id, attempt) for task_id in selected for attempt in range(1,k+1)} + replacement = {(row["task_id"],row["attempt"]):row for row in retries} + if len(replacement) != len(retries) or set(replacement) != expected: + raise ValueError("incomplete or unexpected layout rerun") + originals={(row["task_id"],row["attempt"]):row for row in initial if row["engine"]=="moli"} + for key,new in replacement.items(): + if new["engine"] != "moli" or new["seed"] != originals[key]["seed"] or new.get("engine_provenance",{}).get("layout_enabled") is not True: + raise ValueError("layout rerun input or launch mismatch") + recovered={task_id for task_id,rows in groups(retries).items() if all(row["status"]=="pass" for row in rows)} + result=[] + for old in initial: + key=(old["task_id"],old["attempt"]) + if old["engine"] != "moli" or old["task_id"] not in recovered: + result.append(old) + continue + new=copy.deepcopy(replacement[key]) + if new["engine"] != "moli" or new["seed"] != old["seed"] or new.get("engine_provenance",{}).get("layout_enabled") is not True: + raise ValueError("layout rerun input or launch mismatch") + new["run_id"]=run_id + new["layout_retry"]={ + "policy_id":POLICY_ID, + "original_artifact_dir":old["artifact_dir"], + "original_run_sha256":hashlib.sha256((run_dir/old["artifact_dir"]/"run.json").read_bytes()).hexdigest(), + "retry_run_sha256":hashlib.sha256((run_dir/new["artifact_dir"]/"run.json").read_bytes()).hexdigest(), + "original_status":old["status"], + "total_execution_duration_ms":old["duration_ms"]+new["duration_ms"], + } + result.append(new) + return result + + +def verify(run_dir: Path, manifest: dict, final_rows: list) -> None: + if (manifest.get("moli_layout_policy") or {}).get("retry_layout") == "on" and manifest.get("completion_status") != "completed": + raise ValueError("layout recovery run is incomplete") + receipt=manifest.get("layout_retry") + if not receipt: + return + import json + matrices=[] + for label in ("initial", "retry"): + name=receipt[label+"_results"] + path=run_dir/name + if Path(name).name != name or path.is_symlink() or hashlib.sha256(path.read_bytes()).hexdigest()!=receipt[label+"_results_sha256"]: + raise ValueError("layout evidence matrix hash mismatch") + matrices.append([json.loads(line) for line in path.read_text().splitlines()]) + if hashlib.sha256((run_dir/'results.jsonl').read_bytes()).hexdigest()!=receipt['final_results_sha256']: + raise ValueError("final matrix hash mismatch") + if replace_cases(*matrices,int(manifest['k_runs']),run_dir,manifest['run_id']) != final_rows: + raise ValueError("final matrix does not match complete successful reruns") diff --git a/runner/run.py b/runner/run.py index e620740..2495061 100644 --- a/runner/run.py +++ b/runner/run.py @@ -32,6 +32,7 @@ if __package__: from runner import bindings as binding_catalog + from runner import layout_retry from runner import resources as resource_metrics from runner import semantics as semantic_model from runner.launch_profiles import DEFAULT_LAUNCH_PROFILE, LAUNCH_PROFILES @@ -40,6 +41,7 @@ # Keep the historical direct-script entry point (`python3 runner/run.py`) # working as well as the preferred module form (`python3 -m runner.run`). import bindings as binding_catalog + import layout_retry import resources as resource_metrics import semantics as semantic_model from launch_profiles import DEFAULT_LAUNCH_PROFILE, LAUNCH_PROFILES @@ -2215,6 +2217,7 @@ def run_manifest_payload( item["mode"] = "headless=new" if engine == "moli": item["resource_fetch_policy"] = "task_scoped_launch_profile" + item["layout_mode"] = getattr(args, "moli_layout", "off") if engine == "obscura": item.update( { @@ -2239,6 +2242,11 @@ def run_manifest_payload( host_telemetry_enabled = str(getattr(args, "host_telemetry", "on") or "on") == "on" calibration_baseline = getattr(args, "resource_calibration_baseline", None) + layout_receipt = None + if "moli" in selected_engines: + layout_receipt = layout_retry.policy(getattr(args, "moli_layout", "off"), getattr(args, "try_layout", False)) + + return { "run_id": run_id, "started_at": now_iso(), @@ -2275,6 +2283,7 @@ def run_manifest_payload( "manifest": rel_to_repo(ACTIVE_ENGINE_SET_PATH) if ACTIVE_ENGINE_SET else None, }, "engines": engines, + **({"moli_layout_policy": layout_receipt} if layout_receipt is not None else {}), "host": resource_metrics.host_provenance( str(getattr(args, "provenance_level", "full") or "full") ), @@ -2498,6 +2507,7 @@ def find_free_port() -> int: def engine_serve_args( engine: str, launch_profile: str = DEFAULT_LAUNCH_PROFILE, + extra_serve_args: tuple[str, ...] = (), ) -> tuple[str, ...]: if launch_profile not in LAUNCH_PROFILES: raise BenchError(f"unsupported launch profile: {launch_profile}") @@ -2507,7 +2517,9 @@ def engine_serve_args( str(arg) for arg in meta.get("launch_profile_args", {}).get(launch_profile, ()) ) - return base_args + profile_args + if extra_serve_args and engine != "moli": + raise BenchError("task-scoped serve arguments are only supported for Moli") + return base_args + profile_args + extra_serve_args def serve_engine_launch_command( @@ -2515,6 +2527,7 @@ def serve_engine_launch_command( binary: pathlib.Path, port: int, launch_profile: str = DEFAULT_LAUNCH_PROFILE, + extra_serve_args: tuple[str, ...] = (), ) -> list[str]: """Build the auditable serve command for a non-Chrome engine.""" return [ @@ -2524,7 +2537,7 @@ def serve_engine_launch_command( LOCAL_HOST, "--port", str(port), - *engine_serve_args(engine, launch_profile), + *engine_serve_args(engine, launch_profile, extra_serve_args), ] @@ -2556,14 +2569,15 @@ def launch( self, engine: str, launch_profile: str = DEFAULT_LAUNCH_PROFILE, + extra_serve_args: tuple[str, ...] = (), ) -> BrowserProcess: with self._lock: if self._closed: raise BenchError("browser manager is closed") - return self._launch_locked(engine, launch_profile) + return self._launch_locked(engine, launch_profile, extra_serve_args) - def _launch_locked(self, engine: str, launch_profile: str) -> BrowserProcess: - desired_serve_args = engine_serve_args(engine, launch_profile) + def _launch_locked(self, engine: str, launch_profile: str, extra_serve_args: tuple[str, ...]) -> BrowserProcess: + desired_serve_args = engine_serve_args(engine, launch_profile, extra_serve_args) if engine in self.processes: browser = self.processes[engine] proc = browser.process @@ -2597,7 +2611,7 @@ def _launch_locked(self, engine: str, launch_profile: str) -> BrowserProcess: for _ in range(attempts): port = find_free_port() if self.dynamic_ports else int(meta["cdp_port"]) try: - return self._launch_on_port(engine, binary, port, launch_profile) + return self._launch_on_port(engine, binary, port, launch_profile, extra_serve_args) except BenchError as exc: last_error = exc if "already in use" not in str(exc): @@ -2610,10 +2624,11 @@ def _launch_on_port( binary: pathlib.Path, port: int, launch_profile: str, + extra_serve_args: tuple[str, ...], ) -> BrowserProcess: if port_is_open(port): raise BenchError(f"{engine}: port {port} is already in use") - serve_args = engine_serve_args(engine, launch_profile) + serve_args = engine_serve_args(engine, launch_profile, extra_serve_args) if engine == "chrome": profile_dir = pathlib.Path(tempfile.mkdtemp(prefix=f"abb-chrome-{port}-")) self._profile_dirs.append(profile_dir) @@ -2645,7 +2660,7 @@ def _launch_on_port( "about:blank", ] else: - cmd = serve_engine_launch_command(engine, binary, port, launch_profile) + cmd = serve_engine_launch_command(engine, binary, port, launch_profile, extra_serve_args) # Browser output goes to spool files: PIPE would deadlock the engine # once the 64 KiB pipe buffer fills (nobody drains it during a run). @@ -4143,7 +4158,7 @@ def engine_provenance(browser: BrowserProcess) -> dict[str, Any]: different local browser visible in every attempt row. """ binary = browser.binary - return { + payload = { "engine": browser.engine, "pid": getattr(browser.process, "pid", None), "binary": rel_to_repo(binary) if binary is not None else None, @@ -4156,6 +4171,9 @@ def engine_provenance(browser: BrowserProcess) -> dict[str, Any]: "browser_ws": browser.version_info.get("webSocketDebuggerUrl"), "http_identity": dict(browser.version_info), } + if browser.engine == "moli": + payload["layout_enabled"] = "--layout" in browser.serve_args + return payload def is_unsupported_error(exc: Exception) -> bool: @@ -6221,8 +6239,13 @@ def run_driver_attempt( resource_runtime: ResourceRuntime | None = None, fixture_server: FixtureServer | None = None, scenario_binding: dict[str, Any] | None = None, + physical_variant: str | None = None, + persist_result: bool = True, ) -> dict[str, Any]: tmp_dir, final_dir, artifact_rel = artifact_paths(run_dir, task, engine, attempt) + if physical_variant: + final_dir = final_dir / physical_variant + artifact_rel = (pathlib.Path(artifact_rel) / physical_variant).as_posix() if final_dir.exists(): raise BenchError(f"artifact directory already exists; refusing to overwrite: {final_dir}") tmp_dir.mkdir(parents=True, exist_ok=False) @@ -6641,7 +6664,8 @@ def run_driver_attempt( ensure_profile_files(tmp_dir, task.artifact_profile) final_dir.parent.mkdir(parents=True, exist_ok=True) tmp_dir.replace(final_dir) - append_result(results_path, result) + if persist_result: + append_result(results_path, result) return result @@ -7071,10 +7095,14 @@ def __init__( ) -> None: self.manager = manager for engine in engines: - manager.launch(engine, initial_task.launch_profile) + manager.launch(engine, initial_task.launch_profile, self.extra_args(engine, initial_task)) + + @staticmethod + def extra_args(engine: str, task: ResolvedTask) -> tuple[str, ...]: + return () def for_task(self, engine: str, task: ResolvedTask) -> BrowserProcess: - browser = self.manager.launch(engine, task.launch_profile) + browser = self.manager.launch(engine, task.launch_profile, self.extra_args(engine, task)) browser.prev_task_id = self.manager.note_task(engine, task.task_id) return browser @@ -7304,6 +7332,52 @@ def worker(item: tuple[ResolvedTask, int]) -> list[dict[str, Any]]: reporter.phase(f"Failed after {reporter.completed_rows}/{reporter.total_rows} result rows") raise BenchError("parallel run failed for some attempts:\n" + "\n".join(errors[:10])) reporter.finish() + if "moli" in selected_engines and args.moli_layout == "off" and getattr(args, "try_layout", False): + initial_rows = read_jsonl(results_path) + failed_ids = layout_retry.failed_cases(initial_rows, args.k) + if failed_ids: + reporter.phase(f"Retrying {len(failed_ids)} failed Moli cases with layout on: {args.k} attempts each") + initial_path = run_dir / "initial_results.jsonl" + initial_path.write_bytes(results_path.read_bytes()) + retry_path = run_dir / "layout_retry_results.jsonl" + retry_rows = [] + manager = BrowserManager(dynamic_ports=True, resource_runtime=resource_runtime, worker_slot=len(managers) + 1) + managers.append(manager) + originals = {(row["task_id"], row["attempt"]): row for row in initial_rows if row["engine"] == "moli"} + for task in tasks: + if task.task_id not in failed_ids: + continue + for attempt in range(1, args.k + 1): + previous = manager.processes.get("moli") + if previous is not None: + manager._kill_process(previous.process) + browser = manager.launch("moli", task.launch_profile, ("--layout",)) + browser.prev_task_id = manager.note_task("moli", task.task_id) + old = originals[(task.task_id, attempt)] + driver_id = CATALOG_DRIVER_BY_TASK_KIND.get(str(task.driver.get("kind") or "")) + binding = unavailable_bindings.get(("moli", driver_id)) + if binding is None and driver_id == "selenium": + binding = selenium_bindings["moli"] + row = run_driver_attempt( + run_dir, retry_path, run_id + "-layout-retry", task, "moli", attempt, + old["seed"], browser, old["chrome_gate"], score_eligible, + fixture_base_url, score_mode, resource_runtime, fixture_server, binding, + physical_variant="layout-on", + ) + retry_rows.append(row) + final_rows = layout_retry.replace_cases(initial_rows, retry_rows, args.k, run_dir, run_id) + final_path = run_dir / ".final_results.jsonl" + final_path.write_text("".join(json.dumps(row, sort_keys=True) + "\n" for row in final_rows), encoding="utf-8") + final_path.replace(results_path) + run_manifest["layout_retry"] = { + "initial_results": "initial_results.jsonl", "initial_results_sha256": sha256_file(initial_path), + "retry_results": "layout_retry_results.jsonl", "retry_results_sha256": sha256_file(retry_path), + "final_results_sha256": sha256_file(results_path), + "retried_cases": sorted(failed_ids), "extra_executions": len(retry_rows), + "extra_execution_duration_ms": sum(row["duration_ms"] for row in retry_rows), + "recovered_cases": sorted(task_id for task_id in failed_ids if all(row["status"] == "pass" for row in retry_rows if row["task_id"] == task_id)), + } + reporter.phase(f"Final Moli cases: {layout_retry.pass_count(final_rows)}/{len(tasks)} passed") run_completed = True finally: for local_manager in managers: @@ -7970,6 +8044,8 @@ def command_run(args: argparse.Namespace) -> int: } payload = { "manifest": rel_to_repo(manifest_path), + "seed": args.seed, + "moli_layout_policy": layout_retry.policy(getattr(args, "moli_layout", "off"), getattr(args, "try_layout", False)), "selected_layers": sorted({task.layer for task in tasks}), "tasks": [ task.to_run_manifest(semantic_index.get(task.task_id)) @@ -8122,6 +8198,7 @@ def summarize_results(run_manifest: dict[str, Any], rows: list[dict[str, Any]]) "harness_version": run_manifest.get("harness_version") or "unknown", "score_eligible": bool(run_manifest.get("score_eligible")), "layers": layers, + "moli_layout_policy": run_manifest.get("moli_layout_policy"), "evaluation_axes": evaluation_axes, "chrome_baseline": chrome_gate, "chrome_gate": chrome_gate, @@ -8140,6 +8217,12 @@ def write_scorecard(run_dir: pathlib.Path, run_manifest: dict[str, Any], rows: l lines.append(f"- score_eligible: `{run_manifest.get('score_eligible')}`") lines.append(f"- enabled_subsets: `{', '.join(run_manifest.get('enabled_subsets', []))}`") lines.append(f"- attempts: `{len(rows)}`") + if (run_manifest.get("moli_layout_policy") or {}).get("retry_layout") == "on": + receipt=run_manifest.get("layout_retry") or {} + lines.append(f"- Layout recovery: failed cases rerun {run_manifest['k_runs']} times with layout on; replace only all-pass reruns.") + lines.append(f"- Recovered cases: {len(receipt.get('recovered_cases', []))}; retried cases: {len(receipt.get('retried_cases', []))}; extra executions: {receipt.get('extra_executions', 0)}.") + lines.append(f"- Extra rerun driver duration: {receipt.get('extra_execution_duration_ms', 0)} ms (excluded from final-execution latency).") + lines.append("") lines.append("| engine | binary | version | sha12 |") lines.append("|---|---|---|---|") @@ -8305,6 +8388,7 @@ def generate_report_files(run_dir: pathlib.Path, emit: bool = True) -> None: raise BenchError(f"run directory not found: {run_dir}") run_manifest = load_json(run_dir / "run_manifest.json") rows = read_jsonl(run_dir / "results.jsonl") + layout_retry.verify(run_dir, run_manifest, rows) scores = summarize_results(run_manifest, rows) write_json(run_dir / "scores.json", scores) write_scorecard(run_dir, run_manifest, rows, scores) @@ -8465,8 +8549,9 @@ def build_parser() -> argparse.ArgumentParser: run.add_argument("--engines", default="chrome,moli,lightpanda,obscura") run.add_argument( "--moli-layout", choices=("off", "on"), default="off", - help="Moli layout policy: off preserves its lightweight default; on enables on-demand real layout and coordinate input", + help="Moli layout: off (default), on (always enabled)", ) + run.add_argument("--try-layout", action="store_true", help="After the normal run, rerun each failed Moli case with layout on for the same k attempts; replace original results only when all rerun attempts pass") run.add_argument("--jobs", type=int, default=1, help="parallel task workers; each worker owns isolated browser processes on ephemeral ports") run.add_argument( "--k", diff --git a/test/test_layout_retry.py b/test/test_layout_retry.py new file mode 100644 index 0000000..c7097b0 --- /dev/null +++ b/test/test_layout_retry.py @@ -0,0 +1,67 @@ +from pathlib import Path +import copy +import json +import pytest +from runner import layout_retry, run + + +def batch(tmp_path, task, statuses, layout=False): + rows=[] + for attempt,status in enumerate(statuses,1): + artifact=f"{task}/{attempt}/{'on' if layout else 'off'}" + row=dict(task_id=task,engine="moli",attempt=attempt,status=status,seed=f"seed-{attempt}",duration_ms=10,artifact_dir=artifact,run_id="physical",engine_provenance={"layout_enabled":layout}) + p=tmp_path/artifact/'run.json';p.parent.mkdir(parents=True);p.write_text(json.dumps(row)) + rows.append(row) + return rows + + +def test_successful_whole_batch_only_replaces_failed_case(tmp_path): + original=batch(tmp_path,'pass',['pass']*3)+batch(tmp_path,'recover',['fail','pass','fail'])+batch(tmp_path,'still-fail',['fail']*3) + retry=batch(tmp_path,'recover',['pass']*3,True)+batch(tmp_path,'still-fail',['pass','fail','pass'],True) + final=layout_retry.replace_cases(original,retry,3,tmp_path,'logical') + assert len(final)==9 + assert final[:3]==original[:3] + assert final[6:]==original[6:] + assert all(row['status']=='pass' and row['run_id']=='logical' for row in final[3:6]) + assert layout_retry.pass_count(original)==1 + assert layout_retry.pass_count(final)==2 + assert all(row['layout_retry']['total_execution_duration_ms']==20 for row in final[3:6]) + + +@pytest.mark.parametrize('mutation',['missing','duplicate','wrong-seed','layout-off']) +def test_incomplete_or_wrong_rerun_cannot_replace_results(tmp_path,mutation): + original=batch(tmp_path,'task',['fail']*3);retry=batch(tmp_path,'task',['pass']*3,True) + if mutation=='missing':retry.pop() + elif mutation=='duplicate':retry.append(copy.deepcopy(retry[0])) + elif mutation=='wrong-seed':retry[0]['seed']='different' + else:retry[0]['engine_provenance']['layout_enabled']=False + with pytest.raises(ValueError):layout_retry.replace_cases(original,retry,3,tmp_path,'logical') + + +def test_gate_skip_and_pass_are_not_retried(tmp_path): + rows=batch(tmp_path,'pass',['pass']*3)+batch(tmp_path,'gate',['chrome_gate_fail']*3) + assert layout_retry.failed_cases(rows,3)==set() + + +def test_cli_defaults_and_opt_in(): + parser=run.build_parser();args=parser.parse_args(['run']) + assert args.moli_layout=='off' and args.try_layout is False + assert parser.parse_args(['run','--try-layout']).try_layout is True + assert layout_retry.policy('on',True)['retry_layout'] is None + with pytest.raises(SystemExit):parser.parse_args(['run','--moli-layout','auto']) + + +def test_recovery_metadata_is_not_a_version_comparison_control(): + import sys + sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'tools')) + from compare_moli_cohort import normalized_manifest + base={'engines':{'moli':{'layout_mode':'off'}},'moli_layout_policy':layout_retry.policy('off',True)} + candidate=copy.deepcopy(base);candidate['layout_retry']={'retried_cases':['one']} + assert normalized_manifest(base)==normalized_manifest(candidate) + candidate['moli_layout_policy']=layout_retry.policy('off',False) + assert normalized_manifest(base)!=normalized_manifest(candidate) + + +def test_interrupted_recovery_cannot_generate_final_report(tmp_path): + manifest={'moli_layout_policy':layout_retry.policy('off',True),'completion_status':'interrupted'} + with pytest.raises(ValueError,match='incomplete'):layout_retry.verify(tmp_path,manifest,[]) diff --git a/test/test_moli_cohort_compare.py b/test/test_moli_cohort_compare.py index 72f2183..e658169 100644 --- a/test/test_moli_cohort_compare.py +++ b/test/test_moli_cohort_compare.py @@ -47,3 +47,12 @@ def test_runner_fixture_task_host_and_jobs_changes_are_rejected(): target = target[part] target[path[-1]] = "changed" assert first_difference(normalized_manifest(left), normalized_manifest(right)) + +def test_fixed_layout_ignores_annotation_drift_but_not_launch_drift(): + left=manifest();left['engines']['moli']['layout_mode']='on' + left['moli_layout_policy']={'policy_id':'task_layout_v3','qualification':None,'assignments_sha256':'first','assignments':[{'task_id':'one','task_sha256':'frozen','layout':'on','requirement':'required','reason':'evidence'}]} + right=copy.deepcopy(left);right['moli_layout_policy']['assignments_sha256']='second' + right['moli_layout_policy']['assignments'][0].update(requirement='unknown',reason='different_binary') + assert first_difference(normalized_manifest(left),normalized_manifest(right)) is None + right['moli_layout_policy']['assignments'][0]['layout']='off' + assert first_difference(normalized_manifest(left),normalized_manifest(right)) diff --git a/test/test_unit_checks.py b/test/test_unit_checks.py index c4cc0f7..3ac51e7 100644 --- a/test/test_unit_checks.py +++ b/test/test_unit_checks.py @@ -477,14 +477,14 @@ def __init__(self, pid): def poll(self): return None - def fake_launch(engine, launched_binary, port, launch_profile): + def fake_launch(engine, launched_binary, port, launch_profile, extra_serve_args): browser = runner_run.BrowserProcess( engine=engine, port=port, process=Proc(1000 + len(launched)), version_info={}, binary=launched_binary, - serve_args=runner_run.engine_serve_args(engine, launch_profile), + serve_args=runner_run.engine_serve_args(engine, launch_profile, extra_serve_args), ) launched.append((launch_profile, browser)) manager.processes[engine] = browser @@ -505,6 +505,12 @@ def fake_launch(engine, launched_binary, port, launch_profile): assert killed == [default.process] assert manager.processes == {"moli": all_resources} + layout = manager.launch("moli", "all_resources", ("--layout",)) + assert layout is not all_resources + assert layout.serve_args == ("--resource", "--layout") + assert manager.launch("moli", "all_resources", ("--layout",)) is layout + assert killed == [default.process, all_resources.process] + # --- write_json / append_jsonl / read_jsonl (TESTING.md §6) ---------------------- diff --git a/test/test_unit_versioning.py b/test/test_unit_versioning.py index 3d6f243..f1b45bd 100644 --- a/test/test_unit_versioning.py +++ b/test/test_unit_versioning.py @@ -9,6 +9,7 @@ from __future__ import annotations import argparse +import hashlib import json import pathlib import re @@ -17,6 +18,7 @@ from runner import run as runner_run from runner.version import HARNESS_VERSION +from runner.layout_retry import POLICY_ID REPO_ROOT = pathlib.Path(runner_run.REPO_ROOT) SEMVER = re.compile(r"\d+\.\d+\.\d+") @@ -76,3 +78,16 @@ def test_run_manifest_records_both_axes(): assert payload["bench_version"] == suite["bench_version"] assert payload["harness_version"] == HARNESS_VERSION assert "site_version" not in payload["site"] + + +def test_layout_retry_receipt_declares_policy_before_calls(): + manifest_path = REPO_ROOT / "manifest.json" + suite, tasks, errors = runner_run.validate_manifest(manifest_path, requested_subsets=["l1.raw_cdp"]) + assert not errors + args = argparse.Namespace(chrome_gate="off", score_mode="independent", jobs=1, k=1, seed="unit", moli_layout="off", try_layout=True) + payload = runner_run.run_manifest_payload(args,suite,manifest_path,tasks[:1],["moli"],"retry",True,[],None) + receipt=payload["moli_layout_policy"] + assert receipt["initial_layout"] == "off" + assert receipt["try_layout"] is True + assert receipt["retry_scope"] == "failed_cases_after_complete_run" + assert receipt["retry_attempts"] == "same_k" diff --git a/tools/compare_moli_cohort.py b/tools/compare_moli_cohort.py index 506f0af..b265ddf 100644 --- a/tools/compare_moli_cohort.py +++ b/tools/compare_moli_cohort.py @@ -15,13 +15,22 @@ def normalized_manifest(manifest: dict) -> dict: """Remove run-local fields and the sole intended treatment variable.""" result = json.loads(json.dumps(manifest)) - for key in ("argv", "run_id", "started_at", "completed_at", "site"): + for key in ("argv", "run_id", "started_at", "completed_at", "site", "layout_retry"): result.pop(key, None) result.get("engine_set", {}).pop("name", None) moli = result["engines"]["moli"] for key in ("version", "sha256", "sha256_12", "expected_sha256", "expected_sha256_12"): moli.pop(key, None) result.get("host_telemetry", {}).pop("summary", None) + # With a fixed global override, version-specific annotations are metadata, + # not treatment changes. Keep every effective launch assignment comparable. + policy = result.get("moli_layout_policy") + if isinstance(policy, dict) and moli.get("layout_mode") in {"off", "on"}: + policy.pop("assignments_sha256", None) + policy["assignments"] = [ + {key:item[key] for key in ("task_id", "task_sha256", "layout")} + for item in policy.get("assignments", []) + ] return result diff --git a/tools/report_four_engine.py b/tools/report_four_engine.py index 42b6367..c248ed4 100644 --- a/tools/report_four_engine.py +++ b/tools/report_four_engine.py @@ -38,6 +38,10 @@ def load_run(run_dir: pathlib.Path) -> tuple[dict, list[dict], dict]: for line in (run_dir / "results.jsonl").read_text(encoding="utf-8").splitlines() if line.strip() ] + if manifest.get("layout_retry"): + sys.path.insert(0,str(pathlib.Path(__file__).resolve().parents[1])) + from runner.layout_retry import verify + verify(run_dir,manifest,rows) return manifest, rows, scores @@ -159,6 +163,10 @@ def build_report(manifest: dict, rows: list[dict], scores: dict) -> str: f"`score_eligible: {str(bool(manifest.get('score_eligible'))).lower()}`, no fallback." ) out("") + if (manifest.get("moli_layout_policy") or {}).get("retry_layout") == "on": + receipt=manifest.get("layout_retry") or {} + out(f"Moli layout recovery: after the normal {k} attempts, failed cases receive {k} layout-on attempts. Only all-pass reruns replace original results; failed reruns leave the original case unchanged. Recovered {len(receipt.get('recovered_cases', []))} of {len(receipt.get('retried_cases', []))} retried cases. The {receipt.get('extra_executions', 0)} extra executions do not increase the case denominator; their driver duration is {receipt.get('extra_execution_duration_ms', 0)} ms, separate from final-execution latency.") + out("") out( "This report covers local pinned-binary engines only. Remote endpoints " "(such as Kitesurf) sit in a different evidence class; see the five-engine report." diff --git a/tools/run_moli_cohort.py b/tools/run_moli_cohort.py index 39a7630..1077db1 100644 --- a/tools/run_moli_cohort.py +++ b/tools/run_moli_cohort.py @@ -62,6 +62,8 @@ def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("moli_binary", type=Path, help="absolute path to the version under test") parser.add_argument("run_id", help="new result directory name") + parser.add_argument("--try-layout", action="store_true", help="Rerun failed cases with layout on for the same k attempts; replace only all-pass reruns") + parser.add_argument("--moli-layout", choices=("on", "off"), default="off") args = parser.parse_args() if not args.moli_binary.is_absolute() or not args.moli_binary.is_file() or not os.access(args.moli_binary, os.X_OK): parser.error("Moli must be an executable absolute path") @@ -117,6 +119,8 @@ def main() -> None: "chromedriver_version": driver_version, "moli_sha256": binary_sha, "moli_version": version, + "moli_layout": args.moli_layout, + "try_layout": args.try_layout, } receipt.write_text(json.dumps(conditions, indent=2) + "\n", encoding="utf-8") env = dict(os.environ) @@ -132,7 +136,7 @@ def main() -> None: sys.executable, "-m", "runner.run", "run", *(part for task_id in task_ids for part in ("--task", task_id)), "--engines", "moli", "--score-mode", profile["score_mode"], - "--moli-layout", "on", + "--moli-layout", args.moli_layout, "--chrome-baseline", profile["chrome_baseline"], "--seed", profile["seed"], "--k", str(profile["attempts_per_task"]), @@ -143,6 +147,8 @@ def main() -> None: "--provenance-level", profile["provenance_level"], "--no-progress", ] print(f"Moli {version} sha256={binary_sha}; {len(task_ids)} tasks × {profile['attempts_per_task']} attempts", flush=True) + if args.try_layout: + command.append("--try-layout") subprocess.run(command, cwd=ROOT, env=env, check=True) if file_sha256(binary) != binary_sha or file_sha256(driver) != driver_sha: raise ValueError("Moli or ChromeDriver binary changed during the run") @@ -151,9 +157,14 @@ def main() -> None: or manifest.get("completed_result_rows") != len(task_ids) * profile["attempts_per_task"] or manifest["engines"]["moli"]["sha256"] != binary_sha): raise ValueError("run did not produce the complete pinned Moli matrix") - if "--layout" not in manifest["engines"]["moli"].get("serve_args", []): - raise ValueError("Moli run omitted the required --layout flag") - print(f"Complete baseline: {run_dir}") + if manifest["engines"]["moli"].get("layout_mode") != args.moli_layout: + raise ValueError("Moli layout mode differs from the declared run") + has_global_layout = "--layout" in manifest["engines"]["moli"].get("serve_args", []) + if has_global_layout != (args.moli_layout == "on"): + raise ValueError("Moli global layout flag differs from the declared run") + if (manifest.get("moli_layout_policy") or {}).get("try_layout", False) != args.try_layout: + raise ValueError("Moli retry policy differs from the declared run") + print(f"Complete cohort: {run_dir}") if __name__ == "__main__":