Add DeepSWE GPT xhigh v1 benchmark archive

benchmark/deepswe-gptxhigh-v1/README.md

@@ -0,0 +1,94 @@
+# DeepSWE Five-Arm Benchmark Harness (v1)
+
+How we evaluate five agent configurations ("arms") on the DeepSWE task set
+(113 SWE tasks). Runner/methodology code only — no API/gateway config, no trajectories.
+Run against LoopX revision `2cef51d` (the evaluated revision, not the PR base).
+This versioned snapshot lives at `benchmark/deepswe-gptxhigh-v1/`.
+
+## Reproduction prerequisites
+
+This is an archived source and methodology snapshot, not a standalone runnable
+bundle. The launch scripts retain the original experiment workspace layout.
+Running them requires the external Pier environment, `run.sh`, task manifests
+(`remaining59.txt`, `goal30_subset.py`, `hard24_subset.py`, and
+`remaining4_subset.py`), a configured Python environment, and separately supplied
+model gateway configuration. Adapt the environment paths and set `MR_LOOPX_ROOT`
+to a checkout of the evaluated revision before running. These prerequisites and
+raw verifier artifacts are not included here; the reported results below are
+preserved from the v1 summary and have not been independently reproduced by this
+publication change.
+
+Set `MR_PYTHON` to the interpreter containing the Pier dependencies (defaults to
+`python3`). Both launchers require `MR_MODELONLY_COMPOSE` to name an existing
+external Docker Compose overlay defining the model-only network. Its gateway
+must be reachable from the task containers; loopback placeholders are not a
+portable network configuration. Supply the overlay and gateway configuration
+for your environment before launching.
+
+The remaining-59 launcher accepts `MR_TASK_LIST` (defaults to `remaining59.txt`)
+and requires 59 unique task directory names, one per line. The 54-task launcher
+loads its external frozen subset modules and validates any explicit task subset.
+Each launcher passes a snapshot of its actual selected task ids to preflight;
+the admission receipt hashes those tasks' `upstream/tasks/<id>/task.toml` files.
+The evaluated LoopX revision is fixed and cannot be overridden by environment.
+All LoopX admissions finish before any arm starts; launcher failure is nonzero
+if admission or any arm fails.
+
+Publication fixes add the missing plain runner and harden launch/admission and
+profile initialization. Unsupported legacy `MR_CODEX_ARM=loopx` and Claude
+adapters are excluded from this five-arm package. These fixes do not constitute
+a rerun or revalidation of the historical results.
+
+Retry decisions use structured provider status/code fields; unknown prose-only
+errors now fail instead of being classified by substring. A terminal Todo with
+invalid delivery stops the Codex CLI runner with failure. Offline regression
+coverage can be run with
+`python3 -m pytest -q benchmark/deepswe-gptxhigh-v1/tests/` from the repository root.
+
+## The five arms
+| Arm | Transport | Goal / LoopX | Continuation |
+|---|---|---|---|
+| `plain` | codex app-server | no Goal, no LoopX | single pass |
+| `goal` | codex app-server | native Codex Goal | app-server continues while Goal active |
+| `heartbeat` | `codex exec` (fresh, then `resume`) | LoopX Goal/Todo | recurring supervisor wakes |
+| `codex-cli` | `codex exec` (CLI) | LoopX control plane | external `loopx turn run-once --host codex-cli`, multi-segment |
+| `ssh-goal` | codex app-server | LoopX + native Goal (official full path) | same thread/Goal; LoopX clears blocked + restarts turn |
+
+- Arm dispatch: `pier_cn.py` (`MR_CODEX_ARM=plain|goal|loopx-native|loopx-native-codex-cli|loopx-native-heartbeat`)
+- Arm classes: `goal_codex.py` (`PlainAppServerCodex`, `GoalCodex`) and
+  `loopx_native_codex.py` (the three LoopX variants)
+- Runners: `loopx_wen_native_runner.py` (ssh-goal), `loopx_codex_cli_runner.py` (codex-cli),
+  `loopx_heartbeat_supervisor.py` (heartbeat)
+- Delivery gate: `workspace_delivery.py` (recover agent work from linked git worktrees into
+  `/app` so the collected patch is non-empty)
+- Admission: `preflight_loopx_rerun.py` (pins LoopX revision, delivery self-test)
+
+## Validity (strict)
+`exception_info == null`, independent `verifier/reward.json` present & consistent, task
+checksum matches, and a **non-empty committed patch** exists. Goal/Todo state is lifecycle
+evidence only; the independent verifier is the sole correctness authority. `partial > 0`
+alone does NOT count as valid delivery.
+
+## Results — v1 (113 tasks, per-task best valid)
+
+Historical reported summary only. The current
+[SWE Marathon publication](../swe-marathon/README.md) withdraws SSH Goal and
+Codex CLI data and conclusions pending revalidation. Their rows are retained
+below as part of this v1 archive, not as currently validated results or ranking
+claims.
+
+| Rank | Arm | Solved | Solve rate | Partial | F2P | P2P |
+|---|---|---:|---:|---:|---:|---:|
+| 1 | heartbeat | 70/113 | 61.9% | 0.9739 | 0.886 | 0.993 |
+| 2 | codex-cli | 66/113 | 58.4% | 0.9546 | 0.884 | 0.996 |
+| 3 | goal | 60/113 | 53.1% | 0.9620 | 0.868 | 0.997 |
+| 4 | ssh-goal | 58/113 | 51.3% | 0.9654 | 0.883 | 0.997 |
+| 5 | plain | 54/113 | 47.8% | 0.9206 | 0.745 | 0.997 |
+
+P2P (regression) ≈ 1.0 for all arms; spread is driven by F2P and solve rate.
+The table records the original v1 comparison; conclusions involving the
+withdrawn arms require revalidation.
+
+> Model & gateway endpoints are configured via `MR_*` env vars (not included).
+> Internal hosts/paths replaced with placeholders (`127.0.0.1`, `<REPO_ROOT>`, `<HOME>`).
+> v2 (latest LoopX main) evaluation is in progress and will be published separately.

benchmark/deepswe-gptxhigh-v1/codex_nosandbox_wrapper.py

@@ -0,0 +1,131 @@
+#!/usr/bin/env python3
+"""A `codex` stand-in that drops LoopX's sandbox flags before running the real one.
+
+LoopX's codex-cli host is the path that works: driven through it, a Turn loop
+ran four times on one task, committed a 22 KB patch and scored f2p 31/35.  Its
+one problem is the sandbox — it always passes `--sandbox <mode>` (or
+`-c sandbox_mode=...` when resuming), only permits read-only and
+workspace-write, and both need bubblewrap, which needs unprivileged user
+namespaces these containers do not have:
+
+    bwrap: No permissions to create a new namespace
+
+Switching to `--host generic-cli` avoided that but bought a worse problem: the
+generic host carries its own scheduler contract, and eleven of sixteen turns
+died at "LoopX Turn route is not host executable" before any model work, with
+no route recorded to explain why.
+
+So keep the working host and fix the flag instead.  This sits earlier on PATH
+than the real codex, strips the sandbox arguments, and substitutes the same
+`--dangerously-bypass-approvals-and-sandbox` the other two arms already use —
+which is also what keeps the three arms identical in permissions.  LoopX's
+contracts are untouched: it still believes it is driving codex-cli, because it
+is.
+
+Set MR_REAL_CODEX to the real binary; defaults to /usr/local/bin/codex.
+MR_LOOPX_CODEX_LOG names the log file; defaults to /tmp/loopx-goal/codex-wrapper.log.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+# Resolve the real binary rather than assuming /usr/local/bin/codex.  Codex is
+# installed into the image through nvm, so it lives under the Node version's
+# bin directory and the hardcoded path does not exist — which made the wrapper
+# die before it ever reached Codex, and LoopX report the indistinguishable
+# `codex_cli_exit_nonzero`.  The wrapper is invoked by absolute path through
+# --codex-bin and is not itself on PATH, so a PATH lookup finds the real one.
+REAL = (
+    os.environ.get("MR_REAL_CODEX")
+    or shutil.which("codex")
+    or "/usr/local/bin/codex"
+)
+BYPASS = "--dangerously-bypass-approvals-and-sandbox"
+REASONING_EFFORT = os.environ.get("MR_CODEX_REASONING_EFFORT", "").strip()
+LOG = Path(
+    os.environ.get("MR_LOOPX_CODEX_LOG", "/tmp/loopx-goal/codex-wrapper.log")
+)
+
+
+def rewrite(argv: list[str]) -> list[str]:
+    out: list[str] = []
+    skip_next = False
+    for i, arg in enumerate(argv):
+        if skip_next:
+            skip_next = False
+            continue
+        # `--sandbox <mode>` — new-session form.
+        if arg == "--sandbox":
+            skip_next = True
+            continue
+        if arg.startswith("--sandbox="):
+            continue
+        # `-c sandbox_mode="..."` — resume form.  The value is a separate argv
+        # item after -c, so both have to go, and only when it is that key: -c
+        # carries every other config override too.
+        if arg == "-c" and i + 1 < len(argv) and argv[i + 1].startswith("sandbox_mode="):
+            skip_next = True
+            continue
+        out.append(arg)
+
+    # Insert the bypass right after the subcommand so it lands before `--`,
+    # which codex treats as the end of flags.
+    if out and out[0] == "exec":
+        out.insert(1, BYPASS)
+        if REASONING_EFFORT and not any(
+            value.startswith("model_reasoning_effort=") for value in out
+        ):
+            out[2:2] = ["-c", f"model_reasoning_effort={REASONING_EFFORT}"]
+    else:
+        out.insert(0, BYPASS)
+    return out
+
+
+def _log(text: str) -> None:
+    try:
+        LOG.parent.mkdir(parents=True, exist_ok=True)
+        with LOG.open("a", encoding="utf-8") as handle:
+            handle.write(text.rstrip("\n") + "\n")
+    except OSError:
+        pass
+
+
+def main() -> int:
+    argv = rewrite(sys.argv[1:])
+    # Run the real codex as a child rather than execv'ing it, so its stderr can
+    # be recorded.  LoopX reports a failed Turn as `codex_cli_exit_nonzero` and
+    # keeps neither the exit code's cause nor any output, and the container is
+    # gone by the time anyone looks — so an execv here means the only evidence
+    # of why Codex refused is destroyed at the moment it is produced.
+    #
+    # stdout stays inherited and untouched: LoopX parses Codex's `--json`
+    # stream off it, so anything written there would corrupt the Turn.
+    _log(f"--- argv in : {sys.argv[1:]}")
+    _log(f"--- argv out: {argv}")
+    _log(f"--- real    : {REAL} (exists={os.path.exists(REAL)})")
+    try:
+        completed = subprocess.run(  # noqa: S603
+            [REAL, *argv], stderr=subprocess.PIPE, check=False
+        )
+    except OSError as exc:
+        # Without this the wrapper's own failure to start Codex is reported by
+        # LoopX as `codex_cli_exit_nonzero`, which reads as "the model refused"
+        # rather than "the binary is not there".
+        _log(f"--- launch failed: {type(exc).__name__}: {exc}")
+        sys.stderr.write(f"codex wrapper could not launch {REAL}: {exc}\n")
+        return 127
+    stderr = completed.stderr.decode("utf-8", "replace") if completed.stderr else ""
+    _log(f"--- exit {completed.returncode}")
+    if stderr:
+        _log(stderr)
+        sys.stderr.write(stderr)
+    return completed.returncode
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())