Add DeepSWE GPT xhigh v1 benchmark archive

[gwh6669999]

Add the DeepSWE five-arm benchmark source package and historical v1 summary for 113 tasks under benchmark/deepswe-gptxhigh-v1/, targeting main. The evaluated LoopX revision remains 2cef51d.

新增五臂运行代码与 v1 113 题历史汇总，保留实验版本与当前 PR 基线的区别。README 说明外部环境依赖，并链接上游 SSH Goal / Codex CLI 结果撤回声明；本次代码修复不代表历史结果已经重新验证。

The package includes the plain app-server runner and the three native LoopX treatments. Obsolete Claude and legacy hand-written Turn adapters are excluded. External Pier tooling, task manifests, a model-only Compose overlay, and gateway setup are still required; no credentials, task text, raw trajectories, or verifier artifacts are published.

Execution and admission fixes:

• Validate and snapshot the actual task selection for both launchers; receipts hash that selection rather than a hard-coded 54-task list.
• Keep the evaluated revision fixed, serialize shared profile initialization, and require an explicit external network overlay.
• Finish admission before launching any arm, propagate each background failure, use per-arm gateway ports, and allow MR_PYTHON selection.
• Fail on terminal Todos without valid delivery and unmatched install scripts; classify retries from structured provider errors instead of prose.

Validation: 33 offline regression cases passed, covering missing/duplicate task lists, exact task subsets, admission rejection, per-arm failures and ports, concurrent profile initialization, plain/no-Goal transport, structured retry classification, invalid terminal delivery, unsupported arms, and external overlays. Python and Bash syntax checks and git diff --check passed. No model calls or real benchmark jobs were run.

Changes are confined to this experiment directory. Removing unused legacy adapters is the related scope reduction; shared product code is unchanged.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings affect benchmark execution, admission controls, and launcher failure handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds an archived, non-standalone DeepSWE GPT xhigh v1 five-arm benchmark snapshot and historical 113-task results.

Changes:

• Adds LoopX, Codex, Claude, Goal, delivery, heartbeat, and admission tooling.
• Adds 54-task and remaining-59 launch scripts.
• Documents prerequisites, methodology, and historical result limitations.

File summaries

File	Reviewed change
benchmark/deepswe-gptxhigh-v1/workspace_delivery.py	Workspace delivery normalization and patch verification.
benchmark/deepswe-gptxhigh-v1/run_loopx_rerun_54_20260908.sh	54-task LoopX rerun launcher.
benchmark/deepswe-gptxhigh-v1/run_five_arms_remaining59_20260910.sh	Five-arm launcher for the remaining 59 tasks.
benchmark/deepswe-gptxhigh-v1/README.md	Archive scope, prerequisites, methodology, and results.
benchmark/deepswe-gptxhigh-v1/preflight_loopx_rerun.py	LoopX revision and fairness admission checks.
benchmark/deepswe-gptxhigh-v1/pier_cn.py	Pier patching, networking, and arm setup.
benchmark/deepswe-gptxhigh-v1/loopx_wen_native_runner.py	Native Goal continuation runner.
benchmark/deepswe-gptxhigh-v1/loopx_turn_runner.py	Governed LoopX Turn execution and validation.
benchmark/deepswe-gptxhigh-v1/loopx_native_codex.py	Native Codex profile installation and integration.
benchmark/deepswe-gptxhigh-v1/loopx_heartbeat_supervisor.py	Recurring heartbeat supervision.
benchmark/deepswe-gptxhigh-v1/loopx_codex_cli_runner.py	Multi-segment Codex CLI execution.
benchmark/deepswe-gptxhigh-v1/goal_codex.py	Codex Goal and plain-arm integration.
benchmark/deepswe-gptxhigh-v1/goal_claude.py	Claude arm integration.
benchmark/deepswe-gptxhigh-v1/codex_nosandbox_wrapper.py	Codex sandbox and effort-argument wrapper.

Review details

Suppressed comments (6)

benchmark/deepswe-gptxhigh-v1/loopx_codex_cli_runner.py:214

• If the Todo has already reached a terminal state but delivery is invalid, this branch does not return or create repair work; it falls through to the next turn run-once with no open benchmark P0. That turn cannot make progress and the runner burns the remaining segments instead of failing clearly or reopening a delivery repair Todo.

        if claim.get("terminal"):
            delivery = normalize_delivery(project, base_sha)
            write_receipt(delivery_path, delivery)
            if delivery["treatment_valid"]:
                result = {
                    "schema_version": "deepswe_codex_cli_runner_v1",
                    "execution_mode": "loopx_turn_run_once",
                    "host_surface": "codex_cli",

benchmark/deepswe-gptxhigh-v1/loopx_wen_native_runner.py:172

• Retry/fatal routing is based on raw message substrings such as server error and timeout. A permanent provider error containing one of these words is retried up to the configured limit, while a transient 429 or temporarily unavailable response without these exact phrases aborts immediately; that can consume the 14,400-second budget and bias arm outcomes. Prefer structured provider error codes/status or a typed retry classification.

def _is_transient_model_error(message: str) -> bool:
    lowered = message.lower()
    return any(
        marker in lowered
        for marker in (
            "rate limit",

benchmark/deepswe-gptxhigh-v1/pier_cn.py:160

• When no install step matches, this code only emits a warning and returns a runnable spec. That contradicts the fail-fast contract described above and lets an upstream install-script change proceed with public registries, producing a slow or otherwise non-comparable benchmark instead of stopping admission. Abort or mark the affected harness inadmissible when no expected step is rewritten.

            if not changed:
                # Pier changed its install script out from under the patch: say
                # so rather than let a sweep crawl or die one task at a time.
                print(
                    f"pier_cn: WARNING — no install step of {_name} matched; its "
                    "build will use upstream's GitHub/PyPI/npm path and may hang.",

benchmark/deepswe-gptxhigh-v1/run_five_arms_remaining59_20260910.sh:48

• launch_pg accepts the per-arm port, but both endpoint variables are hard-coded to 4250. The goal and plain arms therefore do not use their advertised isolated gateways (4393/4394), and can share or connect to the wrong server while MR_GATEWAY_PORT records a different port. Use $port for both values.

    PIER_CUSTOM_NETWORKS=1 MR_MODELONLY_NET=1 MR_MODELONLY_HOST=127.0.0.1 MR_MODELONLY_PORT=4250 \
    MR_API_BASE=http://127.0.0.1:4250/v1 MR_AGENT=codex MR_MODEL="$MODEL" MR_EFFORT="$EFFORT" MR_REASONING_EFFORT="$EFFORT" \

benchmark/deepswe-gptxhigh-v1/run_five_arms_remaining59_20260910.sh:7

• This archive hard-codes a workspace-specific virtualenv name, so the launcher fails on any reproduction checkout that does not have .venv-user-395647. The README says the Python environment is a prerequisite, but provides no way to select it; use an overridable interpreter (MR_PYTHON/python3) instead of embedding the original workspace path.

PY="$DIR/.venv-user-395647/bin/python"

benchmark/deepswe-gptxhigh-v1/run_loopx_rerun_54_20260908.sh:97

• The same workspace-specific interpreter path is hard-coded again for this launcher's preflight and cross-arm admission check. That makes the primary 54-task rerun fail outside the original machine, even though the README presents the Python environment as an adaptable prerequisite; make the interpreter configurable and use it consistently at both call sites.

  "$DIR/.venv-user-395647/bin/python" "$DIR/preflight_loopx_rerun.py" \
    --arm "$arm_mode" --loopx-root "$LOOPX_ROOT" --port "$port" \
    --model "$MODEL" --effort "$EFFORT" --goal-timeout "$GOAL_TIMEOUT" \
    --heartbeat-segment-timeout "$HEARTBEAT_SEGMENT_TIMEOUT" \
    --turn-idle-timeout "$TURN_IDLE_TIMEOUT" \
    --agent-timeout-multiplier "$AGENT_TIMEOUT_MULTIPLIER" \
    --output "$LOG_ROOT/admission-$arm_mode.json"

• Files reviewed: 14/14 changed files
• Comments generated: 16
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[gwh6669999]

Superseded by #4495. The replacement preserves the original v1 snapshot byte-for-byte and places subsequent execution fixes in a separate v1-revised directory. 原始 v1 快照与后续执行逻辑修复已拆分，详见替代 PR #4495。