feat(swe-bench): add persistent Pyxis runtime

AGENTS.md

@@ -100,6 +100,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
 | **VideoGen**                        | `src/inference_endpoint/videogen/`                                                                                                                                             | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader.                                                                                                                           |
 | **SWE-bench**                       | `src/inference_endpoint/dataset_manager/predefined/swe_bench/`, `src/inference_endpoint/evaluation/swe_bench_scorer.py`, `src/inference_endpoint/evaluation/swebench_service/` | `SWEBench` predefined dataset (HuggingFace `princeton-nlp/SWE-bench_Verified` or `_Lite`; `ACCURACY_ONLY=True`). `SWEBenchScorer` sets `SKIP_ENDPOINT_PHASE=True` and bypasses the built-in accuracy phase entirely: it delegates agent execution and grading to the configured SWE-bench service via `accuracy_config.extras.swebench_service_url`. The service is an isolated `uv` subproject; its host owns Docker/runtime execution, artifacts, and credentials, while the benchmark client remains the report-producing entrypoint.                                                                                                                                                                                                                                                                                                                                  |
 | **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py`                                                                                                  | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`accuracy/accuracy_results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass).                                                                                                                                                                                                                        |
+| **SWE-bench (distributed)**         | `src/inference_endpoint/evaluation/swe_bench_distributed/`, `src/inference_endpoint/evaluation/swe_bench_fleet_scorer.py`, `scripts/swe_bench_wq.py` | `SWEBenchFleetScorer` (`eval_method: swe_bench_fleet`) shards the instance list into units and runs them across several SWE-bench services concurrently, reusing the same service HTTP protocol. Adds what the single-service path lacks: a durable `mkdir`-atomic work queue (resume after a client crash), an eval-phase infra-vs-genuine classifier driving in-unit retry, pre-dispatch gates on the inference endpoints (checkpoint identity, tool call at >=2k-token scale, endpoint fingerprint), a memory guard, and an all-or-nothing merge gate that compares instance **ids** (never counts) and is scoped to exactly one run id. Operator CLI: `scripts/swe_bench_wq.py {status,merge,requeue,reap}` — `requeue` is the only way to re-run a unit; deleting a result leaves the claim tombstone in place. See `docs/evaluation/SWE_BENCH_DISTRIBUTED.md`. |
 | **Compliance (audit tests)**        | `src/inference_endpoint/compliance/`, `commands/audit.py`                                                                                                                      | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_<TEST>.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Perf-only by default (a phase may opt into accuracy via `AuditRunSpec.test_mode`, but this is unused today). |
 
 ### Hot-Path Architecture
@@ -266,6 +267,8 @@ src/inference_endpoint/
 │   └── adapter.py             # VideoGenAdapter (HttpRequestAdapter) + VideoGenAccumulator (no-op)
 ├── evaluation/                # Accuracy evaluation (extractor, scoring, livecodebench)
 │   └── swebench_service/      # Isolated uv service for Docker-backed SWE-bench runs
+│   ├── swe_bench_distributed/ # Fleet dispatch: unit plan, work queue, reaper, classifier, gates, guards, merge gate
+│   └── swe_bench_fleet_scorer.py  # SWEBenchFleetScorer (scorer_id swe_bench_fleet)
 ├── compliance/                # Submission compliance checks (config-lock, accuracy gate, run validity)
 │   ├── __init__.py
 │   └── checker.py             # check_submission() + Check/ComplianceReport (Edge-Agentic ruleset)

docs/evaluation/DESIGN.md

@@ -93,6 +93,21 @@ Code execution cannot be done safely in-process. The evaluation server runs in a
 with resource limits. This is a deliberate architecture choice — not a shortcut — and is
 documented prominently in the dataset README.
 
+**An externally-scored accuracy-only run builds an idle issuer**
+
+A scorer with `SKIP_ENDPOINT_PHASE` evaluates through its own service, so the run
+issues no samples and `total_samples` is zero. `BenchmarkSession` still requires a
+non-None issuer, so one is built and never used. Accuracy-only runs are pinned to
+`num_workers=1` for deterministic single-stream ordering, and `HTTPClientConfig`
+requires `num_workers` to divide the endpoint count, so that idle client is handed a
+single endpoint. This is scoped strictly to the zero-sample case: a run that will
+issue keeps the full endpoint list and the divisibility invariant applies to it
+unchanged.
+
+Scorers that themselves fan work across several endpoints read the endpoint list from
+the run's `config.yaml`, not from this client, so the narrowing does not narrow the
+run.
+
 ## Integration Points
 
 | Component                        | Role                                                   |