From c756e98b30d5d737b472ece46fcf41741457bde4 Mon Sep 17 00:00:00 2001 From: DavidKoleczek <45405824+DavidKoleczek@users.noreply.github.com> Date: Fri, 14 Aug 2026 13:04:22 -0400 Subject: [PATCH] Add deep-swe benchmark harness and document it in the evaluation README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deep-swe is a 113-task agentic SWE benchmark whose tasks are Harbor-format directories pinned to prebuilt Docker images. The pier CLI owns the environment, verification and grading, so this harness is kept separate from the DTU-based matrix harness rather than folded into it: there are no DTUs, graders or task definitions here, only the agent adapters and a runner. Covers the same four arms as the matrix harness (amplifier-agent, amplifier-foundation, opencode-amplifier-agent, opencode-vanilla). Task data is cloned at runtime and never vendored. Adds a high-level section to the evaluation README so the harness is discoverable from the top level, with setup and correctness constraints left in deep-swe/README.md. Also removes the two remaining notes/ files, which were left behind by the earlier move to e2e-first testing. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com> --- .amplifier/evaluation/README.md | 35 + .amplifier/evaluation/deep-swe/.gitignore | 8 + .amplifier/evaluation/deep-swe/README.md | 191 +++++ .amplifier/evaluation/deep-swe/pyproject.toml | 19 + .amplifier/evaluation/deep-swe/run.py | 678 ++++++++++++++++ .../deep-swe/src/deepswe_agents/__init__.py | 20 + .../src/deepswe_agents/amplifier_agent.py | 113 +++ .../deepswe_agents/amplifier_foundation.py | 230 ++++++ .../deep-swe/src/deepswe_agents/base.py | 608 +++++++++++++++ .../deep-swe/src/deepswe_agents/metrics.py | 727 ++++++++++++++++++ .../src/deepswe_agents/opencode_amplifier.py | 77 ++ .../src/deepswe_agents/opencode_vanilla.py | 147 ++++ .../evaluation/deep-swe/tests/conftest.py | 104 +++ .../deep-swe/tests/test_metrics_dedup.py | 200 +++++ .../deep-swe/tests/test_opencode_metrics.py | 257 +++++++ .../deep-swe/tests/test_teardown_guards.py | 290 +++++++ notes/e2e-coverage-gaps.md | 129 ---- notes/foundation-pin-reproducibility.md | 82 -- 18 files changed, 3704 insertions(+), 211 deletions(-) create mode 100644 .amplifier/evaluation/deep-swe/.gitignore create mode 100644 .amplifier/evaluation/deep-swe/README.md create mode 100644 .amplifier/evaluation/deep-swe/pyproject.toml create mode 100644 .amplifier/evaluation/deep-swe/run.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/__init__.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_agent.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_foundation.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/base.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/metrics.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_amplifier.py create mode 100644 .amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_vanilla.py create mode 100644 .amplifier/evaluation/deep-swe/tests/conftest.py create mode 100644 .amplifier/evaluation/deep-swe/tests/test_metrics_dedup.py create mode 100644 .amplifier/evaluation/deep-swe/tests/test_opencode_metrics.py create mode 100644 .amplifier/evaluation/deep-swe/tests/test_teardown_guards.py delete mode 100644 notes/e2e-coverage-gaps.md delete mode 100644 notes/foundation-pin-reproducibility.md diff --git a/.amplifier/evaluation/README.md b/.amplifier/evaluation/README.md index 001e9e92..2a8ee490 100644 --- a/.amplifier/evaluation/README.md +++ b/.amplifier/evaluation/README.md @@ -32,6 +32,8 @@ src/eval/ the harness package run.py entry point runs/ gitignored per-run outputs + +deep-swe/ separate harness for the deep-swe benchmark (see below) ``` ## Task groups @@ -133,6 +135,39 @@ uv run python run.py automationbench \ - `--launch-timeout`, `--grade-timeout` tune the launch and in-DTU grade steps. Output defaults to `runs/automationbench/`. +## deep-swe + +`deep-swe/` is a self-contained harness for +[deep-swe](https://github.com/datacurve-ai/deep-swe), a 113-task agentic SWE +benchmark. It is deliberately separate from the matrix harness above: deep-swe +tasks are Harbor-format directories pinned to prebuilt Docker images, and the +`pier` CLI (a Harbor fork) owns the environment, verification and grading. There +are no DTUs, no graders and no task definitions here, only the agent adapters and +a thin runner. + +``` +deep-swe/run.py entry point (--list-agents, --list-tasks, --dry-run) +deep-swe/src/deepswe_agents/ agent adapters installed into pier's venv +deep-swe/tests/ unit tests for metrics parsing and teardown guards +``` + +The same four arms as the matrix harness: `amplifier-agent`, +`amplifier-foundation`, `opencode-amplifier-agent`, `opencode-vanilla`. + +``` +python run.py --agents amplifier-agent,opencode-vanilla -n 15 --seed 1234 +``` + +Scoring is binary `reward` (all fail-to-pass and pass-to-pass tests pass) plus a +`partial` fraction that is the useful dev signal. Task data is cloned at runtime +into `~/.cache/deep-swe//` and is never vendored here. Results land under +`/evaluation_results//`, not `runs/`. + +Setup differs from the rest of this directory (Docker, `pier` installed from git, +agents installed into pier's venv), and there are several correctness constraints +that make numbers comparable or worthless. See `deep-swe/README.md` before running +it. + ## Runtime-fetched data Some task groups store only a selector and pull content on the fly, so no diff --git a/.amplifier/evaluation/deep-swe/.gitignore b/.amplifier/evaluation/deep-swe/.gitignore new file mode 100644 index 00000000..a9cb2ce7 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/.gitignore @@ -0,0 +1,8 @@ +jobs/ +tasks/ +.venv/ +__pycache__/ +*.pyc +.ruff_cache/ +*.egg-info/ +rendered-dockerfiles.txt diff --git a/.amplifier/evaluation/deep-swe/README.md b/.amplifier/evaluation/deep-swe/README.md new file mode 100644 index 00000000..a388929b --- /dev/null +++ b/.amplifier/evaluation/deep-swe/README.md @@ -0,0 +1,191 @@ +# deep-swe benchmark harness + +Runs Amplifier agents against [deep-swe](https://github.com/datacurve-ai/deep-swe), +a 113-task agentic SWE benchmark. Each task is a Harbor-format directory pinned to +a prebuilt Docker image. The `pier` CLI (a Harbor fork) owns the environment, +verification and grading. This directory supplies only the agents plus a thin runner. + +``` +reward binary 1 only if ALL fail-to-pass AND ALL pass-to-pass tests pass +partial 0..1 fraction of tests passing -- this is the useful dev signal +``` + +Work is graded as `git diff HEAD` in `/app`, so **uncommitted work +scores zero**. deep-swe's own `instruction.md` already tells the agent to commit; +we pass it through verbatim and add nothing. As a backstop the harness runs a +fallback commit afterward, which announces itself in the trial log: + +``` +PIER_AMPLIFIER_FALLBACK_COMMIT: committed <- dev signal only, NOT comparable +PIER_AMPLIFIER_FALLBACK_COMMIT: nothing-to-commit <- agent committed its own work +``` + +Check that marker before quoting any number as leaderboard-comparable -- no +leaderboard run gets a fallback commit. + +## Setup + +Docker running, `ANTHROPIC_API_KEY` exported, and: + +```bash +uv tool install --force git+https://github.com/datacurve-ai/pier@0daf53d3599e58c4506cf0bcff5e12c77dc282d2 + +cd .amplifier/evaluation/deep-swe +uv pip install --python "$(dirname "$(readlink -f "$(which pier)")")/python" -e . +``` + +The second command installs the agents into pier's venv so `--agent-import-path` +can resolve them. + +Install pier **from git, not PyPI**. The PyPI 0.3.0 build has no +`[[verifier.collect]]` hook support, so `model.patch` is never produced and every +task silently scores 0. Both builds self-report `0.3.0`, so the version string +proves nothing; `run.py` probes for the feature and refuses to run on the PyPI build. + +## Running + +```bash +python run.py --list-agents +python run.py --list-tasks + +# one agent, one task +python run.py --agents amplifier-agent --tasks textual-richlog-follow-state + +# the same deterministic 15 tasks for three arms +python run.py --agents amplifier-foundation,amplifier-agent,opencode-vanilla \ + -n 15 --seed 1234 + +# print the pier commands, resolved task list and pins without running +python run.py --dry-run --agents amplifier-agent -n 15 --seed 1234 +``` + +An explicit `--tasks` or `-n` is required: a full 113-task sweep is expensive in +both API spend and wall-clock time, and per-task cost varies widely -- a task +solved early costs far less than one that runs to its timeout. + +Task data is cloned at runtime into `~/.cache/deep-swe//` (`--tasks-dir` or +`DEEP_SWE_CACHE_DIR`). It is never copied into this repo. + +``` +-n N --seed S deterministic sample; same seed, same subset, any machine +--n-concurrent N trials pier runs in parallel WITHIN one agent's job +--agent-timeout-multiplier multiplier on each task's timeout_sec (default 1.5) +--local-source PATH install a local checkout instead of the pinned ref +--no-pin install from moving branches instead of resolved SHAs +--pier-arg=--foo forward a raw arg to `pier run` (repeatable) +``` + +Agents run sequentially, one `pier run` each. To run arms in parallel, launch one +`run.py` per arm against a shared `--jobs-dir`. + +`--n-concurrent` and `--agent-timeout-multiplier` are first-class flags; passing +them through `--pier-arg` duplicates them. + +## What keeps the numbers valid + +**The instruction is passed through untouched.** Do not append scoring hints or +commit reminders. Every leaderboard score came from `mini-swe-agent`, which +augments nothing; any addition makes our numbers incomparable in our own favor. + +**Every agent runs the identical task list.** Selection is resolved once on the +host and passed as explicit `--include-task-name` args. pier's own `--n-tasks` / +`--sample-seed` are never used -- this runner invokes pier once per agent in a +separate process, so delegating the sampling would let each arm draw its own +subset while every summary still lined them up side by side. + +**Moving refs are pinned to SHAs at run start** (`--no-pin` opts out). A multi-task +run takes hours; installing from `@main` lets upstream move mid-run so tasks get +graded against different code. The pins, task list, seed, deep-swe SHA and pier +SHA land in `/run-manifest.json`. + +**Timeouts get 1.5x headroom by default.** A timed-out trial scores 0 +indistinguishably from a capability failure. + +**Only one duration is reported:** `agent_run_s`, measured with `time.monotonic()` +around the agent command. Wall clocks can step backward under NTP correction -- +observed here by enough to make pier report a trial finishing before the timeout +that killed it -- so no pier-derived duration is read at all. + +**A $0 is never reported as a free run.** If no cost figure could be produced, the +field is the string `not_available`. + +**opencode's cost is recomputed, not read.** It prices from a models.dev card that +disagrees with the reference card on cache rates and ignores the `cost.cache` +override in `opencode.json`, so `metrics.parse_opencode_db` takes only token +counts and applies `MODEL_RATES_PER_M` (mirroring `_RATES` in +`amplifier-module-provider-anthropic/_cost.py`). If Anthropic changes pricing, +both tables must move together. opencode's own figure is kept in `notes`. + +## Output + +One timestamped directory per invocation under `/evaluation_results/` +(`DEEP_SWE_RESULTS_DIR`), one job dir per agent. `run.py` prints a per-trial +summary at the end. + +``` +/run-manifest.json task list, seed, resolved pins, deep-swe + pier SHAs +/deepswe-/__/ + verifier/reward.json reward, f2p_passed/total, p2p, partial + result.json pier timings + agent_result (tokens, cost, metadata) + artifacts/model.patch the graded submission + agent/agent.log agent stdout/stderr + agent/metrics.json token/cost detail, dedup notes, timing + agent/sessions/... the agent's session tree, pulled per SESSION_DIRS +``` + +## Agents + +``` +amplifier-agent amplifier-agent CLI driven directly +amplifier-foundation full amplifier stack (`amplifier run`) with the anchors bundle +opencode-amplifier-agent OpenCode frontend backed by amplifier-agent +opencode-vanilla stock OpenCode talking straight to Anthropic (control arm) +``` + +`--local-source` replaces the pinned ref with a filtered upload of a local +checkout: `amplifier-agent` and `opencode-amplifier-agent` -> amplifier-agent, +`amplifier-foundation` -> amplifier, `opencode-vanilla` -> unsupported. The agent +logs `INSTALLED VERSION: '...'` so the trial log proves which build ran. A missing +path is a hard error, never a silent fallback to the git ref. + +Per-arm gotchas, each learned the hard way: + +**amplifier-agent** writes nothing to disk unless `--session-id` is passed -- +without it the CLI mints a telemetry-only `ephemeral-` id and silently skips +`transcript.jsonl`, `metadata.json` and `audits/`. The adapter passes +`--session-id deepswe-trial`. Do NOT relocate `AMPLIFIER_AGENT_HOME` to get the +session onto the bind mount: the context-intelligence hook's `base_path` is a +literal that does not expand it, so relocating splits the trajectory in two and +loses the token data. + +**amplifier-foundation** runs `amplifier run --bundle --mode single`. +`--bundle` is explicit so a stray `/app/.amplifier/settings.yaml` in a task repo +cannot swap the stack out from under the benchmark. The anchors bundle composes +both logging hooks, so every LLM call is written to disk twice in two envelope +shapes; `parse_events` de-duplicates on the provider response id. Do not "fix" +the duplication by narrowing the glob. Do not add anchors with +`amplifier bundle add --app` -- it is already the default primary bundle, so +`--app` composes it on top of itself. + +**opencode-vanilla** records usage in SQLite, not events.jsonl. The whole data dir +is pulled rather than `opencode.db` alone: opencode runs SQLite in WAL mode and +the newest writes live in the `opencode.db-wal` sidecar, so pulling the db by +itself yields a stale database that silently under-reports. + +**opencode-amplifier-agent** declares no `SESSION_DIRS`, so it collects no +trajectory and reports no token or cost data. + +## Raw LLM payloads (amplifier-agent) + +Off by default -- it multiplies the size of `events.jsonl`, and a full-budget run +makes well over a hundred requests. + +```bash +python run.py --agents amplifier-agent --tasks --pier-arg=--ak --pier-arg=raw_llm_payloads=true +``` + +Sets `{"debug": {"rawLlmPayloads": true}}` in the container host-config, which +attaches full `data.raw` (messages, system, tools, model, untruncated) to every +`llm:request` and `llm:response`. It must be a real JSON boolean; a string is +rejected deliberately, because `"false"` is truthy and would silently enable +capture. The trial log records which mode was used. diff --git a/.amplifier/evaluation/deep-swe/pyproject.toml b/.amplifier/evaluation/deep-swe/pyproject.toml new file mode 100644 index 00000000..e68920e0 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/pyproject.toml @@ -0,0 +1,19 @@ +[project] +name = "deepswe-agents" +version = "0.1.0" +description = "Amplifier agent adapters for the deep-swe benchmark (run via the pier CLI)" +requires-python = ">=3.11" +# Intentionally empty. This package is installed with `uv pip install -e .` INTO +# pier's own venv, which already provides `pier`; declaring it here would try to +# resolve a tool that is installed separately. pytest is a dev-only need. +dependencies = [] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/deepswe_agents"] + +[tool.ruff] +line-length = 100 diff --git a/.amplifier/evaluation/deep-swe/run.py b/.amplifier/evaluation/deep-swe/run.py new file mode 100644 index 00000000..536dd5b7 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/run.py @@ -0,0 +1,678 @@ +#!/usr/bin/env python3 +"""Runner for the deep-swe benchmark. + +Thin wrapper over the `pier` CLI: resolves the pinned deep-swe task checkout, +validates the agent/task selection, and shells out to `pier run` once per agent. + +Task data is cloned at runtime into a cache dir OUTSIDE this repo and is never +redistributed here. +""" + +from __future__ import annotations + +import argparse +import json +import os +import random +import shutil +import subprocess +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent / "src")) + +from deepswe_agents import AGENTS, LOCAL_SOURCE_AGENTS + +DEEP_SWE_REPO = "https://github.com/datacurve-ai/deep-swe" +DEEP_SWE_SHA = "435ee89ec2f2e2289f33b0da4f992f0b7b7266b9" + +PIER_GIT_SHA = "0daf53d3599e58c4506cf0bcff5e12c77dc282d2" +PIER_INSTALL_CMD = ( + f"uv tool install --force git+https://github.com/datacurve-ai/pier@{PIER_GIT_SHA}" +) + +DEFAULT_MODEL = "anthropic/claude-sonnet-5" +HERE = Path(__file__).parent + +# Multiplier applied to each task's `[agent] timeout_sec` (5400s for the tasks +# seen so far). At 1.0 a harder task has little headroom, and a timeout produces +# a 0 that looks like a capability result rather than a truncation. 1.5 buys +# headroom without materially changing the ceiling on a well-behaved run, since +# a fast agent simply finishes earlier. +DEFAULT_AGENT_TIMEOUT_MULTIPLIER = 1.5 + +# Per-agent constructor kwarg -> upstream repo whose HEAD gets resolved to a +# commit SHA at run start. Every arm that installs from a moving branch belongs +# here; opencode-vanilla does not, because it already pins an installer VERSION. +# +# WHY THIS EXISTS. A multi-task run takes hours. Installing from `@main` means +# upstream can move between the first task and the last, so tasks get graded +# against different agent code and the comparison quietly stops being valid. +# Resolving once per run and passing the SHA to every trial makes a run +# internally consistent and, via run-manifest.json, reproducible afterwards. +PINNABLE_REFS: dict[str, dict[str, str]] = { + "amplifier-agent": { + "amplifier_agent_ref": "https://github.com/microsoft/amplifier-agent", + }, + "amplifier-foundation": { + "amplifier_ref": "https://github.com/microsoft/amplifier", + "anchors_ref": ( + "https://github.com/microsoft/amplifier-foundation" + "#subdirectory=bundles/anchors/bundle.md" + ), + }, +} + +# One timestamped directory per invocation, holding one job dir per agent. +# Everything a run produces -- pier job output, agent session trees, extracted +# trajectories, metrics -- lands under here so a run is a single self-contained +# artifact that can be archived or deleted as a unit. +RUN_TIMESTAMP = time.strftime("%Y%m%d-%H%M%S") + + +def default_results_root() -> Path: + root = os.environ.get("DEEP_SWE_RESULTS_DIR") + if root: + return Path(root).expanduser() + # deep-swe/ -> evaluation/ -> .amplifier/ -> amplifier-agent/ -> workspace root + return HERE.parents[3] / "evaluation_results" + + +# ---------------------------------------------------------------------- +# Task checkout +# ---------------------------------------------------------------------- + + +def default_tasks_dir() -> Path: + root = os.environ.get("DEEP_SWE_CACHE_DIR") + base = Path(root).expanduser() if root else Path.home() / ".cache" / "deep-swe" + return base / DEEP_SWE_SHA + + +def ensure_tasks(tasks_dir: Path) -> Path: + """Clone deep-swe at the pinned SHA if not already present. Idempotent.""" + checkout = tasks_dir + tasks = checkout / "tasks" + if tasks.is_dir(): + head = _git(checkout, "rev-parse", "HEAD", check=False) + if head == DEEP_SWE_SHA: + return tasks + print(f"Task checkout at {checkout} is at {head}, re-fetching {DEEP_SWE_SHA}...") + + checkout.mkdir(parents=True, exist_ok=True) + if not (checkout / ".git").is_dir(): + _git(checkout, "init", "-q") + _git(checkout, "remote", "add", "origin", DEEP_SWE_REPO) + print(f"Fetching deep-swe@{DEEP_SWE_SHA[:8]} into {checkout} ...") + _git(checkout, "fetch", "-q", "--depth", "1", "origin", DEEP_SWE_SHA) + _git(checkout, "checkout", "-q", "FETCH_HEAD") + if not tasks.is_dir(): + die(f"deep-swe checkout at {checkout} has no tasks/ directory.") + return tasks + + +def _git(cwd: Path, *args: str, check: bool = True) -> str: + proc = subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True, check=False) + if check and proc.returncode != 0: + die(f"git {' '.join(args)} failed in {cwd}:\n{proc.stderr.strip()}") + return proc.stdout.strip() + + +def list_task_names(tasks: Path) -> list[str]: + return sorted(p.name for p in tasks.iterdir() if p.is_dir()) + + +# ---------------------------------------------------------------------- +# Preflight +# ---------------------------------------------------------------------- + + +def preflight(require_docker: bool = True) -> None: + pier = shutil.which("pier") + if not pier: + die(f"`pier` is not on PATH. Install it with:\n {PIER_INSTALL_CMD}") + check_pier_is_git_build(pier) + + if not os.environ.get("ANTHROPIC_API_KEY"): + die("ANTHROPIC_API_KEY is not set. Export it before running.") + + if require_docker: + if not shutil.which("docker"): + die("`docker` is not on PATH. deep-swe tasks run in Docker containers.") + proc = subprocess.run(["docker", "info"], capture_output=True, text=True, check=False) + if proc.returncode != 0: + die("Docker daemon is not reachable. Start Docker and retry.") + + +def check_pier_is_git_build(pier_path: str) -> None: + """PyPI datacurve-pier==0.3.0 lacks [[verifier.collect]] hooks. + + Without them `model.patch` is never produced and EVERY task silently scores + zero. Both builds self-report version "0.3.0", so probe for the feature. + """ + venv_python = Path(pier_path).resolve().parent / "python" + proc = subprocess.run( + [ + str(venv_python), + "-c", + "import importlib.util;s=importlib.util.find_spec('pier');print(s.origin or '')", + ], + capture_output=True, + text=True, + check=False, + ) + origin = proc.stdout.strip() + if proc.returncode != 0 or not origin: + die( + "Could not locate the installed `pier` package to verify the build.\n" + f"Reinstall the known-good build:\n {PIER_INSTALL_CMD}" + ) + + trial_py = Path(origin).parent / "trial" / "trial.py" + if not trial_py.exists() or "_run_collect_hooks" not in trial_py.read_text(encoding="utf-8"): + die( + "Installed `pier` is the PyPI build: it has no [[verifier.collect]] hook\n" + "support, so model.patch is never produced and every task silently\n" + "scores 0. Install the pinned git build:\n" + f" {PIER_INSTALL_CMD}" + ) + + +# ---------------------------------------------------------------------- +# Command construction +# ---------------------------------------------------------------------- + + +def resolve_task_selection(args: argparse.Namespace, available: list[str]) -> list[str]: + """Resolve the task list ONCE, host-side, before any agent runs. + + WHY NOT `--n-tasks`/`--sample-seed`: those are pier flags, and this runner + invokes pier once per agent in a separate process. Delegating the sampling + would mean each arm draws its own subset, and any difference in pier's + sampling -- version, task-dir ordering, implementation change -- yields arms + graded on DIFFERENT tasks while every summary still lines them up + side by side. That failure is invisible in the output. + + Resolving here and emitting explicit `--include-task-name` for every arm + makes the identical-subset property structural rather than assumed, and + `run-manifest.json` records exactly which tasks ran. + """ + if args.tasks: + missing = [t for t in args.tasks if t not in set(available)] + if missing: + die( + f"unknown task(s): {', '.join(missing)}.\n" + " Run `python run.py --list-tasks` to see valid ids." + ) + return list(args.tasks) + + if args.n_tasks > len(available): + die(f"-n {args.n_tasks} exceeds the {len(available)} tasks in the checkout.") + + # Sorted input + explicit seed: the same seed always yields the same subset, + # on any machine, independent of filesystem ordering. + return sorted(random.Random(args.seed).sample(sorted(available), args.n_tasks)) + + +def resolve_pins(agents: list[str], *, enabled: bool) -> dict[str, dict[str, str]]: + """Resolve each pinnable arm's moving ref to a commit SHA, once per run. + + Returns {agent: {kwarg: "git+@[#subdirectory=...]"}}. An arm with + nothing to pin (opencode-vanilla) is simply absent. + + Never fabricates: if `git ls-remote` fails for a ref, that ref is left + unpinned and the reason is printed, so the manifest shows a moving ref + rather than a SHA that was never verified. + """ + if not enabled: + return {} + pins: dict[str, dict[str, str]] = {} + for agent in agents: + for kwarg, url in PINNABLE_REFS.get(agent, {}).items(): + base, _, fragment = url.partition("#") + proc = subprocess.run( + ["git", "ls-remote", base, "HEAD"], + capture_output=True, + text=True, + check=False, + ) + sha = proc.stdout.split("\t", 1)[0].strip() if proc.returncode == 0 else "" + if len(sha) != 40: + print( + f" WARNING: could not resolve {base} to a commit " + f"({(proc.stderr or 'no SHA in output').strip()[:120]}); " + f"{agent}.{kwarg} stays on its moving default.", + file=sys.stderr, + ) + continue + suffix = f"#{fragment}" if fragment else "" + pins.setdefault(agent, {})[kwarg] = f"git+{base}@{sha}{suffix}" + return pins + + +def write_run_manifest( + args: argparse.Namespace, + agents: list[str], + tasks: list[str], + pins: dict[str, dict[str, str]], +) -> None: + """Record what this run actually pinned and selected, next to its results. + + This is the file that makes a run reproducible and auditable after the fact: + the task subset, the seed that produced it, the deep-swe checkout SHA, and + the exact agent refs each arm installed. + """ + manifest = { + "run_timestamp": RUN_TIMESTAMP, + "deep_swe_sha": DEEP_SWE_SHA, + "pier_sha": PIER_GIT_SHA, + "model": args.model, + "agents": agents, + "tasks": tasks, + "task_selection": { + "mode": "explicit" if args.tasks else "sample", + "n_tasks": args.n_tasks, + "seed": args.seed, + "identical_across_agents": True, + }, + "agent_timeout_multiplier": args.agent_timeout_multiplier, + "n_concurrent": args.n_concurrent, + "pins": pins or None, + "local_source": args.local_source, + } + path = args.jobs_dir / "run-manifest.json" + path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + print(f"\nwrote {path}") + + +def build_command(args: argparse.Namespace, agent: str, tasks: Path) -> list[str]: + cmd = [ + "pier", + "run", + "--agent-import-path", + AGENTS[agent], + "--model", + args.model, + "--path", + str(tasks), + "--jobs-dir", + str(args.jobs_dir), + "--job-name", + f"{args.job_name}-{agent}", + "-y", + ] + # Every arm gets the SAME explicit task list, resolved once in main(). pier's + # own --n-tasks/--sample-seed are deliberately never used: see + # resolve_task_selection. + for task in args.selected_tasks: + cmd += ["--include-task-name", task] + if args.agent_timeout_multiplier is not None: + cmd += ["--agent-timeout-multiplier", str(args.agent_timeout_multiplier)] + if args.n_concurrent is not None: + cmd += ["--n-concurrent", str(args.n_concurrent)] + for kwarg, ref in (args.pins or {}).get(agent, {}).items(): + cmd += ["--ak", f"{kwarg}={ref}"] + if args.local_source: + cmd += ["--ak", f"local_source={args.local_source}"] + cmd += args.pier_arg or [] + return cmd + + +# ---------------------------------------------------------------------- +# Summary +# ---------------------------------------------------------------------- + + +# Substrings that mean the agent process died rather than merely scored badly. +# pier reports such trials as n_errored_trials=0 when the process still exited 0, +# so "solved nothing" is otherwise indistinguishable from "died on turn 3". +CRASH_MARKERS = ("[amplifier-agent error:",) + +# Markers matched only at the START of a line. `Error:` is too generic to search +# for anywhere in a line -- an agent legitimately printing compiler output would +# trip it -- but opencode's fatal message always begins one. +CRASH_LINE_PREFIXES = ("Error:",) + + +def _read_json(path: Path) -> dict | None: + """Read a JSON object, or None if absent/unreadable/not an object.""" + try: + data = json.loads(path.read_text(encoding="utf-8")) + except Exception: # noqa: BLE001 - summary must never crash the run + return None + return data if isinstance(data, dict) else None + + +def find_crash_markers(trial_dir: Path) -> list[str]: + """Return the distinct agent.log lines that indicate an early crash.""" + log = trial_dir / "agent" / "agent.log" + try: + text = log.read_text(encoding="utf-8", errors="replace") + except Exception: # noqa: BLE001 - missing/unreadable log is not a crash signal + return [] + hits: list[str] = [] + for line in text.splitlines(): + stripped = line.strip() + matched = any(marker in line for marker in CRASH_MARKERS) or stripped.startswith( + CRASH_LINE_PREFIXES + ) + if matched: + trimmed = stripped[:200] + if trimmed not in hits: + hits.append(trimmed) + return hits + + +def _num(value: object) -> float | None: + """Return value as a float if it is a real number, else None.""" + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _fmt(value: float | None, spec: str) -> str: + return "n/a" if value is None else format(value, spec) + + +# ---------------------------------------------------------------------- +# Trial timings (one measurement: the adapter's monotonic agent clock) +# ---------------------------------------------------------------------- +# +# The ONLY duration reported here is `agent_run_s`, measured by the adapter with +# `time.monotonic()`. pier derives its own durations from the wall clock, which +# can step backward under NTP correction; `time.monotonic()` cannot. Those +# pier timings are still written to the trial result.json, they are simply not +# reported here -- do not reinstate them, not even as a fallback. + + +def trial_timings(result: dict) -> dict[str, object]: + """Derive the agent duration and the timeout flag.""" + exc_info = result.get("exception_info") + exc_type = exc_info.get("exception_type") if isinstance(exc_info, dict) else None + agent_result = result.get("agent_result") + metadata = agent_result.get("metadata") if isinstance(agent_result, dict) else None + agent_run_s = _num(metadata.get("agent_run_s")) if isinstance(metadata, dict) else None + return { + # The only duration we report. Adapter-measured with time.monotonic(). + # pier's wall-clock-derived durations (agent_execution, + # environment_setup, verifier, ...) are deliberately not read here. + "agent_run_s": agent_run_s, + "timed_out": exc_type == "AgentTimeoutError", + } + + +def _agent_field(timings: dict[str, object]) -> str: + """`agent=1037s`, from the adapter's monotonic measurement. + + There is no fallback: a trial without that measurement (any trial predating + it) reports `agent=n/a` rather than a number derived from a clock we do not + trust. + """ + agent_s = timings.get("agent_run_s") + if not isinstance(agent_s, (int, float)): + return "agent=n/a" + return f"agent={agent_s:.0f}s" + + +def summarize_trial(trial_dir: Path) -> tuple[str, float | None]: + """Return (summary line, cost_usd or None) for one trial directory.""" + result = _read_json(trial_dir / "result.json") or {} + reward = _read_json(trial_dir / "verifier" / "reward.json") or {} + # Fall back to the in-result copy when verifier/reward.json is absent. + if not reward: + reward = (result.get("verifier_result") or {}).get("rewards") or {} + + name = result.get("task_name") or trial_dir.name + agent_result = result.get("agent_result") or {} + cost = _num(agent_result.get("cost_usd")) + token_parts = [ + _num(agent_result.get(key)) + for key in ("n_input_tokens", "n_cache_tokens", "n_output_tokens") + ] + tokens = None if all(t is None for t in token_parts) else sum(t or 0 for t in token_parts) + + f2p_passed = _fmt(_num(reward.get("f2p_passed")), "g") + f2p_total = _fmt(_num(reward.get("f2p_total")), "g") + fields = [ + f"reward={_fmt(_num(reward.get('reward')), 'g')}", + f"f2p={f2p_passed}/{f2p_total}", + f"partial={_fmt(_num(reward.get('partial')), '.3f')}", + "cost=n/a" if cost is None else f"cost=${cost:.2f}", + f"tokens={_fmt(tokens, ',.0f')}", + ] + timings = trial_timings(result) + fields.append(_agent_field(timings)) + + exc_info = result.get("exception_info") or {} + exc_type = exc_info.get("exception_type") + if timings.get("timed_out"): + fields.append("TIMEOUT") + if exc_type: + fields.append(f"EXC: {exc_type}") + line = f" {name:<50} {' '.join(fields)}" + return line, cost + + +def summarize(jobs_dir: Path, job_names: list[str]) -> None: + """Print per-trial results read from the trial dirs. + + Deliberately NOT read from the job-level result.json: pier writes that file + with `exclude={"trial_results"}` at every call site, so the key is never + present. The per-trial `verifier/reward.json` and `result.json` are the real + source of truth. + + Never raises -- this runs after a paid run and must not destroy the output. + """ + try: + print("\n" + "=" * 64) + print("SUMMARY (reward is binary; partial is the dev signal)") + print("=" * 64) + for job_name in job_names: + job_dir = jobs_dir / job_name + print(f"\n{job_name}") + if not job_dir.is_dir(): + print(" no job dir (run errored or was cancelled)") + continue + trials = sorted( + p + for p in job_dir.iterdir() + if p.is_dir() and ((p / "result.json").exists() or (p / "verifier").is_dir()) + ) + if not trials: + print(" no trials recorded") + continue + total_cost = 0.0 + saw_cost = False + for trial_dir in trials: + line, cost = summarize_trial(trial_dir) + print(line) + if cost is not None: + saw_cost = True + total_cost += cost + for hit in find_crash_markers(trial_dir): + print(f" !! CRASHED: {trial_dir.name}: {hit}") + total = f"${total_cost:.2f}" if saw_cost else "n/a" + print(f" {'TOTAL COST':<50} {total}") + except Exception as exc: # noqa: BLE001 - summary must never crash the run + print(f"could not summarize results: {exc}", file=sys.stderr) + + +# ---------------------------------------------------------------------- +# CLI +# ---------------------------------------------------------------------- + + +def die(message: str) -> None: + print(f"error: {message}", file=sys.stderr) + raise SystemExit(1) + + +def parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser( + prog="run.py", + description="Run Amplifier agents against the deep-swe benchmark via pier.", + epilog=( + "Extra raw pier args: repeat --pier-arg.\n e.g. --pier-arg --max-retries --pier-arg 2" + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument("--agents", help="comma-separated agent short names") + parser.add_argument("--tasks", help="comma-separated deep-swe task ids") + parser.add_argument( + "-n", + "--n-tasks", + type=int, + help="run a deterministic sample of N tasks (same N for every agent)", + ) + parser.add_argument( + "--seed", + type=int, + default=0, + help="sample seed used with -n (default: 0). The same seed always " + "selects the same tasks, so two runs are directly comparable.", + ) + parser.add_argument( + "--n-concurrent", + type=int, + default=None, + help="trials pier runs in parallel within one agent (pier --n-concurrent). " + "Unset means pier's default (sequential). Each trial is a container with " + "its own cpu/memory budget from the task's task.toml.", + ) + parser.add_argument( + "--agent-timeout-multiplier", + type=float, + default=DEFAULT_AGENT_TIMEOUT_MULTIPLIER, + help=f"multiplier on each task's agent timeout_sec " + f"(default: {DEFAULT_AGENT_TIMEOUT_MULTIPLIER}). A timed-out trial scores 0, " + f"which is indistinguishable from a capability failure, so headroom matters.", + ) + parser.add_argument( + "--no-pin", + action="store_true", + help="do NOT resolve agent refs to commit SHAs; install from moving " + "branches instead. Faster to start, but a multi-hour run can span " + "upstream commits and grade different tasks against different code.", + ) + parser.add_argument( + "--local-source", help="path to a local checkout to install instead of the pinned ref" + ) + parser.add_argument("--model", default=DEFAULT_MODEL) + parser.add_argument( + "--jobs-dir", + type=Path, + default=default_results_root() / RUN_TIMESTAMP, + help="output dir for this run (default: /evaluation_results/)", + ) + parser.add_argument("--tasks-dir", type=Path, default=None, help="deep-swe checkout cache dir") + # The results dir is already timestamped, so the job name only has to + # separate agents within one run. pier raises FileExistsError on collision. + parser.add_argument("--job-name", default="deepswe") + parser.add_argument("--list-agents", action="store_true") + parser.add_argument("--list-tasks", action="store_true") + parser.add_argument("--dry-run", action="store_true", help="print pier commands, do not run") + parser.add_argument( + "--pier-arg", action="append", help="raw arg forwarded to pier (repeatable)" + ) + return parser.parse_args(argv) + + +def main(argv: list[str]) -> int: + args = parse_args(argv) + + if args.list_agents: + for name, path in AGENTS.items(): + local = " (supports --local-source)" if name in LOCAL_SOURCE_AGENTS else "" + print(f"{name:<28} {path}{local}") + return 0 + + tasks_dir = args.tasks_dir or default_tasks_dir() + + if args.list_tasks: + for name in list_task_names(ensure_tasks(tasks_dir)): + print(name) + return 0 + + if not args.agents: + die(f"--agents is required. Valid names: {', '.join(AGENTS)}") + agents = [a.strip() for a in args.agents.split(",") if a.strip()] + unknown = [a for a in agents if a not in AGENTS] + if unknown: + die(f"unknown agent(s): {', '.join(unknown)}. Valid names: {', '.join(AGENTS)}") + + args.tasks = [t.strip() for t in (args.tasks or "").split(",") if t.strip()] + + if not args.tasks and args.n_tasks is None: + die( + "a task selection is required -- refusing to run the full 113-task matrix.\n" + f" 113 tasks x {len(agents)} agent(s) is hours-to-days of runtime and\n" + " hundreds of dollars in API spend. Use --tasks or -n ." + ) + + if args.local_source: + bad = [a for a in agents if a not in LOCAL_SOURCE_AGENTS] + if bad: + die( + f"--local-source is not supported by: {', '.join(bad)}.\n" + f" Supported: {', '.join(sorted(LOCAL_SOURCE_AGENTS))}" + ) + src = Path(args.local_source).expanduser().resolve() + if not src.is_dir(): + die(f"--local-source path does not exist: {src}") + args.local_source = str(src) + + preflight(require_docker=not args.dry_run) + tasks = ensure_tasks(tasks_dir) + + # Resolve ONCE, then hand the identical explicit list to every agent. + args.selected_tasks = resolve_task_selection(args, list_task_names(tasks)) + print(f"\ntask selection ({len(args.selected_tasks)} task(s), identical for every agent):") + for name in args.selected_tasks: + print(f" {name}") + if args.n_tasks is not None: + print(f" (deterministic sample, seed={args.seed})") + + args.pins = resolve_pins(agents, enabled=not args.no_pin) + if args.pins: + print("\nagent refs pinned for this run:") + for agent, refs in args.pins.items(): + for kwarg, ref in refs.items(): + print(f" {agent}.{kwarg} = {ref}") + elif not args.no_pin: + print("\nno agent refs required pinning (or none could be resolved).") + + args.jobs_dir = Path(args.jobs_dir).expanduser().resolve() + # A dry run prints commands only; creating the timestamped results dir here + # would litter evaluation_results/ with empty dirs. + if not args.dry_run: + args.jobs_dir.mkdir(parents=True, exist_ok=True) + + if not args.dry_run: + write_run_manifest(args, agents, args.selected_tasks, args.pins) + + job_names: list[str] = [] + failures = 0 + for agent in agents: + cmd = build_command(args, agent, tasks) + job_names.append(f"{args.job_name}-{agent}") + print("\n" + "=" * 64) + print(f"AGENT: {agent}") + print("=" * 64) + print(" ".join(cmd)) + if args.dry_run: + continue + proc = subprocess.run(cmd, check=False) + if proc.returncode != 0: + failures += 1 + print(f"pier run failed for {agent} (exit {proc.returncode})", file=sys.stderr) + + if args.dry_run: + return 0 + + summarize(args.jobs_dir, job_names) + return 1 if failures else 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/__init__.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/__init__.py new file mode 100644 index 00000000..78083c4c --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/__init__.py @@ -0,0 +1,20 @@ +"""Amplifier agent adapters for the deep-swe benchmark. + +Short CLI names -> pier `--agent-import-path` values. +""" + +AGENTS = { + "amplifier-agent": "deepswe_agents.amplifier_agent:AmplifierAgent", + "amplifier-foundation": "deepswe_agents.amplifier_foundation:AmplifierFoundationAgent", + "opencode-amplifier-agent": "deepswe_agents.opencode_amplifier:OpencodeAmplifierAgent", + "opencode-vanilla": "deepswe_agents.opencode_vanilla:OpencodeVanillaAgent", +} + +#: Agents that accept `--local-source` (they install an Amplifier component). +LOCAL_SOURCE_AGENTS = { + "amplifier-agent", + "amplifier-foundation", + "opencode-amplifier-agent", +} + +__all__ = ["AGENTS", "LOCAL_SOURCE_AGENTS"] diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_agent.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_agent.py new file mode 100644 index 00000000..04e40de6 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_agent.py @@ -0,0 +1,113 @@ +"""amplifier-agent standalone CLI, driven directly.""" + +from __future__ import annotations + +import json +import shlex +from typing import Any + +from pier.environments.base import BaseEnvironment +from pier.models.agent.install import InstallStep + +from deepswe_agents.base import ( + DEFAULT_AMPLIFIER_AGENT_REF, + UV_PRELUDE, + AmplifierBaseAgent, + _as_bool, +) + +HOST_CONFIG_PATH = "/root/host-config.json" + +# Explicit session id. WITHOUT `--session-id` the runtime mints a telemetry-only +# `ephemeral-` id and writes NO persistence at all -- no transcript.jsonl, +# no metadata.json, no audits/. A fixed value is safe and deterministic: every +# trial gets a fresh container. `--fresh` is deliberately NOT passed -- it is a +# no-op in a fresh container and would destroy the prior trajectory if pier ever +# retried in place. +SESSION_ID = "deepswe-trial" + + +class AmplifierAgent(AmplifierBaseAgent): + LOCAL_SOURCE_PACKAGE = "amplifier-agent" + VERSION_BINARY = "amplifier-agent" + + # Session state (events.jsonl with per-response token usage and cost) lives + # under the agent's default home. Pulled to /agent/sessions/ so the + # metrics pass can read it host-side. + SESSION_DIRS = ("/root/.amplifier-agent/state/workspaces",) + + def __init__( + self, + *args: Any, + amplifier_agent_ref: str = DEFAULT_AMPLIFIER_AGENT_REF, + raw_llm_payloads: Any = False, + **kwargs: Any, + ): + super().__init__(*args, **kwargs) + self._amplifier_agent_ref = amplifier_agent_ref + # Reachable as `--ak raw_llm_payloads=true`; pier hands it over as a + # string, hence the coercion rather than a bare truthiness test. + self._raw_llm_payloads = _as_bool(raw_llm_payloads) + + @staticmethod + def name() -> str: + return "amplifier-agent" + + async def setup(self, environment: BaseEnvironment) -> None: + await super().setup(environment) + # Always state the capture mode: a trial log must never be ambiguous + # about whether its events.jsonl carries full prompts. + self.logger.info( + "[%s] raw LLM payload capture: %s", + self.name(), + "ON (debug.rawLlmPayloads)" if self._raw_llm_payloads else "off", + ) + + def _host_config(self) -> str: + config: dict[str, Any] = { + "approval": {"mode": "yes"}, + "provider": { + "module": "anthropic", + "config": {"default_model": self.model}, + }, + } + if self._raw_llm_payloads: + # Adds `data.raw` (full messages/system/tools/model/max_tokens) to + # every `llm:request` event. Must be a real JSON boolean -- a string + # is rejected by the runtime. OPT-IN because it multiplies the size + # of events.jsonl on a run that makes hundreds of LLM calls. + config["debug"] = {"rawLlmPayloads": True} + return json.dumps(config) + + def agent_install_steps(self) -> list[InstallStep]: + return [ + *UV_PRELUDE, + InstallStep( + user="root", + run=( + 'export PATH="$HOME/.local/bin:$PATH"\n' + f"uv tool install --reinstall --force --from " + f'"{self._amplifier_agent_ref}" amplifier-agent\n' + "amplifier-agent-post-install || true" + ), + ), + InstallStep( + user="root", + run=f"echo {shlex.quote(self._host_config())} > {HOST_CONFIG_PATH}", + ), + InstallStep( + user="root", + run='export PATH="$HOME/.local/bin:$PATH"; amplifier-agent --version', + ), + ] + + def run_command(self, instruction_path: str) -> str: + # `--session-id` is what makes the run persist a trajectory at all (see + # SESSION_ID). `--output json` appends the final envelope -- including + # metadata.durationMs -- to agent.log; it does NOT suppress the + # human-readable `[usage]` lines, so nothing is lost by adding it. + return ( + f"amplifier-agent run -y --config {HOST_CONFIG_PATH} " + f"--session-id {shlex.quote(SESSION_ID)} --output json " + f'"$(cat {instruction_path})"' + ) diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_foundation.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_foundation.py new file mode 100644 index 00000000..27182bba --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/amplifier_foundation.py @@ -0,0 +1,230 @@ +"""Full Amplifier foundation stack (`amplifier run`) with the anchors bundle.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from pier.environments.base import BaseEnvironment +from pier.models.agent.install import InstallStep +from pier.models.agent.network import NetworkAllowlist + +from deepswe_agents.base import UV_PRELUDE, WORKDIR, AmplifierBaseAgent + +SETTINGS_PATH = "$HOME/.amplifier/settings.yaml" + +# Env vars used to smuggle provider config into the heredoc without it appearing +# in argv. The key must never be logged; the base URL rides along for symmetry. +API_KEY_ENV = "AMPLIFIER_BENCH_API_KEY" +BASE_URL_ENV = "AMPLIFIER_BENCH_BASE_URL" + +# The bundle under benchmark. `anchors` is also the CLI's built-in default when +# no `bundle.active` is configured, but we pass it EXPLICITLY on the run command: +# `--bundle` has the highest precedence, so a stray `/app/.amplifier/settings.yaml` +# in a task repo cannot silently swap the stack out from under the benchmark. +ANCHORS_BUNDLE = ( + "git+https://github.com/microsoft/amplifier-foundation@main" + "#subdirectory=bundles/anchors/bundle.md" +) + +#: The CLI package. Overridable so a run can pin an exact commit. +DEFAULT_AMPLIFIER_REF = "git+https://github.com/microsoft/amplifier" + + +class AmplifierFoundationAgent(AmplifierBaseAgent): + LOCAL_SOURCE_PACKAGE = "amplifier" + VERSION_BINARY = "amplifier" + + # Trajectory + metrics source. `amplifier run` writes one session tree per + # project slug, and the slug is the cwd with separators replaced, so a + # container running in /app yields `/root/.amplifier/projects/-app/`. + # + # We collect the whole `projects/` dir rather than just `-app`: the install + # performs no LLM call, so `-app` is the only project that can exist, and + # pulling the parent survives a slug surprise instead of silently yielding + # zero metrics. `_log_session_provenance` reports what actually landed. + # + # Under `anchors` a single session writes events.jsonl TWICE (hooks-logging + # at the session root, hook-context-intelligence under + # `context-intelligence/`). Both are collected on purpose; `parse_events` + # de-duplicates by the provider's response id, so cost is not doubled. + SESSION_DIRS = ("/root/.amplifier/projects",) + + def __init__( + self, + *args: Any, + amplifier_ref: str = DEFAULT_AMPLIFIER_REF, + anchors_ref: str = ANCHORS_BUNDLE, + **kwargs: Any, + ): + """Both refs are injectable so a run can pin exact commits. + + Reachable as `--ak amplifier_ref=...` / `--ak anchors_ref=...`. Left at + the defaults this arm tracks the moving branch, which is fine for a + one-off but silently invalidates a long multi-task run: upstream can + move between the first task and the last, so different tasks would be + graded against different agent code. `run.py --pin` resolves both to + commit SHAs once per run and records them. + """ + super().__init__(*args, **kwargs) + self._amplifier_ref = amplifier_ref + self._anchors_ref = anchors_ref + + @staticmethod + def name() -> str: + return "amplifier-foundation" + + def agent_install_steps(self) -> list[InstallStep]: + return [ + *UV_PRELUDE, + InstallStep( + user="root", + run=( + 'export PATH="$HOME/.local/bin:$PATH"\n' + f"uv tool install {self._amplifier_ref}\n" + "amplifier --version" + ), + ), + InstallStep( + user="root", + # Pre-resolve the anchors composition at BUILD time so the timed + # run does not include ~30 module git clones. `bundle show` + # downloads every module it resolves and needs no API key, which + # is why it (not a warm-up `amplifier run`) is used here: a + # warm-up would need the key baked into a layer AND would write a + # second session that pollutes the token/cost numbers. + # + # Best effort: a resolution failure here must not fail the build, + # since the run can still resolve behind the egress proxy. + run=( + 'export PATH="$HOME/.local/bin:$PATH"\n' + f"amplifier bundle show '{self._anchors_ref}' >/dev/null 2>&1 || " + 'echo "anchors pre-warm failed; modules will resolve at run time" >&2' + ), + ), + ] + + def network_allowlist(self) -> NetworkAllowlist: + # The provider module is declared by source in settings.yaml, which is + # only written at runtime, so amplifier resolves it from git on first + # run -- behind the egress proxy. Without these, the run cannot start. + base = super().network_allowlist() + return NetworkAllowlist( + domains=[*base.domains, "github.com", "pypi.org", "files.pythonhosted.org"] + ) + + async def setup(self, environment: BaseEnvironment) -> None: + await super().setup(environment) + await self._write_settings(environment) + + async def _write_settings(self, environment: BaseEnvironment) -> None: + """Write settings.yaml at RUNTIME, never at install time. + + Install steps become Docker layers: baking the API key there would put + the secret in the image and make the install fingerprint key-dependent + (defeating layer caching across runs). + """ + env = self.agent_env() + api_key = self._get_env("ANTHROPIC_API_KEY") or "" + env[API_KEY_ENV] = api_key + # provider-anthropic reads base_url from CONFIG ONLY -- it has no direct + # ANTHROPIC_BASE_URL fallback at runtime. Omitting this key was silently + # sending this arm to api.anthropic.com while the other arms honored the + # proxy, i.e. benchmarking a different endpoint. + base_url = self._get_env("ANTHROPIC_BASE_URL") or "https://api.anthropic.com" + env[BASE_URL_ENV] = base_url + + # Unquoted heredoc so ${API_KEY_ENV} expands in the container -- the key + # never appears in argv or in the logged command. + settings = ( + "config:\n" + " providers:\n" + " - module: provider-anthropic\n" + " source: git+https://github.com/microsoft/" + "amplifier-module-provider-anthropic@main\n" + " config:\n" + f" api_key: ${{{API_KEY_ENV}}}\n" + f" base_url: ${{{BASE_URL_ENV}}}\n" + f" default_model: {self.model}\n" + " enable_1m_context: 'true'\n" + " enable_prompt_caching: 'true'\n" + " priority: 1\n" + ) + # NOTE: no routing.matrix key -- it re-introduces role-based fan-out to opus. + # NOTE: no `bundle:` key -- the run command pins the bundle explicitly. + command = ( + 'mkdir -p "$HOME/.amplifier" && ' + f'cat > "{SETTINGS_PATH}" < str: + """Single-shot, non-interactive, on a pinned bundle. + + `--mode single` is already the CLI default, and is stated anyway so a + future default flip cannot turn the benchmark into an interactive session + that hangs until the timeout. `--output-format json` puts a machine- + readable `{"status": ...}` envelope at the end of agent.log. + + No `--model` flag: it requires `--provider` alongside it, and the model is + already pinned by `default_model` in settings.yaml. + """ + return ( + f"amplifier run --bundle '{self._anchors_ref}' " + f"--mode single --output-format json " + f'"$(cat {instruction_path})"' + ) + + def populate_context_post_run(self, context) -> None: # type: ignore[no-untyped-def] + super().populate_context_post_run(context) + self._log_session_provenance() + + def _log_session_provenance(self) -> None: + """Report which project/bundle actually produced the collected sessions. + + This is the cheap, direct proof that the trial exercised foundation with + the anchors bundle in the task workspace -- rather than, say, a stray + project slug whose events would be counted as if they were the task's. + Never raises: provenance logging must not fail a good trial. + """ + try: + projects = self.logs_dir / "sessions" / "projects" + if not projects.is_dir(): + self.logger.warning( + "No session projects dir collected; token/cost metrics will be not_available." + ) + return + slugs = sorted(p.name for p in projects.iterdir() if p.is_dir()) + expected = "-" + WORKDIR.strip("/").replace("/", "-") + unexpected = [s for s in slugs if s != expected] + self.logger.info(f"session projects collected: {slugs} (expected {expected!r})") + if unexpected: + self.logger.warning( + f"Session projects OTHER than {expected!r} were collected: " + f"{unexpected}. Their events are included in the token/cost " + f"totals and may not belong to this task." + ) + bundles = { + str(meta.get("bundle")) + for path in projects.rglob("metadata.json") + if isinstance(meta := self._read_json(path), dict) and meta.get("bundle") + } + if bundles: + self.logger.info(f"bundle(s) recorded in session metadata: {sorted(bundles)}") + if not any("anchors" in b for b in bundles): + self.logger.warning( + f"No collected session recorded the anchors bundle: {sorted(bundles)}" + ) + except Exception as exc: # noqa: BLE001 - provenance must never fail a trial + self.logger.warning(f"Could not log session provenance: {exc!r}") + + @staticmethod + def _read_json(path: Path) -> object: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/base.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/base.py new file mode 100644 index 00000000..0ef59023 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/base.py @@ -0,0 +1,608 @@ +"""Shared base class for the Amplifier-family deep-swe agents. + +Everything here exists to satisfy three hard constraints of the deep-swe/pier +runner: + +1. Work is scored as ``git diff HEAD`` in ``/app``. Uncommitted + work scores ZERO, so we both instruct the agent to commit and run a + belt-and-braces fallback commit ourselves. +2. ``environment.exec`` runs ``bash -c``, not a login shell. ``/etc/profile.d`` + is never sourced, so PATH must be set explicitly on every exec. +3. Instructions are long markdown blobs with quotes/backticks/newlines. They are + uploaded as a file and referenced with ``$(cat ...)`` -- never interpolated + into a shell string. +""" + +from __future__ import annotations + +import asyncio +import json +import shutil +import tempfile +import time +from abc import abstractmethod +from collections.abc import Coroutine +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + +from pier.agents.installed.base import BaseInstalledAgent +from pier.environments.base import BaseEnvironment +from pier.models.agent.context import AgentContext +from pier.models.agent.install import InstallStep +from pier.models.agent.network import NetworkAllowlist + +from deepswe_agents.metrics import ( + find_events_files, + find_opencode_db_files, + normalize_metrics, + normalize_opencode_metrics, +) + +# Container path the task repo lives at. deep-swe grades a git diff of this dir. +WORKDIR = "/app" + +# Explicit PATH for every exec. `bash -c` does not source /etc/profile.d, so the +# uv tool bin dir and the opencode bin dir must be named here or nothing resolves. +CONTAINER_PATH = ( + "/root/.local/bin:/root/.opencode/bin:" + "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" +) + +# Where the task instruction is staged inside the container. +INSTRUCTION_PATH = "/tmp/pier-instruction.txt" + +# amplifier-agent install ref, shared by every arm that installs it. Tracks the +# default branch so we benchmark the agent as it actually ships. Override for a +# reproducible run with: +# --ak amplifier_agent_ref=git+https://github.com/microsoft/amplifier-agent@ +DEFAULT_AMPLIFIER_AGENT_REF = "git+https://github.com/microsoft/amplifier-agent" + +# Staging root for `--ak local_source=...` uploads. +LOCAL_SOURCE_ROOT = "/src" + +# The instruction reaches the agent EXACTLY as deep-swe wrote it. Do not append +# scoring hints, commit reminders, or any other scaffolding here. +# +# deep-swe already tells the agent to commit: all 113 instruction.md files end +# with "IMPORTANT: Please work on this in a new branch from main and commit +# everything when you are done." A harness-added restatement on top of that made +# our numbers incomparable to the public leaderboard, whose scores were all +# produced by mini-swe-agent, which passes the instruction through untouched +# (pier/agents/installed/mini_swe_agent.py: `augmented_instruction = instruction`). +# No other pier adapter augments the prompt either. +# +# Uncommitted work is handled mechanically by the fallback commit below, which +# needs no cooperation from the model and does not change what the model sees. + +# Marker echoed by the fallback commit so trial logs are greppable. +FALLBACK_MARKER = "PIER_AMPLIFIER_FALLBACK_COMMIT" + +# Hard ceiling on the fallback commit itself so a wedged container cannot hang +# teardown forever. +FALLBACK_COMMIT_TIMEOUT_SEC = 120 + +# Hard ceiling on the teardown-time host collection of the session dirs, for +# the same reason and deliberately the same bound as the fallback +# commit: no single teardown step may add more than two minutes to a trial. The +# session tree is a handful of MB of jsonl, so this is generous by an order of +# magnitude -- it exists to bound a WEDGED container, not a slow one. +TEARDOWN_COLLECT_TIMEOUT_SEC = 120 + +# Directories/files never uploaded with `--ak local_source`. `.amplifier` is +# ~971MB in the amplifier-agent checkout and would dominate every trial. +LOCAL_SOURCE_IGNORE = shutil.ignore_patterns( + ".git", + ".venv", + "venv", + "node_modules", + "__pycache__", + "*.pyc", + "*.pyo", + "*.egg-info", + ".mypy_cache", + ".pytest_cache", + ".ruff_cache", + ".tox", + "build", + "dist", + ".DS_Store", + ".amplifier", + ".amplifier-agent", + "target", +) + +# Reusable install preludes. Subclasses opt in from `agent_install_steps()`. +UV_PRELUDE = [ + InstallStep(user="root", run="curl -LsSf https://astral.sh/uv/install.sh | sh"), +] + +OPENCODE_PRELUDE = [ + InstallStep( + user="root", + run=( + "for i in 1 2 3 4 5; do\n" + " curl -fsSL https://opencode.ai/install | VERSION=1.17.20 bash && break\n" + ' echo "opencode install attempt $i failed; retrying in $((i*10))s..." >&2\n' + " sleep $((i*10))\n" + "done\n" + 'export PATH="$HOME/.opencode/bin:$PATH"; opencode --version' + ), + ), +] + + +# Values accepted as "on" for boolean `--ak` kwargs. pier passes every `--ak` +# value as a STRING, so a bare `bool()` would make "false" truthy. +_TRUTHY = frozenset({"true", "1", "yes"}) + + +def _as_bool(value: Any) -> bool: + """Coerce an `--ak` value to bool. Anything not explicitly truthy is False.""" + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in _TRUTHY + return False + + +class AmplifierBaseAgent(BaseInstalledAgent): + """Base for all Amplifier-family agents run against deep-swe tasks.""" + + #: Which installable component `--ak local_source=` replaces. + #: ``None`` means the agent does not support local-source installs. + LOCAL_SOURCE_PACKAGE: str | None = None + + #: Default model when the job does not pass one. + DEFAULT_MODEL = "claude-sonnet-5" + + #: Binary whose ``--version`` identifies the build under benchmark. + VERSION_BINARY: str = "" + + #: Basename of the captured stdout/stderr log. + LOG_FILENAME = "agent.log" + + #: In-container absolute directories pulled out to the host after the run. + #: Each lands at ``/agent/sessions/``. Used for the session + #: state trees (events.jsonl) the token/cost metrics pass reads. + SESSION_DIRS: tuple[str, ...] = () + + #: Which usage store this agent writes, and therefore which parser + #: ``populate_context_post_run`` must use. + #: + #: ``"events"`` Amplifier events.jsonl (amplifier-agent, foundation) + #: ``"opencode_db"`` opencode's SQLite ``session`` table + #: + #: This is a per-agent fact, not a harness-wide one. Hardcoding the + #: events.jsonl route meant every opencode trial searched for a file that + #: agent never writes, found none, and reported all-``not_available`` -- + #: indistinguishable from a collection failure. + METRICS_SOURCE: str = "events" + + def __init__(self, *args: Any, local_source: str | None = None, **kwargs: Any): + super().__init__(*args, **kwargs) + # Self-measured wall clock for the agent COMMAND itself, on a monotonic + # clock so it cannot be skewed by a clock adjustment mid-trial. Set in + # run(); read in populate_context_post_run(). + self._run_started_at: float | None = None + self._run_ended_at: float | None = None + self._local_source: Path | None = None + if local_source: + if not self.LOCAL_SOURCE_PACKAGE: + raise ValueError( + f"Agent '{self.name()}' does not support --ak local_source " + "(LOCAL_SOURCE_PACKAGE is None)." + ) + path = Path(local_source).expanduser().resolve() + if not path.is_dir(): + # Never silently fall back to the git ref -- a run that quietly + # tests the wrong code is worse than a failed run. + raise ValueError(f"local_source path does not exist or is not a directory: {path}") + self._local_source = path + + # ------------------------------------------------------------------ + # Subclass hooks + # ------------------------------------------------------------------ + + @abstractmethod + def agent_install_steps(self) -> list[InstallStep]: + """Install steps baked into the Docker image at build time.""" + + @abstractmethod + def run_command(self, instruction_path: str) -> str: + """Shell command that runs the agent on the instruction at *instruction_path*.""" + + # ------------------------------------------------------------------ + # Model / env helpers + # ------------------------------------------------------------------ + + @property + def model(self) -> str: + """Bare model id, accepting both ``anthropic/claude-sonnet-5`` and ``claude-sonnet-5``.""" + return self._parsed_model_name or self.DEFAULT_MODEL + + def agent_env(self) -> dict[str, str]: + """Env for every exec: explicit PATH plus the Anthropic credentials.""" + base: dict[str, str | None] = { + "PATH": CONTAINER_PATH, + "HOME": "/root", + "ANTHROPIC_API_KEY": self._get_env("ANTHROPIC_API_KEY"), + "ANTHROPIC_BASE_URL": self._get_env("ANTHROPIC_BASE_URL"), + } + return self.build_process_env(base) + + def _wrap(self, command: str) -> str: + """Re-export PATH inline: belt and braces on top of the ``env=`` dict.""" + return f'export PATH="$HOME/.local/bin:$HOME/.opencode/bin:{CONTAINER_PATH}"; {command}' + + def get_version_command(self) -> str | None: + """Record which build was benchmarked. ``_wrap`` supplies the PATH.""" + if not self.VERSION_BINARY: + return None + return self._wrap(f"{self.VERSION_BINARY} --version") + + def install_spec(self): + from pier.models.agent.install import AgentInstallSpec + + return AgentInstallSpec( + agent_name=self.name(), + version=self._version, + steps=self.agent_install_steps(), + ) + + def network_allowlist(self) -> NetworkAllowlist: + domains = [] + base_url = self._get_env("ANTHROPIC_BASE_URL") + if base_url: + host = urlparse(base_url).hostname + if host: + domains.append(host) + domains.append("api.anthropic.com") + if self._local_source: + # The local-source install runs at RUNTIME (behind the egress proxy) + # and still resolves dependencies from PyPI/GitHub. + domains += ["pypi.org", "files.pythonhosted.org", "github.com"] + return NetworkAllowlist(domains=domains) + + def populate_context_post_run(self, context: AgentContext) -> None: + """Fill token/cost accounting from the collected session artifacts. + + Host-side and synchronous: nothing here may await or touch the + container. A field is set only when metrics produced a real number -- + a bogus 0 would silently report a $0 run. + """ + try: + run_s = self._agent_run_s() + if run_s is not None: + # Merge: never clobber metadata another layer may have set. + existing = getattr(context, "metadata", None) + metadata = dict(existing) if isinstance(existing, dict) else {} + metadata["agent_run_s"] = run_s + context.metadata = metadata + + if self.METRICS_SOURCE == "opencode_db": + sources = find_opencode_db_files(self.logs_dir) + # workspace_dir must be THIS harness's workdir. The parser + # filters the opencode `session` table by its absolute + # `directory` column and has NO fallback: a mismatch matches no + # session, so the run is reported as not_available with a note + # naming the mismatch, rather than silently summing unrelated + # sessions. + record = normalize_opencode_metrics( + sources, + source=self.name(), + workspace_dir=WORKDIR, + ) + source_label = "opencode.db" + else: + sources = find_events_files(self.logs_dir) + record = normalize_metrics(sources, source=self.name()) + source_label = "events.jsonl" + + def number(key: str) -> float | None: + value = record.get(key) + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return value + + input_tokens = number("input_tokens") + output_tokens = number("output_tokens") + cost_usd = number("cost_usd") + # AgentContext has one cache field; metrics splits read from write. + cache_read, cache_write = number("cache_read"), number("cache_write") + cache_tokens = ( + None + if cache_read is None and cache_write is None + else (cache_read or 0) + (cache_write or 0) + ) + + if input_tokens is not None: + context.n_input_tokens = int(input_tokens) + if output_tokens is not None: + context.n_output_tokens = int(output_tokens) + if cache_tokens is not None: + context.n_cache_tokens = int(cache_tokens) + if cost_usd is not None: + context.cost_usd = float(cost_usd) + + # Drop the inferred figure entirely. metrics.py's + # `agent_wallclock_s` is the earliest-to-latest EVENT timestamp + # span, measured on the CONTAINER clock, and only ever a floor on + # LLM-active time -- never the command duration. `agent_run_s` + # supersedes it, so the record carries exactly ONE duration. + # Per-event timing is still available by reading events.jsonl. + record.pop("agent_wallclock_s", None) + # `agent_run_s` is this adapter's own `time.monotonic()` measurement + # of the agent command; pier's wall-clock durations are not read. + record["agent_run_s"] = run_s if run_s is not None else "not_available" + + # The full record carries what AgentContext has no slot for: + # llm_responses, wallclock, de-duplication notes, source files. + self.logs_dir.mkdir(parents=True, exist_ok=True) + (self.logs_dir / "metrics.json").write_text( + json.dumps(record, indent=2), encoding="utf-8" + ) + self.logger.info( + f"metrics: {len(sources)} {source_label} file(s), " + f"cost_usd={record.get('cost_usd')}, " + f"llm_responses={record.get('llm_responses')}, " + f"agent_run_s={record.get('agent_run_s')}" + ) + except Exception as exc: # noqa: BLE001 - metrics must never fail a good trial + self.logger.warning(f"Could not compute token/cost metrics: {exc}") + + def _agent_run_s(self) -> float | None: + """Seconds the agent command took, as measured by this adapter. + + None until `run()` has both started and finished (including its timeout + path, where the `finally` block still stops the clock). Never negative: + `time.monotonic()` cannot go backwards. + """ + if self._run_started_at is None or self._run_ended_at is None: + return None + return round(self._run_ended_at - self._run_started_at, 3) + + # ------------------------------------------------------------------ + # Setup + # ------------------------------------------------------------------ + + async def setup(self, environment: BaseEnvironment) -> None: + # super().setup() runs install_spec steps (when not preinstalled) and + # version detection. Never skip it. + await super().setup(environment) + if self._local_source: + await self._install_local_source(environment) + # Always record what actually got installed. Without this the trial log + # cannot tell you which build was benchmarked -- which matters most for + # the git-ref path, where the ref is a moving branch. + self.logger.info("[%s] benchmarking version: %s", self.name(), self._version or "") + + async def _install_local_source(self, environment: BaseEnvironment) -> None: + assert self._local_source is not None + package = self.LOCAL_SOURCE_PACKAGE + assert package is not None + + dest = f"{LOCAL_SOURCE_ROOT}/{package}" + env = self.agent_env() + + with tempfile.TemporaryDirectory() as tmp: + staged = Path(tmp) / package + shutil.copytree(self._local_source, staged, ignore=LOCAL_SOURCE_IGNORE) + n_files = sum(1 for _ in staged.rglob("*") if _.is_file()) + self.logger.info( + f"LOCAL SOURCE: {self._local_source} -> {dest} ({n_files} files staged)" + ) + # upload_dir is `docker compose cp`; the destination must already exist. + await self.exec_as_root(environment, self._wrap(f"mkdir -p {dest}"), env=env) + await environment.upload_dir(staged, dest) + + await self.exec_as_root( + environment, + self._wrap(f"uv tool install --reinstall --force --from {dest} {package}"), + env=env, + ) + + # Provenance: without this the trial log cannot distinguish a local-source + # run from a git-ref run. + result = await environment.exec( + command=self._wrap(f"{package} --version"), + env=environment.agent_process_env(env), + user="root", + ) + version = (result.stdout or result.stderr or "").strip() + self.logger.info(f"INSTALLED VERSION: {version!r}") + + # ------------------------------------------------------------------ + # Run + # ------------------------------------------------------------------ + + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + # FIRST statement: nothing above it may be counted as agent time. The + # value is parked on `self`, never on `context` -- pier only calls + # populate_context_post_run() when `context.is_empty()`, so writing the + # timing here would suppress ALL token/cost population. + self._run_started_at = time.monotonic() + + env = self.agent_env() + await self._upload_instruction(instruction, environment) + + log_path = f"{environment.env_paths.agent_dir}/{self.LOG_FILENAME}" + command = self._wrap( + f"cd {WORKDIR} && " + f"{self.run_command(INSTRUCTION_PATH)} " + f"2>&1 None: + """Stage the instruction as a file. NEVER interpolate it into a shell string.""" + text = self.render_instruction(instruction) + with tempfile.TemporaryDirectory() as tmp: + local = Path(tmp) / "instruction.txt" + local.write_text(text, encoding="utf-8") + await environment.upload_file(local, INSTRUCTION_PATH) + + # ------------------------------------------------------------------ + # Fallback commit (cancellation-safe) + # ------------------------------------------------------------------ + + def _fallback_commit_command(self) -> str: + return ( + f"cd {WORKDIR} && " + "git config --global --add safe.directory /app; " + "git config --global user.email 'agent@amplifier.local'; " + "git config --global user.name 'Amplifier Agent'; " + 'if git add -A && git commit -m "agent work" >/dev/null 2>&1; then ' + f'echo "{FALLBACK_MARKER}: committed"; ' + f'else echo "{FALLBACK_MARKER}: nothing-to-commit"; fi' + ) + + async def _fallback_commit(self, environment: BaseEnvironment, env: dict[str, str]) -> None: + result = await asyncio.wait_for( + environment.exec( + command=self._wrap(self._fallback_commit_command()), + env=environment.agent_process_env(env), + user="root", + ), + timeout=FALLBACK_COMMIT_TIMEOUT_SEC, + ) + out = (result.stdout or "") + (result.stderr or "") + if f"{FALLBACK_MARKER}: committed" in out: + self.logger.warning( + "Agent did not commit its own work; fallback commit captured the worktree. " + f"({FALLBACK_MARKER}: committed)" + ) + else: + self.logger.info(f"{FALLBACK_MARKER}: nothing-to-commit") + + async def _await_guarded(self, coro: Coroutine[Any, Any, None], description: str) -> None: + """Run *coro* to completion even while THIS coroutine is being cancelled. + + This is subtle and load-bearing. Do not simplify. + + pier runs ``asyncio.wait_for(agent.run(...), timeout=agent_timeout_sec)``. + On timeout our task is cancelled, so a bare ``await`` inside the + ``finally`` block can itself raise ``CancelledError`` and the teardown + work would never happen. ``except Exception`` does NOT help here: + ``CancelledError`` inherits from ``BaseException``. + + Mechanism: + * The work runs in a SEPARATE task (``ensure_future``), so cancelling + *us* does not cancel *it*. + * We await it through ``asyncio.shield`` so an interrupted await leaves + the work task running. + * If our await is interrupted, we retry the shielded await (bounded). + ``wait_for`` only cancels once, so one retry is normally enough; the + bound just guarantees we cannot spin. + * Callers' coroutines carry their own hard timeout, so a wedged + container cannot hang teardown. + + Every teardown step in ``run()``'s ``finally`` block goes through here. + The mechanism lives in one place so a fix cannot be applied to one copy + and missed on the others. + """ + task = asyncio.ensure_future(coro) + for _ in range(3): + try: + await asyncio.shield(task) + return + except asyncio.CancelledError: + if task.done(): + # Work finished; swallow so the original exception (the + # agent timeout) is the one that propagates. + return + continue + except Exception as exc: # noqa: BLE001 - cleanup must never mask the real failure + self.logger.warning(f"{description} failed: {exc}") + return + task.cancel() + self.logger.warning(f"{description} could not be awaited to completion.") + + async def _fallback_commit_guarded( + self, environment: BaseEnvironment, env: dict[str, str] + ) -> None: + """Run the fallback commit even while this coroutine is being cancelled. + + This is subtle and load-bearing. Do not simplify. + + Without the shielding in ``_await_guarded``, an agent timeout would + cancel us before the commit was issued -- producing a 0-byte + ``model.patch`` and a total loss instead of partial credit. The commit + coroutine carries its own hard timeout (``FALLBACK_COMMIT_TIMEOUT_SEC``) + so a wedged container cannot hang teardown. + """ + await self._await_guarded(self._fallback_commit(environment, env), "Fallback commit") + + async def _collect_session_dirs(self, environment: BaseEnvironment) -> None: + """Pull the agent's in-container session trees onto the host. + + Each entry of ``SESSION_DIRS`` lands at + ``/agent/sessions/``, which is inside the ``/logs/agent`` + bind mount, so ``populate_context_post_run`` can read the events.jsonl + files host-side afterwards. + + Best effort by construction: every download is guarded and a failure is + logged, never raised. + + The downloads share ONE overall deadline + (``TEARDOWN_COLLECT_TIMEOUT_SEC``) rather than a per-directory timeout, + so adding directories to ``SESSION_DIRS`` can never extend how long + teardown may stall on a wedged container. + """ + if not self.SESSION_DIRS: + return + loop = asyncio.get_running_loop() + deadline = loop.time() + TEARDOWN_COLLECT_TIMEOUT_SEC + for source in self.SESSION_DIRS: + remaining = deadline - loop.time() + if remaining <= 0: + self.logger.warning( + f"Session collection budget ({TEARDOWN_COLLECT_TIMEOUT_SEC}s) exhausted; " + f"skipped {source}" + ) + continue + target = self.logs_dir / "sessions" / Path(source).name + try: + target.mkdir(parents=True, exist_ok=True) + await asyncio.wait_for(environment.download_dir(source, target), timeout=remaining) + except Exception as exc: # noqa: BLE001 - collection is best effort + # repr, not str: a bare asyncio TimeoutError stringifies to "" + # and would produce a warning that says nothing. + self.logger.warning(f"Could not collect session dir {source}: {exc!r}") + + async def _collect_session_dirs_guarded(self, environment: BaseEnvironment) -> None: + """Collect session dirs even while this coroutine is being cancelled. + + Same hazard as the fallback commit. This is NOT an edge case: on a + full-budget run the agent hitting its timeout is a likely outcome, and + that is exactly the trial whose trajectory and token/cost data we most + want. A bare await here would be cancelled and lose it. See + ``_await_guarded``. + """ + await self._await_guarded(self._collect_session_dirs(environment), "Session collection") diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/metrics.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/metrics.py new file mode 100644 index 00000000..a67abba7 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/metrics.py @@ -0,0 +1,727 @@ +"""Parse extracted session logs and emit one normalized metrics.json. + +Forked from an earlier internal evaluation harness and since diverged; this +copy is the source of truth for deep-swe. + +Two token/cost sources are supported, and both produce the same record shape: + +- Amplifier `events.jsonl` (one JSON object per line), in either of the two + envelope shapes the runtimes emit (`input_tokens`/`timestamp` or the bare + `input`/`ts` names). A single call is written to disk once per logging hook + the session composes, so `parse_events` de-duplicates by response identity. +- Vanilla opencode, which has NO events.jsonl and records per-session usage in + the SQLite `opencode.db`. Only the session whose `directory` matches this + harness's workdir (`/app`) is counted, excluding the install-time warm-up + session, and cost is recomputed from `MODEL_RATES_PER_M` (see + `parse_opencode_db` for why). + +`not_available` discipline (never fabricate): every normalized field is either +a real number or the exact string `"not_available"` -- never a silent 0. +""" + +from __future__ import annotations + +import datetime +import hashlib +import json +import re +import sqlite3 +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +# The normalized, agent-agnostic efficiency schema. Every field must appear in +# metrics.json as a number or the NOT_AVAILABLE marker. +NOT_AVAILABLE = "not_available" + +METRIC_FIELDS = ( + "cost_usd", + "input_tokens", + "output_tokens", + "cache_read", + "cache_write", + "total_tokens", + "llm_responses", + "agent_wallclock_s", + "total_wallclock_s", +) + + +def _parse_iso(tstr: str) -> float | None: + """Parse an ISO-8601 string to epoch seconds, trimming sub-microsecond fraction. + + Handles both the amplifier-agent `timestamp` (nanoseconds, e.g. + "2026-07-07T17:22:40.591486782+00:00") and the Python amplifier `ts` (e.g. + "2026-02-05T22:33:33.323+00:00"). datetime.fromisoformat only accepts up to + microseconds, so the fraction is trimmed to '.' + 6 digits. + """ + m = re.match(r"^(.*T\d{2}:\d{2}:\d{2})(\.\d+)?(.*)$", tstr) + if not m: + return None + base, frac, tz = m.groups() + frac = frac[:7] if frac else "" # '.' + up to 6 digits + try: + return datetime.datetime.fromisoformat(f"{base}{frac}{tz}").timestamp() + except ValueError: + return None + + +def _event_epoch(obj: dict) -> float | None: + """Return an event's time as epoch seconds. + + Reads whichever time field is present -- amplifier-agent uses `timestamp`, + the Python amplifier hooks-logging format uses `ts`. Both emit ISO-8601. + """ + for key in ("ts", "timestamp"): + val = obj.get(key) + if isinstance(val, str): + epoch = _parse_iso(val) + if epoch is not None: + return epoch + return None + + +def _pick(usage: dict, *keys: str) -> object: + """Return the first present key from `usage`, else None. + + Providers disagree on token field names: the amplifier-agent stack emits + `input_tokens`/`cache_read_tokens`; the Python Anthropic provider emits + `input`/`cache_read`. Try the `_tokens` name first, then the bare name. + """ + for key in keys: + if key in usage: + return usage[key] + return None + + +def _to_int(value: object) -> int: + """Coerce a usage field to int; missing/malformed -> 0.""" + try: + return int(value) # type: ignore[arg-type] + except (TypeError, ValueError): + return 0 + + +def _to_float(value: object) -> float: + """Coerce a cost field (often a string like '0.01867525') to float; else 0.0.""" + try: + return float(value) # type: ignore[arg-type] + except (TypeError, ValueError): + return 0.0 + + +def _ms_to_epoch(value: object) -> float | None: + """Coerce an opencode timestamp (epoch milliseconds) to epoch seconds.""" + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) / 1000.0 + + +@dataclass +class _Usage: + """Token/cost/timestamp accumulator shared by both parser branches. + + The two parsers read genuinely different sources (SQLite rows vs JSONL + lines) but accumulate the same figures and must return the same dict shape, + because `_finalize` treats the two branches uniformly. That common part + lives here once; the reading bodies stay separate. + """ + + input_tokens: int = 0 + output_tokens: int = 0 + cache_read_tokens: int = 0 + cache_write_tokens: int = 0 + cost_usd: float = 0.0 + saw_cost: bool = False + llm_responses: int = 0 + files_read: int = 0 + min_ts: float | None = None + max_ts: float | None = None + + def observe_time(self, epoch: float | None) -> None: + """Widen the earliest-to-latest span with one timestamp; None is a no-op.""" + if epoch is None: + return + self.min_ts = epoch if self.min_ts is None else min(self.min_ts, epoch) + self.max_ts = epoch if self.max_ts is None else max(self.max_ts, epoch) + + def as_dict(self) -> dict[str, Any]: + """The keys `_finalize` reads, identical for both branches. + + `agent_wallclock_s` is 0.0 when no timestamp was seen -- callers must + consult `had_timestamps` to tell that apart from a genuine 0-length run. + """ + lo, hi = self.min_ts, self.max_ts + had_timestamps = lo is not None and hi is not None + return { + "input_tokens": self.input_tokens, + "output_tokens": self.output_tokens, + "cache_read_tokens": self.cache_read_tokens, + "cache_write_tokens": self.cache_write_tokens, + "total_tokens": self.input_tokens + self.output_tokens, + "cost_usd": self.cost_usd, + "cost_from_events": self.saw_cost, + "llm_responses": self.llm_responses, + "files_read": self.files_read, + "had_timestamps": had_timestamps, + "agent_wallclock_s": (hi - lo) if lo is not None and hi is not None else 0.0, + } + + +# --------------------------------------------------------------------------- +# Reference rate card +# --------------------------------------------------------------------------- + +#: USD per 1M tokens, keyed by model id. THE single rate card for this harness. +#: +#: Mirrors `_RATES` in amplifier-module-provider-anthropic/_cost.py, which is +#: what stamps `cost_usd` into the amplifier arms' events.jsonl. Arms are only +#: comparable if every dollar figure comes from the same card, so this is also +#: what the opencode arm's cost is RECOMPUTED with -- see +#: `compute_cost_from_tokens` and the WHY in `parse_opencode_db`. +#: +#: `opencode_vanilla.py` imports this to populate the model's `cost` block in +#: opencode.json. One definition, one home. +MODEL_RATES_PER_M: dict[str, dict[str, float]] = { + "claude-sonnet-5": {"input": 3.00, "output": 15.00, "cache_read": 0.30, "cache_write": 3.75}, + "claude-sonnet-4-5": {"input": 3.00, "output": 15.00, "cache_read": 0.30, "cache_write": 3.75}, + "claude-opus-5": {"input": 5.00, "output": 25.00, "cache_read": 0.50, "cache_write": 6.25}, +} + + +def compute_cost_from_tokens( + model: str | None, + *, + input_tokens: int = 0, + output_tokens: int = 0, + cache_read_tokens: int = 0, + cache_write_tokens: int = 0, +) -> float | None: + """USD for one call/session from token counts and the reference card. + + Returns None for an unrecognised model -- semantically distinct from 0.0 + (a genuinely free call). Callers must propagate that as `not_available` + rather than as a free run. + """ + rates = MODEL_RATES_PER_M.get(model or "") + if rates is None: + return None + return ( + input_tokens * rates["input"] + + output_tokens * rates["output"] + + cache_read_tokens * rates["cache_read"] + + cache_write_tokens * rates["cache_write"] + ) / 1_000_000.0 + + +def _opencode_model_id(raw: Any) -> str | None: + """Extract the bare model id from opencode's `session.model` column. + + Stored as a JSON object, e.g. + ``{"id":"claude-sonnet-5","providerID":"anthropic","variant":"default"}``, + which SQLite hands back as the raw JSON text. + """ + if isinstance(raw, str): + try: + raw = json.loads(raw) + except (json.JSONDecodeError, ValueError): + return None + if isinstance(raw, dict): + mid = raw.get("id") + return mid if isinstance(mid, str) and mid else None + return None + + +def _opencode_sessions(db_path: str, workspace_dir: str) -> tuple[list[dict] | None, int, int]: + """Read per-session usage rows from one opencode SQLite db (`opencode.db`). + + Returns (sessions, assistant_message_count, total_session_count), or + (None, 0, 0) when the file is not a usable opencode db (unreadable, or + missing the expected `session` columns). The db is opened READ-ONLY + (`mode=ro`); the WAL sidecar (`opencode.db-wal`) must be co-located for the + newest writes to be visible. + + ONLY sessions whose `directory == workspace_dir` are returned. There is + deliberately NO "if nothing matched, return everything" fallback. + + That fallback was worse than useless. `directory` is an absolute path, so a + caller passing the wrong workspace matched nothing and silently summed + EVERY session in the database -- including unrelated ones -- publishing a + plausible-looking number that was wrong by orders of magnitude. An empty + result surfaces as `not_available`, which is honest and fixable; a + fabricated total is neither. + """ + try: + con = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) + except sqlite3.Error: + return None, 0, 0 + try: + con.row_factory = sqlite3.Row + cols = {row[1] for row in con.execute("PRAGMA table_info(session)")} + if not {"tokens_input", "tokens_output", "cost", "directory"}.issubset(cols): + return None, 0, 0 + sessions = [dict(r) for r in con.execute("SELECT * FROM session")] + matched = [s for s in sessions if s.get("directory") == workspace_dir] + + ids = {s.get("id") for s in matched} + assistant = 0 + try: + for sid, data in con.execute("SELECT session_id, data FROM message"): + if sid not in ids: + continue + try: + md = json.loads(data) + except (json.JSONDecodeError, ValueError, TypeError): + continue + if isinstance(md, dict) and md.get("role") == "assistant": + assistant += 1 + except sqlite3.Error: + assistant = 0 + return matched, assistant, len(sessions) + except sqlite3.Error: + return None, 0, 0 + finally: + con.close() + + +def parse_opencode_db(db_paths: list[str], workspace_dir: str) -> dict[str, Any]: + """Sum token/cost usage for a vanilla opencode run from its SQLite db(s). + + Vanilla opencode has NO amplifier events.jsonl; the `session` table in + `opencode.db` already aggregates per-session usage (tokens_input/output/ + cache_read/cache_write, cost) with a `directory` and epoch-ms + time_created/time_updated. llm_responses is the count of assistant rows in + the `message` table for the task session. + + Returns the SAME dict shape as `parse_events` (see `_Usage.as_dict`) plus + `extra_notes`, so `_finalize` can treat both branches uniformly. + `cost_from_events` is True when a real cost figure was produced, so a + genuine $0 is not mistaken for a free run. + + COST IS RECOMPUTED, NOT READ. The `session.cost` column is deliberately + ignored for the published figure and reported only as an audit note. + + Why: opencode prices calls from a rate card it fetches from models.dev, and + that card is NOT the one this harness bills against -- it diverges on the + cache rates, and opencode ignores the explicit `cost.cache` override baked + into opencode.json. Comparing arms priced from two different cards is a + silent error: both numbers look plausible and neither is flagged. The token + counts are ground truth from the agent; only the multiplication belongs to + the harness, so cost is recomputed from `MODEL_RATES_PER_M`. + + Args: + db_paths: Paths to extracted `opencode.db` files (the WAL sidecar must + be co-located for completeness). + workspace_dir: Session directory identifying the task run, used to + exclude the install-time warm-up session. Required: a wrong value + silently matches nothing, so there is no safe default. + """ + usage = _Usage() + extra_notes: list[str] = [] + + for db_path in db_paths: + sessions, assistant, total_sessions = _opencode_sessions(db_path, workspace_dir) + if sessions is None: + continue + if not sessions: + # The db is a valid opencode db but holds no session for this + # workspace. Report it; do NOT substitute the other sessions. + extra_notes.append( + f"{Path(db_path).name} held {total_sessions} session(s), none with " + f"directory == {workspace_dir!r}; contributed nothing. If the agent " + f"really ran, the workspace_dir passed to the parser is wrong." + ) + continue + usage.files_read += 1 + # Prefer the true assistant-turn count; fall back to session count so a + # run with clear usage is never reported as 0 responses. + usage.llm_responses += assistant or len(sessions) + for s in sessions: + s_in = _to_int(s.get("tokens_input")) + s_out = _to_int(s.get("tokens_output")) + s_cr = _to_int(s.get("tokens_cache_read")) + s_cw = _to_int(s.get("tokens_cache_write")) + usage.input_tokens += s_in + usage.output_tokens += s_out + usage.cache_read_tokens += s_cr + usage.cache_write_tokens += s_cw + + model = _opencode_model_id(s.get("model")) + recomputed = compute_cost_from_tokens( + model, + input_tokens=s_in, + output_tokens=s_out, + cache_read_tokens=s_cr, + cache_write_tokens=s_cw, + ) + reported = _to_float(s.get("cost")) + if recomputed is None: + # Unknown model => no rate card => no honest dollar figure. + # NOT 0.0: the run was not free, we just cannot price it. + extra_notes.append( + f"Model {model!r} is not in the reference rate card, so cost_usd is " + f"not_available for this session. opencode self-reported " + f"${reported:.6f}, which is priced from ITS card and is not " + f"comparable to the amplifier arms." + ) + else: + usage.saw_cost = True + usage.cost_usd += recomputed + # Always record the divergence. If the two ever agree this note + # is the evidence; when they disagree it is the explanation. + delta = recomputed - reported + pct = (delta / recomputed * 100.0) if recomputed else 0.0 + extra_notes.append( + f"cost_usd RECOMPUTED from token counts at the reference rate card " + f"for {model}: ${recomputed:.6f}. opencode self-reported " + f"${reported:.6f} (delta ${delta:+.6f}, {pct:+.1f}%); its figure is " + f"priced from a models.dev card that differs on cache rates and is " + f"NOT used." + ) + for key in ("time_created", "time_updated"): + usage.observe_time(_ms_to_epoch(s.get(key))) + + return {**usage.as_dict(), "extra_notes": extra_notes} + + +def _response_identity(obj: dict) -> str | None: + """Return a stable identity for an `llm:response` event, or None. + + WHY THIS EXISTS. A single logical LLM call is written to disk more than once + whenever a session composes more than one logging hook. The published + `anchors` bundle does exactly this: it includes BOTH + `foundation:behaviors/logging` (hooks-logging -> `/events.jsonl`) + and `context-intelligence:behaviors/context-intelligence-logging` + (hook-context-intelligence -> `/context-intelligence/events.jsonl`). + Both files are legitimate telemetry and both are pulled by extraction, so + summing across them counted every call, token and dollar TWICE. + + The two copies are not detectable by comparing bytes: the loggers use + different envelope shapes (`ts` vs `timestamp`, metadata at the top level vs + nested under `data`), so the files share zero identical lines. They are only + recognisable by what they DESCRIBE. Hence identity, not equality: + + strong the provider's own response id (`data.raw.id`). Globally unique + per call, present whenever raw payload capture is on. + fallback session id + event timestamp + usage, hashed. Used when raw + capture is off. The timestamp is sub-microsecond and is byte + identical across both loggers (verified on real run artifacts), + so it discriminates distinct calls reliably. + None neither is available -- see the caller. We do NOT guess. + + Returning None on a weak signal is deliberate. Over-de-duplicating would + silently UNDERSTATE cost, which is a worse failure than the overcount this + function exists to fix: an inflated number invites scrutiny, a deflated one + does not. + """ + # Bind before narrowing: `x.get(k) if isinstance(x.get(k), dict) else {}` + # calls get() twice, and the isinstance on the first call does not narrow + # the second, so the result stays `Any | None`. + raw_data = obj.get("data") + data: dict[str, Any] = raw_data if isinstance(raw_data, dict) else {} + + raw_payload = data.get("raw") + if isinstance(raw_payload, dict): + rid = raw_payload.get("id") + if isinstance(rid, str) and rid: + return f"id:{rid}" + + # Fallback fingerprint. Require a timestamp: without one, two genuinely + # distinct calls that happened to use the same token counts would collide. + ts = data.get("timestamp") or obj.get("ts") or obj.get("timestamp") + if not isinstance(ts, (str, int, float)) or isinstance(ts, bool): + return None + + session_id = data.get("session_id") or obj.get("session_id") + raw_usage = data.get("usage") + usage: dict[str, Any] = raw_usage if isinstance(raw_usage, dict) else {} + fingerprint = json.dumps([session_id, str(ts), usage], sort_keys=True, default=str) + return "fp:" + hashlib.sha1(fingerprint.encode()).hexdigest() + + +def parse_events(events_paths: list[str]) -> dict[str, Any]: + """Sum token/cost usage and compute wallclock from Amplifier events.jsonl. + + Dual-shape aware (see the module docstring). Malformed lines and unreadable + files are skipped defensively; a run is never silently dropped because one + line failed to parse. + + Args: + events_paths: Paths to events.jsonl files (one JSON object per line). + + Returns: + The keys listed in `_Usage.as_dict`, plus `duplicate_responses` (count + of duplicate `llm:response` events dropped) and + `unidentified_responses` (count that carried no usable identity and + were therefore counted without de-duplication). + """ + usage = _Usage() + duplicate_responses = 0 + unidentified_responses = 0 + seen_responses: set[str] = set() + + for path in events_paths: + try: + with open(path, encoding="utf-8") as f: + lines = f.readlines() + except OSError: + continue + usage.files_read += 1 + + for line in lines: + line = line.strip() + if not line: + continue + try: + obj = json.loads(line) + except (json.JSONDecodeError, ValueError): + continue + if not isinstance(obj, dict): + continue + + # Track wallclock across ALL events that carry a usable time field. + usage.observe_time(_event_epoch(obj)) + + if obj.get("event") != "llm:response": + continue + + # De-duplicate by identity. The same call reaches this loop once per + # logging hook the session composed (see _response_identity), and + # summing every copy is what inflated cost, tokens and call counts. + identity = _response_identity(obj) + if identity is None: + # No usable identity: count it rather than risk collapsing two + # genuinely distinct calls. Surfaced in the notes. + unidentified_responses += 1 + elif identity in seen_responses: + duplicate_responses += 1 + continue + else: + seen_responses.add(identity) + + data = obj.get("data") + event_usage = data.get("usage") if isinstance(data, dict) else None + if not isinstance(event_usage, dict): + # Still count the response even if usage is absent. + usage.llm_responses += 1 + continue + + usage.llm_responses += 1 + # Field names differ by runtime/provider: amplifier-agent emits the + # `_tokens`-suffixed names, the Python Anthropic provider emits the + # bare names. Accept either. + usage.input_tokens += _to_int(_pick(event_usage, "input_tokens", "input")) + usage.output_tokens += _to_int(_pick(event_usage, "output_tokens", "output")) + usage.cache_read_tokens += _to_int( + _pick(event_usage, "cache_read_tokens", "cache_read") + ) + usage.cache_write_tokens += _to_int( + _pick(event_usage, "cache_write_tokens", "cache_write") + ) + # cost_usd is only emitted by the amplifier-agent stack. Track + # whether we ever saw it so a $0 from a stack that does not record + # cost is not reported as a real, free run. + if "cost_usd" in event_usage: + usage.saw_cost = True + usage.cost_usd += _to_float(event_usage.get("cost_usd")) + + return { + **usage.as_dict(), + "duplicate_responses": duplicate_responses, + "unidentified_responses": unidentified_responses, + } + + +def _find(output_dir: Path | str, filename: str) -> list[str]: + """Return every `filename` under a run's output dir, sorted for determinism. + + The extractor pulls session logs under `output_dir/sessions/`, preserving + each session's own layout, so the whole tree is globbed: narrowing the glob + to pick a "primary" file would be fragile, because the layout differs per + agent and would silently yield zero on any change. + + For `events.jsonl` this deliberately returns EVERY file, including the + several a single session produces when it composes more than one logging + hook; the duplicates are handled where they can be handled correctly, by + `parse_events` de-duplicating on response identity. For `opencode.db` it + matches the main db only, not the `-wal`/`-shm` sidecars -- sqlite3 reads a + co-located WAL automatically and the extractor preserves the layout, so the + sidecars land next to the db. + """ + root = Path(output_dir).expanduser().resolve() + return sorted(str(p) for p in root.rglob(filename)) + + +def find_events_files(output_dir: Path | str) -> list[str]: + """Return all extracted `events.jsonl` paths under a run's output dir.""" + return _find(output_dir, "events.jsonl") + + +def find_opencode_db_files(output_dir: Path | str) -> list[str]: + """Return all extracted `opencode.db` paths under a run's output dir.""" + return _find(output_dir, "opencode.db") + + +def normalize_metrics(events_paths: list[str], *, source: str | None = None) -> dict[str, Any]: + """Produce the normalized metrics.json record from extracted events. + + Applies the `not_available` discipline: + - If no events file was readable, every token/cost/response/agent-wallclock + field is `"not_available"` (the source is genuinely absent). + - `cost_usd` is `"not_available"` when no event carried a `cost_usd` field + (e.g. the Python amplifier stack), never a fabricated 0. + - `agent_wallclock_s` is `"not_available"` when no event timestamp was + found; otherwise it is the earliest-to-latest event span, a floor on true + agent time. + + Args: + events_paths: Paths to extracted events.jsonl files. + source: Optional identifier for the agent/stack (e.g. the agent id). + + Returns: + A JSON-safe dict with every METRIC_FIELDS key present, plus `notes`, + `source`, and `events_files` (the files this record was computed from). + """ + parsed = parse_events(events_paths) + files_read = parsed["files_read"] + notes: list[str] = [] + + if files_read == 0: + notes.append( + "No events.jsonl files were readable under the extraction output dir; " + "all token/cost/response/agent-wallclock fields are not_available." + ) + else: + notes.append( + f"Parsed {parsed['llm_responses']} llm:response event(s) across " + f"{files_read} events.jsonl file(s). Token keys read with dual-shape " + f"fallback (input_tokens/input, etc.)." + ) + # State the de-duplication explicitly. Without this the corrected figure + # is indistinguishable from a run that simply made fewer calls. + dupes = parsed["duplicate_responses"] + if dupes: + notes.append( + f"Dropped {dupes} duplicate llm:response event(s): this session composes more " + f"than one logging hook, so each call was written to disk more than once. " + f"Counted once each by response identity." + ) + unknown = parsed["unidentified_responses"] + if unknown: + notes.append( + f"{unknown} llm:response event(s) carried no response id and no timestamp, so " + f"they could not be de-duplicated and are counted as-is; if this session " + f"composes multiple logging hooks these may be overcounted." + ) + if parsed["cost_from_events"]: + notes.append("cost_usd summed from per-response cost_usd fields.") + else: + notes.append("cost_usd is not_available: no event carried a cost_usd field.") + if parsed["had_timestamps"]: + notes.append( + "agent_wallclock_s is the earliest-to-latest event timestamp span " + "(a floor on true agent time; agent_run_s is the measured duration)." + ) + else: + notes.append("agent_wallclock_s is not_available: no timestamps found.") + + return _finalize(parsed, source=source, source_files=events_paths, notes=notes) + + +def normalize_opencode_metrics( + db_paths: list[str], + *, + source: str | None = None, + workspace_dir: str, +) -> dict[str, Any]: + """Produce the normalized metrics.json record from a vanilla opencode db. + + Same `not_available` discipline and output schema as `normalize_metrics`, + but the token/cost source is the opencode SQLite `session` table rather than + Amplifier events.jsonl. `workspace_dir` is required for the reason given in + `parse_opencode_db`: a wrong value silently matches no session. + """ + parsed = parse_opencode_db(db_paths, workspace_dir) + files_read = parsed["files_read"] + extra_notes: list[str] = list(parsed["extra_notes"]) + notes: list[str] = [] + + if files_read == 0: + # Only claim "nothing was readable" when nothing explains it better. A + # source that WAS read but contributed no rows has already said so, and + # the two statements together read as a contradiction. + if not extra_notes: + notes.append( + "No usable opencode.db was readable under the extraction output dir; " + "all token/cost/response/agent-wallclock fields are not_available." + ) + else: + notes.append( + f"Parsed {parsed['llm_responses']} assistant message(s) across " + f"{files_read} opencode.db file(s). ONLY sessions with directory == " + f"{workspace_dir} were counted; every other session in the database was " + f"excluded, with no fallback." + ) + if parsed["cost_from_events"]: + notes.append("cost_usd summed from the opencode session table `cost` column.") + else: + notes.append("cost_usd is not_available: the opencode session carried no cost.") + if parsed["had_timestamps"]: + notes.append( + "agent_wallclock_s is the task session's time_created-to-time_updated span " + "(opencode epoch-ms timestamps; a floor on true agent time)." + ) + else: + notes.append("agent_wallclock_s is not_available: no timestamps found.") + + # Parser-level warnings must reach metrics.json either way -- e.g. a db that + # held sessions but none for this workspace, which is the difference between + # "no data" and "wrong query". + notes.extend(extra_notes) + return _finalize(parsed, source=source, source_files=db_paths, notes=notes) + + +def _finalize( + parsed: dict[str, Any], + *, + source: str | None, + source_files: list[str], + notes: list[str], +) -> dict[str, Any]: + """Turn a parser's `parsed` dict into the normalized metrics.json record. + + Shared by both the events.jsonl and opencode.db branches: they produce the + same `parsed` shape, so the `not_available` discipline and the rounding live + here once. Branch-specific prose is composed by the caller and passed in. + """ + if parsed["files_read"] == 0: + # No session sources were pulled: everything derived from them is absent. + record: dict[str, Any] = {name: NOT_AVAILABLE for name in METRIC_FIELDS} + else: + record = { + "cost_usd": parsed["cost_usd"] if parsed["cost_from_events"] else NOT_AVAILABLE, + "input_tokens": parsed["input_tokens"], + "output_tokens": parsed["output_tokens"], + "cache_read": parsed["cache_read_tokens"], + "cache_write": parsed["cache_write_tokens"], + "total_tokens": parsed["total_tokens"], + "llm_responses": parsed["llm_responses"], + "agent_wallclock_s": ( + parsed["agent_wallclock_s"] if parsed["had_timestamps"] else NOT_AVAILABLE + ), + } + + # Whole-trial elapsed is not measured here; the adapter records the agent + # command duration instead. Never fabricate a number for it. + record["total_wallclock_s"] = NOT_AVAILABLE + + # Round the span/cost figures for readability when they are numbers. + if isinstance(record["agent_wallclock_s"], (int, float)): + record["agent_wallclock_s"] = round(float(record["agent_wallclock_s"]), 3) + if isinstance(record["cost_usd"], (int, float)): + record["cost_usd"] = round(float(record["cost_usd"]), 6) + + record["source"] = source + record["events_files"] = list(source_files) + record["notes"] = " ".join([*notes, "total_wallclock_s is not measured by this harness."]) + return record diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_amplifier.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_amplifier.py new file mode 100644 index 00000000..d79e2260 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_amplifier.py @@ -0,0 +1,77 @@ +"""OpenCode frontend backed by amplifier-agent (amplifier-app-opencode).""" + +from __future__ import annotations + +from typing import Any + +from pier.models.agent.install import InstallStep + +from deepswe_agents.base import ( + DEFAULT_AMPLIFIER_AGENT_REF, + OPENCODE_PRELUDE, + UV_PRELUDE, + AmplifierBaseAgent, +) + +# amplifier-agent imports httpx at import time, on the path of every CLI command. +# It used to arrive transitively via `mcp`; that pull no longer exists, so it is +# pinned here for anyone installing an older SHA via --ak amplifier_agent_ref. +AMPLIFIER_AGENT_WITH = "httpx>=0.27,<1" + +# Tracks the default branch so we benchmark the app as it actually ships. +# Override with --ak amplifier_app_opencode_ref=...@ for a reproducible run. +DEFAULT_AMPLIFIER_APP_OPENCODE_REF = "git+https://github.com/microsoft/amplifier-app-opencode" + + +class OpencodeAmplifierAgent(AmplifierBaseAgent): + LOCAL_SOURCE_PACKAGE = "amplifier-agent" + VERSION_BINARY = "opencode" + + def __init__( + self, + *args: Any, + amplifier_agent_ref: str = DEFAULT_AMPLIFIER_AGENT_REF, + amplifier_app_opencode_ref: str = DEFAULT_AMPLIFIER_APP_OPENCODE_REF, + **kwargs: Any, + ): + super().__init__(*args, **kwargs) + self._amplifier_agent_ref = amplifier_agent_ref + self._amplifier_app_opencode_ref = amplifier_app_opencode_ref + + @staticmethod + def name() -> str: + return "opencode-amplifier-agent" + + def agent_install_steps(self) -> list[InstallStep]: + return [ + *UV_PRELUDE, + InstallStep( + user="root", + run=( + 'export PATH="$HOME/.local/bin:$PATH"; ' + f'uv tool install --with "{AMPLIFIER_AGENT_WITH}" ' + f'"{self._amplifier_agent_ref}"' + ), + ), + *OPENCODE_PRELUDE, + InstallStep( + user="root", + run=( + 'export PATH="$HOME/.local/bin:$HOME/.opencode/bin:$PATH"\n' + f'uv tool install --from "{self._amplifier_app_opencode_ref}" ' + "amplifier-app-opencode\n" + "amplifier-opencode --help >/dev/null" + ), + ), + InstallStep( + user="root", + env={"DEBIAN_FRONTEND": "noninteractive"}, + run="apt-get update -qq && apt-get install -y --no-install-recommends jq", + ), + ] + + def run_command(self, instruction_path: str) -> str: + return ( + f"amplifier-opencode launch -- run --auto --model amplifier/{self.model} " + f'"$(cat {instruction_path})"' + ) diff --git a/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_vanilla.py b/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_vanilla.py new file mode 100644 index 00000000..e12bb0c6 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/src/deepswe_agents/opencode_vanilla.py @@ -0,0 +1,147 @@ +"""Stock OpenCode talking straight to Anthropic. The control arm.""" + +from __future__ import annotations + +import json +import shlex +from typing import Any + +from pier.models.agent.install import InstallStep + +from deepswe_agents.base import OPENCODE_PRELUDE, AmplifierBaseAgent +from deepswe_agents.metrics import MODEL_RATES_PER_M + +# The `cost` block written into opencode.json is BEST EFFORT only: opencode +# ignores the `cost.cache` override, so the published dollar figure comes from +# `metrics.parse_opencode_db`, which recomputes cost from the recorded token +# counts. See its docstring for why. + + +class OpencodeVanillaAgent(AmplifierBaseAgent): + # No Amplifier component to replace. + LOCAL_SOURCE_PACKAGE = None + VERSION_BINARY = "opencode" + + # opencode records usage in SQLite, not events.jsonl. + METRICS_SOURCE = "opencode_db" + + # Collect the WHOLE data dir, not just the db file. opencode runs SQLite in + # WAL mode, so the newest writes -- including, in practice, the entire final + # session -- live in the `opencode.db-wal` sidecar. Pulling `opencode.db` + # alone yields a stale database that silently under-reports. `download_dir` + # is directory-granular, so taking the parent is what keeps the sidecars + # co-located for sqlite3 to replay. + SESSION_DIRS = ("/root/.local/share/opencode",) + + @staticmethod + def name() -> str: + return "opencode-vanilla" + + def agent_env(self) -> dict[str, str]: + """Normalize ANTHROPIC_BASE_URL for opencode's ai-sdk provider. + + The two clients disagree on what "base URL" means: + * the Anthropic SDK (amplifier-agent) wants the host root and appends + `/v1` itself -> https://api.anthropic.com + * ai-sdk `@ai-sdk/anthropic` (opencode) treats it as the full API root + and appends only `/messages` -> needs https://api.anthropic.com/v1 + + Forwarding the host value unchanged makes opencode request + `https://api.anthropic.com/messages`, which 404s. opencode reports that + as a bare `Error: Not Found` and aborts the run -- with no mention of a + URL, which makes it look like a model or auth problem. + """ + env = super().agent_env() + base = env.get("ANTHROPIC_BASE_URL") + if base: + trimmed = base.rstrip("/") + if not trimmed.endswith("/v1"): + env["ANTHROPIC_BASE_URL"] = f"{trimmed}/v1" + return env + + def _model_entry(self) -> dict[str, Any]: + entry: dict[str, Any] = {"name": self.model} + rates = MODEL_RATES_PER_M.get(self.model) + if rates: + entry["cost"] = { + "input": rates["input"], + "output": rates["output"], + "cache": {"read": rates["cache_read"], "write": rates["cache_write"]}, + } + return entry + + def _opencode_config(self) -> str: + model = self.model + return json.dumps( + { + "$schema": "https://opencode.ai/config.json", + "model": f"anthropic/{model}", + # Pin the SMALL model to the benchmark model. opencode uses a + # separate "small" model for the session-title agent, and its + # default family priority ends at claude-haiku -- a model this + # endpoint does not serve. That request fails with a bare + # `AI_APICallError: Not Found` and kills the process (exit 1) + # before any task work happens. It was the single most common + # failure of this arm. + "small_model": f"anthropic/{model}", + "provider": { + "anthropic": { + "npm": "@ai-sdk/anthropic", + "models": {model: self._model_entry()}, + } + }, + } + ) + + def agent_install_steps(self) -> list[InstallStep]: + return [ + *OPENCODE_PRELUDE, + InstallStep(user="root", run='mkdir -p "$HOME/.config/opencode"'), + InstallStep( + user="root", + run=( + f"echo {shlex.quote(self._opencode_config())} " + '> "$HOME/.config/opencode/opencode.json"' + ), + ), + InstallStep( + user="root", + run='export PATH="$HOME/.opencode/bin:$PATH"; opencode --version', + ), + ] + + async def run(self, instruction, environment, context) -> None: # type: ignore[override] + try: + await super().run(instruction, environment, context) + finally: + # Guarded: on the timeout path this `finally` runs while the + # coroutine is being cancelled, and a bare await here would itself + # raise CancelledError -- a BaseException that the handler inside + # _dump_opencode_log cannot catch -- masking the real timeout with a + # confusing secondary exception. + await self._await_guarded(self._dump_opencode_log(environment), "opencode log dump") + + async def _dump_opencode_log(self, environment) -> None: + """Surface opencode's own log. + + opencode reports failures to stdout as a bare `Error: ` with no + context; the detail (including the failing request) only lands in + ~/.local/share/opencode/log/. + """ + try: + res = await self.exec_as_agent( + environment, + self._wrap( + "echo '--- OPENCODE LOG ---'; " + 'tail -80 "$HOME"/.local/share/opencode/log/*.log 2>&1 ' + "|| echo '(no opencode log)'" + ), + env=self.agent_env(), + timeout_sec=60, + ) + self.logger.warning("[opencode log]\n%s", (getattr(res, "stdout", "") or "").strip()) + except Exception as exc: # noqa: BLE001 - diagnostics must never break a trial + self.logger.warning(f"could not read opencode log: {exc}") + + def run_command(self, instruction_path: str) -> str: + return f'opencode run --model anthropic/{self.model} --auto "$(cat {instruction_path})"' diff --git a/.amplifier/evaluation/deep-swe/tests/conftest.py b/.amplifier/evaluation/deep-swe/tests/conftest.py new file mode 100644 index 00000000..ac11c4c4 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/tests/conftest.py @@ -0,0 +1,104 @@ +"""Import shims so the harness tests run with a plain `python3 -m pytest tests/`. + +Two shims, both deliberately narrow: + +1. ``src/`` goes on ``sys.path`` so ``deepswe_agents`` imports without an + editable install. + +2. ``pier`` is stubbed IF AND ONLY IF it is not importable in the interpreter + running pytest. pier is installed as a `uv tool`, i.e. into its own venv, so + the obvious developer command would otherwise die at import time before a + single assertion ran. When the real pier IS importable it is used unchanged, + so these tests keep their full value in a pier-equipped interpreter. + +The stub exists only to satisfy MODULE-LEVEL imports in ``deepswe_agents.base``. +The tests exercise the agent's own teardown methods -- shielding, timeouts, +cancellation -- and never call into pier, so nothing here is load-bearing for +what is being asserted. If a test ever needs real pier behaviour, it must run +under an interpreter that has pier rather than gaining a richer fake. +""" + +from __future__ import annotations + +import importlib.util +import sys +import types +from pathlib import Path + +SRC = Path(__file__).resolve().parent.parent / "src" +if str(SRC) not in sys.path: + sys.path.insert(0, str(SRC)) + + +def _pier_importable() -> bool: + try: + return importlib.util.find_spec("pier") is not None + except (ImportError, ValueError): + return False + + +def _module(name: str) -> types.ModuleType: + """Create, register, and attach a stub module to its parent package.""" + mod = types.ModuleType(name) + mod.__path__ = [] # type: ignore[attr-defined] - make it importable as a package + sys.modules[name] = mod + parent, _, leaf = name.rpartition(".") + if parent: + setattr(sys.modules[parent], leaf, mod) + return mod + + +def _install_pier_stub() -> None: + for name in ( + "pier", + "pier.agents", + "pier.agents.installed", + "pier.agents.installed.base", + "pier.environments", + "pier.environments.base", + "pier.models", + "pier.models.agent", + "pier.models.agent.context", + "pier.models.agent.install", + "pier.models.agent.network", + ): + _module(name) + + class BaseInstalledAgent: + """Stand-in base. Real pier plumbing is never exercised by these tests.""" + + class BaseEnvironment: + """Stand-in environment type (tests pass their own fakes).""" + + class AgentContext: + """Mirrors the four fields the metrics pass populates.""" + + def __init__(self, **kwargs: object) -> None: + self.n_input_tokens = None + self.n_cache_tokens = None + self.n_output_tokens = None + self.cost_usd = None + for key, value in kwargs.items(): + setattr(self, key, value) + + def is_empty(self) -> bool: + return all(value is None for value in vars(self).values()) + + class _Record: + """Permissive stand-in for the pydantic models used at import time.""" + + def __init__(self, **kwargs: object) -> None: + for key, value in kwargs.items(): + setattr(self, key, value) + + sys.modules["pier.agents.installed.base"].BaseInstalledAgent = BaseInstalledAgent # type: ignore[attr-defined] + sys.modules["pier.environments.base"].BaseEnvironment = BaseEnvironment # type: ignore[attr-defined] + sys.modules["pier.models.agent.context"].AgentContext = AgentContext # type: ignore[attr-defined] + sys.modules["pier.models.agent.install"].InstallStep = _Record # type: ignore[attr-defined] + sys.modules["pier.models.agent.install"].AgentInstallSpec = _Record # type: ignore[attr-defined] + sys.modules["pier.models.agent.network"].NetworkAllowlist = _Record # type: ignore[attr-defined] + + +PIER_IS_REAL = _pier_importable() +if not PIER_IS_REAL: + _install_pier_stub() diff --git a/.amplifier/evaluation/deep-swe/tests/test_metrics_dedup.py b/.amplifier/evaluation/deep-swe/tests/test_metrics_dedup.py new file mode 100644 index 00000000..f7cd780b --- /dev/null +++ b/.amplifier/evaluation/deep-swe/tests/test_metrics_dedup.py @@ -0,0 +1,200 @@ +"""Regression tests for `llm:response` de-duplication in the metrics pass. + +WHY THIS FILE EXISTS. A session that composes more than one logging hook writes +every LLM call to disk more than once. The published `anchors` bundle does this: +it includes both `foundation:behaviors/logging` (-> `/events.jsonl`) +and `context-intelligence:behaviors/context-intelligence-logging` (-> +`/context-intelligence/events.jsonl`). Extraction pulls both files and +the metrics pass summed across them, so the amplifier-foundation agent reported +exactly DOUBLE its real calls, tokens and cost -- 20 calls at $2.98 for a trial +that actually made 10 calls at $1.49. + +The bug was invisible to every cheap check. The two files share zero identical +lines, because the loggers use different envelope shapes (`ts` vs `timestamp`, +metadata at the top level vs nested under `data`). Only the payload identity +gives it away. These tests pin that behaviour: + + - the two real envelope shapes, carrying one call, count as one + - distinct calls are still counted separately (the fix must not over-collapse) + - de-duplication works with raw capture OFF, via the timestamp fingerprint + - an event with no usable identity is counted rather than silently dropped + - the correction is stated in `notes`, not applied silently + +Run: uv run python -m pytest tests/ -q +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from deepswe_agents.metrics import normalize_metrics, parse_events + +# One LLM call, as each of the two loggers actually writes it. Shapes derived +# from real run artifacts. +RESPONSE_ID = "msg_011CdjqAotUMQC2DjuMppaYo" +TIMESTAMP = "2026-08-05T15:23:15.602291443+00:00" +SESSION = "fa3b1b70-e043-406a-8c8f-1fbc3edd24f0" +USAGE = { + "input_tokens": 2, + "output_tokens": 398, + "cache_write_tokens": 16534, + "cost_usd": "0.1132975", +} + + +def ci_shape( + response_id: str = RESPONSE_ID, ts: str = TIMESTAMP, usage: dict | None = None +) -> dict: + """hook-context-intelligence: `timestamp` inside `data`, `workspace` on top.""" + return { + "event": "llm:response", + "timestamp": ts, + "workspace": "-workspace", + "data": { + "session_id": SESSION, + "timestamp": ts, + "model": "claude-sonnet-5", + "provider": "anthropic", + "status": "ok", + "duration_ms": 8812, + "usage": dict(usage or USAGE), + "raw": {"id": response_id, "role": "assistant", "content": []}, + }, + } + + +def logging_shape( + response_id: str = RESPONSE_ID, ts: str = TIMESTAMP, usage: dict | None = None +) -> dict: + """foundation hooks-logging: `ts` on top, metadata hoisted out of `data`.""" + return { + "event": "llm:response", + "ts": ts, + "lvl": "INFO", + "status": "ok", + "duration_ms": 8812, + "session_id": SESSION, + "schema": {"name": "amplifier.log", "ver": "1.0.0"}, + "data": { + "model": "claude-sonnet-5", + "provider": "anthropic", + "usage": dict(usage or USAGE), + "raw": {"id": response_id, "role": "assistant", "content": []}, + }, + } + + +def write_events(path: Path, events: list[dict]) -> str: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("".join(json.dumps(e) + "\n" for e in events)) + return str(path) + + +def test_two_loggers_one_call_counts_once(tmp_path): + """The exact production bug: same call, two files, two envelope shapes.""" + a = write_events(tmp_path / "s" / "context-intelligence" / "events.jsonl", [ci_shape()]) + b = write_events(tmp_path / "s" / "events.jsonl", [logging_shape()]) + + parsed = parse_events([a, b]) + + assert parsed["llm_responses"] == 1 + assert parsed["duplicate_responses"] == 1 + assert parsed["output_tokens"] == 398 + assert parsed["cost_usd"] == pytest.approx(0.1132975) + + +def test_distinct_calls_are_not_collapsed(tmp_path): + """The fix must not over-collapse: understating cost is the worse failure.""" + events = [ + ci_shape(response_id="msg_A", ts="2026-08-05T15:23:15.000000001+00:00"), + ci_shape(response_id="msg_B", ts="2026-08-05T15:23:16.000000002+00:00"), + ci_shape(response_id="msg_C", ts="2026-08-05T15:23:17.000000003+00:00"), + ] + path = write_events(tmp_path / "events.jsonl", events) + + parsed = parse_events([path]) + + assert parsed["llm_responses"] == 3 + assert parsed["duplicate_responses"] == 0 + assert parsed["output_tokens"] == 398 * 3 + + +def test_dedup_without_raw_capture_uses_timestamp_fingerprint(tmp_path): + """Raw capture is opt-in; de-duplication must still hold when it is off.""" + + def strip_raw(event: dict) -> dict: + event["data"].pop("raw", None) + return event + + a = write_events(tmp_path / "a" / "events.jsonl", [strip_raw(ci_shape())]) + b = write_events(tmp_path / "b" / "events.jsonl", [strip_raw(logging_shape())]) + + parsed = parse_events([a, b]) + + assert parsed["llm_responses"] == 1 + assert parsed["duplicate_responses"] == 1 + + +def test_distinct_calls_without_raw_still_distinct(tmp_path): + """Fingerprint must discriminate on timestamp, not collapse on equal usage.""" + first = ci_shape(ts="2026-08-05T15:23:15.000000001+00:00") + second = ci_shape(ts="2026-08-05T15:23:16.000000002+00:00") + for e in (first, second): + e["data"].pop("raw") + path = write_events(tmp_path / "events.jsonl", [first, second]) + + parsed = parse_events([path]) + + assert parsed["llm_responses"] == 2 + assert parsed["duplicate_responses"] == 0 + + +def test_unidentifiable_event_is_counted_not_dropped(tmp_path): + """No id and no timestamp: count it, flag it, never silently discard it. + + Dropping here would understate cost invisibly. Counting may overstate, which + is the failure mode that gets noticed and investigated. + """ + event = {"event": "llm:response", "data": {"usage": {"output_tokens": 10}}} + path = write_events(tmp_path / "events.jsonl", [event, dict(event)]) + + parsed = parse_events([path]) + + assert parsed["llm_responses"] == 2 + assert parsed["unidentified_responses"] == 2 + assert parsed["duplicate_responses"] == 0 + + +def test_notes_state_the_correction(tmp_path): + """A silent correction is one nobody can audit. It must appear in notes.""" + a = write_events(tmp_path / "a" / "events.jsonl", [ci_shape()]) + b = write_events(tmp_path / "b" / "events.jsonl", [logging_shape()]) + + record = normalize_metrics([a, b], source="test") + + assert record["llm_responses"] == 1 + assert "Dropped 1 duplicate" in record["notes"] + assert "more than one logging hook" in record["notes"] + + +def test_wallclock_unaffected_by_duplicates(tmp_path): + """Duplicates share timestamps, so the span must not change.""" + early, late = "2026-08-05T15:23:15.000000+00:00", "2026-08-05T15:23:45.000000+00:00" + single = write_events( + tmp_path / "one" / "events.jsonl", + [ci_shape(response_id="a", ts=early), ci_shape(response_id="b", ts=late)], + ) + dupe = write_events( + tmp_path / "two" / "events.jsonl", + [logging_shape(response_id="a", ts=early), logging_shape(response_id="b", ts=late)], + ) + + one_logger = parse_events([single]) + two_loggers = parse_events([single, dupe]) + + assert two_loggers["llm_responses"] == one_logger["llm_responses"] == 2 + assert two_loggers["agent_wallclock_s"] == pytest.approx(one_logger["agent_wallclock_s"]) + assert two_loggers["agent_wallclock_s"] == pytest.approx(30.0) diff --git a/.amplifier/evaluation/deep-swe/tests/test_opencode_metrics.py b/.amplifier/evaluation/deep-swe/tests/test_opencode_metrics.py new file mode 100644 index 00000000..9e493cf3 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/tests/test_opencode_metrics.py @@ -0,0 +1,257 @@ +"""Accuracy guards for the opencode-vanilla arm's token/cost numbers. + +Every test here exists because the failure it pins produces a WRONG number +rather than a missing one. A missing number is visible in the summary as `n/a` +and gets investigated; a plausible wrong number gets published. +""" + +from __future__ import annotations + +import json +import sqlite3 +from pathlib import Path + +import pytest + +from deepswe_agents.metrics import ( + NOT_AVAILABLE, + compute_cost_from_tokens, + normalize_opencode_metrics, + parse_opencode_db, +) +from deepswe_agents.opencode_vanilla import OpencodeVanillaAgent + +SESSION_DDL = """ +CREATE TABLE session ( + id TEXT PRIMARY KEY, + directory TEXT NOT NULL, + model TEXT, + cost REAL NOT NULL DEFAULT 0, + tokens_input INTEGER NOT NULL DEFAULT 0, + tokens_output INTEGER NOT NULL DEFAULT 0, + tokens_cache_read INTEGER NOT NULL DEFAULT 0, + tokens_cache_write INTEGER NOT NULL DEFAULT 0, + time_created INTEGER, + time_updated INTEGER +); +CREATE TABLE message (session_id TEXT, data TEXT); +""" + + +def _make_db( + path: Path, sessions: list[dict], assistant_turns: dict[str, int] | None = None +) -> str: + con = sqlite3.connect(path) + try: + con.executescript(SESSION_DDL) + for s in sessions: + con.execute( + "INSERT INTO session (id, directory, model, cost, tokens_input," + " tokens_output, tokens_cache_read, tokens_cache_write, time_created," + " time_updated) VALUES (?,?,?,?,?,?,?,?,?,?)", + ( + s["id"], + s["directory"], + s.get( + "model", + json.dumps({"id": "claude-sonnet-5", "providerID": "anthropic"}), + ), + s.get("cost", 0.0), + s.get("tokens_input", 0), + s.get("tokens_output", 0), + s.get("tokens_cache_read", 0), + s.get("tokens_cache_write", 0), + s.get("time_created", 1_700_000_000_000), + s.get("time_updated", 1_700_000_060_000), + ), + ) + for sid, count in (assistant_turns or {}).items(): + for _ in range(count): + con.execute( + "INSERT INTO message (session_id, data) VALUES (?,?)", + (sid, json.dumps({"role": "assistant"})), + ) + con.commit() + finally: + con.close() + return str(path) + + +# --------------------------------------------------------------------------- +# The workspace filter must not invent a total +# --------------------------------------------------------------------------- + + +def test_workspace_mismatch_yields_not_available_not_a_wrong_total(tmp_path): + """A wrong workspace_dir must produce nothing, never everything. + + The old `matched or sessions` fallback summed every session in the database + when the filter missed -- inflating a real /app run by whatever else the DB + happened to hold. That is the single most dangerous behavior in this parser. + """ + db = _make_db( + tmp_path / "opencode.db", + [ + {"id": "s1", "directory": "/home/someone", "cost": 4.0, "tokens_input": 5_000_000}, + {"id": "s2", "directory": "/tmp/other", "cost": 2.0, "tokens_input": 3_000_000}, + ], + ) + + parsed = parse_opencode_db([db], workspace_dir="/app") + + assert parsed["files_read"] == 0 + assert parsed["input_tokens"] == 0 + assert parsed["cost_usd"] == 0.0 + assert parsed["cost_from_events"] is False + + record = normalize_opencode_metrics([db], workspace_dir="/app") + assert record["cost_usd"] == NOT_AVAILABLE + assert record["input_tokens"] == NOT_AVAILABLE + assert record["llm_responses"] == NOT_AVAILABLE + # And it must say WHY, so "no data" is distinguishable from "wrong query". + assert "none with directory == '/app'" in record["notes"] + assert "2 session(s)" in record["notes"] + + +def test_only_matching_workspace_sessions_are_counted(tmp_path): + """Sessions from other directories must never leak into the task totals.""" + db = _make_db( + tmp_path / "opencode.db", + [ + { + "id": "task", + "directory": "/app", + "cost": 1.5, + "tokens_input": 1000, + "tokens_output": 200, + "tokens_cache_read": 50, + "tokens_cache_write": 75, + }, + {"id": "noise", "directory": "/root", "cost": 99.0, "tokens_input": 9_999_999}, + ], + assistant_turns={"task": 7, "noise": 3}, + ) + + parsed = parse_opencode_db([db], workspace_dir="/app") + + assert parsed["input_tokens"] == 1000 + assert parsed["output_tokens"] == 200 + assert parsed["cache_read_tokens"] == 50 + assert parsed["cache_write_tokens"] == 75 + assert parsed["llm_responses"] == 7, "assistant turns from /root must not be counted" + + +# --------------------------------------------------------------------------- +# Cost is recomputed from tokens, never read from opencode's own column +# --------------------------------------------------------------------------- + + +def test_cost_is_recomputed_and_ignores_opencodes_own_figure(tmp_path): + """opencode prices from a models.dev card that differs on cache rates. + + See `parse_opencode_db` in metrics.py for why its own figure is ignored. + """ + db = _make_db( + tmp_path / "opencode.db", + [ + { + "id": "task", + "directory": "/app", + # Deliberately absurd: if this leaks into the output, the test fails. + "cost": 999.0, + "tokens_input": 1_000_000, + "tokens_output": 1_000_000, + "tokens_cache_read": 1_000_000, + "tokens_cache_write": 1_000_000, + } + ], + assistant_turns={"task": 5}, + ) + + # 1M of each at the reference card: 3.00 + 15.00 + 0.30 + 3.75 + expected = 3.00 + 15.00 + 0.30 + 3.75 + + parsed = parse_opencode_db([db], workspace_dir="/app") + assert parsed["cost_usd"] == pytest.approx(expected) + assert parsed["cost_from_events"] is True + + record = normalize_opencode_metrics([db], workspace_dir="/app") + assert record["cost_usd"] == pytest.approx(expected) + assert record["cost_usd"] != pytest.approx(999.0) + # The divergence must be recorded, not silently swallowed. + assert "RECOMPUTED" in record["notes"] + assert "999.000000" in record["notes"], "opencode's own figure must be kept as an audit note" + + +def test_unknown_model_yields_not_available_not_a_wrong_price(tmp_path): + """No rate card => no dollar figure. Never 0.0, never opencode's number.""" + db = _make_db( + tmp_path / "opencode.db", + [ + { + "id": "task", + "directory": "/app", + "model": json.dumps({"id": "some-future-model", "providerID": "anthropic"}), + "cost": 7.5, + "tokens_input": 2_000_000, + } + ], + assistant_turns={"task": 40}, + ) + + record = normalize_opencode_metrics([db], workspace_dir="/app") + + assert record["cost_usd"] == NOT_AVAILABLE + # Tokens are still real and must survive. + assert record["input_tokens"] == 2_000_000 + assert record["llm_responses"] == 40 + assert "not in the reference rate card" in record["notes"] + + +def test_compute_cost_from_tokens_matches_the_card(): + assert compute_cost_from_tokens( + "claude-sonnet-5", + input_tokens=1_000_000, + output_tokens=2_000_000, + cache_read_tokens=10_000_000, + cache_write_tokens=4_000_000, + ) == pytest.approx(3.00 + 30.00 + 3.00 + 15.00) + + assert compute_cost_from_tokens("nope", input_tokens=1_000_000) is None + + +def _agent_with_model(model: str) -> OpencodeVanillaAgent: + """Build an adapter without pier's constructor. + + `model` is a read-only property computed from the parsed `--model` value, + so the backing attribute is what a test can set. + """ + agent = OpencodeVanillaAgent.__new__(OpencodeVanillaAgent) + agent._parsed_model_name = model # type: ignore[attr-defined] + return agent + + +def test_opencode_config_pins_cost_and_small_model(): + """The generated opencode.json must not depend on models.dev being reachable.""" + agent = _agent_with_model("claude-sonnet-5") + + config = json.loads(agent._opencode_config()) + + # Small model pinned: its default family ends at claude-haiku, which this + # endpoint does not serve, and the failure kills the process at exit 1. + assert config["small_model"] == "anthropic/claude-sonnet-5" + + model = config["provider"]["anthropic"]["models"]["claude-sonnet-5"] + assert model["cost"] == { + "input": 3.00, + "output": 15.00, + "cache": {"read": 0.30, "write": 3.75}, + } + + +def test_unknown_model_gets_no_fabricated_rate_card(): + """An unpriced model must fall through to `not_available`, not to wrong rates.""" + agent = _agent_with_model("some-future-model") + + config = json.loads(agent._opencode_config()) + assert "cost" not in config["provider"]["anthropic"]["models"]["some-future-model"] diff --git a/.amplifier/evaluation/deep-swe/tests/test_teardown_guards.py b/.amplifier/evaluation/deep-swe/tests/test_teardown_guards.py new file mode 100644 index 00000000..5f9b34d2 --- /dev/null +++ b/.amplifier/evaluation/deep-swe/tests/test_teardown_guards.py @@ -0,0 +1,290 @@ +"""Regression tests for cancellation-safe trial teardown. + +WHY THIS FILE EXISTS. pier runs the agent as +``asyncio.wait_for(agent.run(...), timeout=agent_timeout_sec)``. When the agent +burns its budget -- a LIKELY outcome on a full-budget deep-swe run, not an edge +case -- our task is cancelled and ``run()``'s ``finally`` block executes while +cancellation is in flight. A bare ``await`` there is cancelled too, and +``except Exception`` cannot save it because ``CancelledError`` inherits from +``BaseException``. + +So the timed-out trial -- the one whose trajectory, token spend and crash +evidence we most want -- is exactly the trial that would silently lose: + + - the fallback commit -> 0-byte model.patch, total loss of partial credit + - the session tree -> no events.jsonl, so no token or cost accounting + +`_await_guarded` prevents that. These tests pin the behaviour, including a +control test that demonstrates the loss WITHOUT the guard, so nobody can +conclude the shielding is decoration and "simplify" it away. + +Run: python3 -m pytest tests/ -x +""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from deepswe_agents import base +from deepswe_agents.base import FALLBACK_MARKER, AmplifierBaseAgent + +SESSION_DIR = "/root/.amplifier-agent/state/workspaces" + + +class StubAgent(AmplifierBaseAgent): + """Concrete agent exposing the real teardown methods and nothing else. + + ``BaseInstalledAgent.__init__`` wants pier runtime plumbing (environment + handles, install specs, model config) that has no bearing on the shielding + logic under test, so it is deliberately not called. ``__abstractmethods__`` + is cleared so instantiation works whether the real pier (an ABC) or the + conftest stub is in play. + """ + + __abstractmethods__ = frozenset() + + SESSION_DIRS = (SESSION_DIR,) + + def __init__(self, logs_dir: Path) -> None: + self.logs_dir = logs_dir + self.logger = logging.getLogger("test.stub-agent") + + @staticmethod + def name() -> str: + return "stub-agent" + + def agent_install_steps(self) -> list: + return [] + + def run_command(self, instruction_path: str) -> str: + return "true" + + +class SlowEnv: + """Environment whose transfers are slow enough to be interrupted mid-flight. + + Records what actually completed, so a test can tell "the work finished" + from "the call returned". + """ + + def __init__(self, delay: float = 0.15) -> None: + self.delay = delay + self.started = asyncio.Event() + self.completed: list[str] = [] + + async def download_dir(self, source_dir: str, target_dir: Path | str) -> None: + self.started.set() + await asyncio.sleep(self.delay) + target = Path(target_dir) + target.mkdir(parents=True, exist_ok=True) + (target / "events.jsonl").write_text('{"event":"llm:response"}\n', encoding="utf-8") + self.completed.append(source_dir) + + +class HangingEnv: + """Environment whose transfers never return (a wedged container).""" + + def __init__(self) -> None: + self.started = asyncio.Event() + + async def download_dir(self, source_dir: str, target_dir: Path | str) -> None: + self.started.set() + await asyncio.sleep(3600) + + +class CommitEnv: + """Environment standing in for the container during the fallback commit.""" + + def __init__(self, delay: float = 0.15) -> None: + self.delay = delay + self.started = asyncio.Event() + self.commands: list[str] = [] + + def agent_process_env(self, env: dict) -> dict: + return env + + async def exec(self, command: str, env: dict | None = None, user: str | None = None): + self.started.set() + await asyncio.sleep(self.delay) + self.commands.append(command) + return SimpleNamespace(stdout=f"{FALLBACK_MARKER}: committed", stderr="") + + +async def _cancelled_trial(teardown, env, n_cancels: int = 1) -> None: + """Reproduce pier's timeout: cancel a running agent task, teardown in finally. + + *n_cancels* is how many times the trial task is cancelled. The first lands + on the simulated agent command; any further ones land while teardown is + mid-transfer, which is the case a bare ``await`` cannot survive. + """ + + async def trial() -> None: + try: + await asyncio.sleep(3600) # the agent command, still running + finally: + await teardown() + + task = asyncio.create_task(trial()) + await asyncio.sleep(0.02) # let it reach the agent command + task.cancel() # pier's wait_for timeout fires + + for _ in range(n_cancels - 1): + await asyncio.wait_for(env.started.wait(), timeout=5) # transfer in flight + task.cancel() + await asyncio.sleep(0) # let the cancellation be delivered + + with pytest.raises(asyncio.CancelledError): + await task + + +def test_session_collection_completes_when_task_cancelled_mid_download(tmp_path): + """THE REQUIRED PROPERTY: a timed-out trial still yields its session tree.""" + agent = StubAgent(tmp_path) + env = SlowEnv() + + asyncio.run( + _cancelled_trial(lambda: agent._collect_session_dirs_guarded(env), env, n_cancels=1) + ) + + assert env.completed == [SESSION_DIR] + assert (tmp_path / "sessions" / "workspaces" / "events.jsonl").exists() + + +def test_session_collection_survives_repeated_cancellation(tmp_path): + """Cancellation delivered again DURING teardown must not lose the work. + + This is what the bounded retry around `asyncio.shield` buys: the first + cancellation interrupts our await, not the download task. + """ + agent = StubAgent(tmp_path) + env = SlowEnv() + + asyncio.run( + _cancelled_trial(lambda: agent._collect_session_dirs_guarded(env), env, n_cancels=3) + ) + + assert env.completed == [SESSION_DIR] + assert (tmp_path / "sessions" / "workspaces" / "events.jsonl").exists() + + +def test_unguarded_collection_loses_the_work_under_cancellation(tmp_path): + """CONTROL: proves the shielding is load-bearing, not ceremony. + + Same scenario against the raw coroutine. `except Exception` inside it does + NOT catch `CancelledError`, so the download dies mid-flight and the trial + ends with no session tree -- the exact data loss the guard exists to stop. + If this test ever starts passing with data collected, the hazard is gone + and the guard can be reconsidered. + """ + agent = StubAgent(tmp_path) + env = SlowEnv() + + # _cancelled_trial already asserts the trial task ends cancelled; what + # matters here is that the download did NOT survive it. + asyncio.run(_cancelled_trial(lambda: agent._collect_session_dirs(env), env, n_cancels=2)) + + assert env.completed == [] + assert not (tmp_path / "sessions" / "workspaces" / "events.jsonl").exists() + + +def test_fallback_commit_still_completes_under_cancellation(tmp_path): + """Equivalence guard: the pre-existing commit shielding is unchanged. + + `_fallback_commit_guarded` now delegates to the shared `_await_guarded`. + This is the property that refactor must not have broken -- losing it means + a 0-byte model.patch on every timed-out trial. + """ + agent = StubAgent(tmp_path) + env = CommitEnv() + + asyncio.run(_cancelled_trial(lambda: agent._fallback_commit_guarded(env, {}), env, n_cancels=3)) + + assert len(env.commands) == 1 + assert "git commit" in env.commands[0] + + +def test_wedged_container_cannot_stall_teardown_forever(tmp_path, monkeypatch, caplog): + """A hung download must hit the hard timeout, not hang trial teardown.""" + monkeypatch.setattr(base, "TEARDOWN_COLLECT_TIMEOUT_SEC", 0.1) + agent = StubAgent(tmp_path) + env = HangingEnv() + + async def scenario() -> float: + loop = asyncio.get_running_loop() + start = loop.time() + await agent._collect_session_dirs_guarded(env) + return loop.time() - start + + with caplog.at_level(logging.WARNING): + elapsed = asyncio.run(scenario()) + + assert elapsed < 2.0, f"teardown took {elapsed:.2f}s; the hard timeout did not fire" + assert "Could not collect session dir" in caplog.text + + +def test_timeout_budget_is_shared_across_session_dirs(tmp_path, monkeypatch, caplog): + """The budget is overall, so adding SESSION_DIRS cannot extend teardown.""" + monkeypatch.setattr(base, "TEARDOWN_COLLECT_TIMEOUT_SEC", 0.1) + agent = StubAgent(tmp_path) + agent.SESSION_DIRS = ("/a/one", "/b/two", "/c/three") + env = HangingEnv() + + async def scenario() -> float: + loop = asyncio.get_running_loop() + start = loop.time() + await agent._collect_session_dirs_guarded(env) + return loop.time() - start + + with caplog.at_level(logging.WARNING): + elapsed = asyncio.run(scenario()) + + # Three dirs must NOT cost three timeouts. + assert elapsed < 0.1 * 3, f"budget was per-directory, not overall ({elapsed:.2f}s)" + assert "budget" in caplog.text + + +def test_guard_gives_up_after_bounded_retries(tmp_path, caplog): + """Relentless cancellation must terminate, not spin forever. + + The retry bound is what makes the guard safe to put in a `finally`: it + cannot become an infinite loop that outlives the trial. + """ + agent = StubAgent(tmp_path) + + async def scenario() -> None: + work = asyncio.sleep(3600) # never completes on its own + guard = asyncio.ensure_future(agent._await_guarded(work, "Session collection")) + for _ in range(6): # more cancellations than the retry bound allows + await asyncio.sleep(0.01) + if guard.done(): + break + guard.cancel() + await asyncio.sleep(0.05) + assert guard.done(), "the guard spun instead of giving up" + with contextlib.suppress(asyncio.CancelledError): + await guard + + with caplog.at_level(logging.WARNING): + asyncio.run(scenario()) + + assert "could not be awaited to completion" in caplog.text + + +def test_guarded_teardown_never_raises_on_failure(tmp_path, caplog): + """A broken environment must be logged, never propagated out of teardown.""" + agent = StubAgent(tmp_path) + + class BoomEnv: + async def download_dir(self, source_dir: str, target_dir: Path | str) -> None: + raise RuntimeError("container gone") + + with caplog.at_level(logging.WARNING): + asyncio.run(agent._collect_session_dirs_guarded(BoomEnv())) + + assert "container gone" in caplog.text diff --git a/notes/e2e-coverage-gaps.md b/notes/e2e-coverage-gaps.md deleted file mode 100644 index 8981434c..00000000 --- a/notes/e2e-coverage-gaps.md +++ /dev/null @@ -1,129 +0,0 @@ -# E2E coverage gaps - -Queue of behavior that lost its only coverage when the unit test suite was -deleted. Each entry is a candidate e2e suite, not a bug. - -This file exists so the loss is explicit rather than silent. Work it with -`/amplifier-agent-new-feature`, one suite at a time, highest risk first. - -## Context - -The repo previously carried ~920 in-process tests across `tests/*.py`, -`tests/cli/`, `tests/http/`, `tests/config/`, `tests/integration/`, and -`tests/bundle/`. They were deleted because they tested implementation rather -than contract, and because their existence made `pytest tests/` mean two -different things depending on whether a DTU happened to be warm. - -What survived that deletion: - -``` -tests/e2e/ 8 suites, 56 cases, the contract -scripts/verify-* release and contract guards, extracted not deleted -.amplifier/evaluation/ 8 capability tasks, probabilistic behavior -[tool.ruff.lint] T20 stdout discipline, formerly a test -``` - -## Covered today - -Collected test counts, not `E2ECase` counts. The two differ because suites -parametrize. Measured with `uv run pytest tests/e2e/suites/ --collect-only -q`. - -``` -modes tests/e2e/suites/modes/ 14 -skills tests/e2e/suites/skills/ 13 -github_copilot tests/e2e/suites/github_copilot/ 11 needs GITHUB_TOKEN -shadowing tests/e2e/suites/shadowing/ 6 -streaming tests/e2e/suites/streaming/ 4 -launch_dir tests/e2e/suites/launch_dir/ 3 -raw_capture tests/e2e/suites/raw_capture/ 3 -run tests/e2e/suites/run/ 2 - -- - 56 -``` - -45 of those run without `GITHUB_TOKEN`. All 45 verified green in a DTU on -2026-08-03. - -## Not covered by anything - -Ordered by how badly a silent regression would hurt a user. - -``` -session persistence and resume - was: test_persistence.py, test_session_store.py, test_resume_continuity.py, - test_incremental_save.py, test_transcript_repair.py - why it matters: a user losing conversation history has no workaround. - Resume is also the most stateful path in the product. - candidate: tests/e2e/suites/persistence/ - write a turn, restart, resume, - assert the prior turn is in context - -workspace isolation - was: test_persistence_workspaces.py, test_session_store_per_workspace.py, - test_session_store_cross_workspace_load.py, test_runtime_workspace.py, - test_runtime_fresh_workspace.py, test_spawn_workspace_propagation.py - why it matters: cross-workspace leakage is a correctness AND privacy bug. - candidate: tests/e2e/suites/workspaces/ - two workspaces, assert sessions - do not bleed across - -XDG / config migration - was: test_xdg_migration.py, test_migration.py, test_runtime_migration_wired.py - why it matters: runs once, on upgrade, on a real user's machine, with their - only copy of their data. Failure is unrecoverable and - invisible until it is too late. - candidate: tests/e2e/suites/migration/ - provision a DTU with an old-layout - home, upgrade, assert data survived - -subagent spawn - was: test_spawn.py, test_spawn_capability_inheritance.py, - cli/test_delegation_e2e.py - why it matters: delegation is a headline capability and the inheritance - rules are subtle. - candidate: tests/e2e/suites/spawn/ - delegate, assert the child ran with the - expected tool set and the result came back - -runtime config merge - was: test_runtime_config_merge.py, test_runtime_initialize_cwd.py, - test_runtime_audit_path.py, test_runtime_hook_mount.py - why it matters: precedence bugs are quiet and produce wrong behavior rather - than errors. - -MCP threading - was: test_runtime_mcp_threading.py - why it matters: no e2e suite touches MCP at all today. - -approval provider wiring - was: test_wire_approval_provider.py - why it matters: approvals are a safety surface. - -wrapper hang detection over a long real turn - was: wrappers/typescript/test/timeout-longwindow-integration.test.ts - why it was deleted: it spawned a mock engine that slept 12 real seconds, - twice, which was 24s of the 25s TypeScript suite and ~40% of total CI - wall clock. Its three cases are already covered faster and better by - session-subprocess.test.ts, which uses 300ms observation windows: - (k) timeoutMs: 0 -> no engine_hung (same regression guard) - (l) timeoutMs: undefined -> no engine_hung (no silent default) - (e) timeoutMs: 250 -> engine_hung fires - (j) timeoutMs: 150 -> engine_hung fires - The 12s sleep also never tested what its comment claimed: the silent - default it guarded against was 10 minutes (session.ts DEFAULT_TIMEOUT_MS), - which a 12 second window cannot detect. - what is genuinely uncovered now: the same contract over a REAL long-running - agent turn through the public spawnAgent() API, rather than a mock engine - through SessionHandle. That is an e2e concern, not a unit one. - candidate: an e2e case that runs a genuinely slow turn and asserts it - completes without a spurious hang error. See ISSUE-002 in ISSUES.md, - which proposes progress-based stuck detection and will need this anyway. - -HTTP surface - was: tests/http/ (80 tests, FastAPI TestClient) - note: tests/e2e/ does start a real server on :9099, so this is partially - covered. Audit before writing anything new. -``` - -## Deliberately not backfilled - -Anything that only ever asserted internal structure. Examples: import -graph shape, dataclass field presence, error-message wording. If a behavior -cannot be observed through the CLI, the HTTP API, or an evaluation, it is not -part of the contract and does not get a test. diff --git a/notes/foundation-pin-reproducibility.md b/notes/foundation-pin-reproducibility.md deleted file mode 100644 index d6b65c3e..00000000 --- a/notes/foundation-pin-reproducibility.md +++ /dev/null @@ -1,82 +0,0 @@ -# The tested artifact is not the shipped artifact - -Known gap, deliberately not fixed yet. Recorded so it is not rediscovered. - -## The problem - -``` -pyproject.toml:79-80 - amplifier-foundation = { git = '...', branch = 'main' } -``` - -`amplifier-foundation` is pinned to a moving branch, not a commit or a tag. - -`uv.lock` exists, but the customer install path does not consume it: - -``` -install.sh:182-184 - uv tool install --reinstall --force \ - --from "git+https://github.com/microsoft/amplifier-agent@$TAG" amplifier-agent -``` - -`uv tool install --from git+...` resolves dependencies fresh. It does not read -the lockfile committed at that tag. - -## What that means - -An engine installed from tag `v0.12.0` resolves whatever `amplifier-foundation@main` -happens to be at install time, which may be months after the tag was cut. - -Two users installing the same tag on different days can get different code. -The artifact verified before release and the artifact a customer receives are -provably not the same thing. - -## Why no CI change closes it - -Every gate added in the e2e-first cleanup verifies the tree at a point in time: - -``` -make verify lint, types, codegen freshness, wheel contents, version - consistency, cross-language wire parity -ci.yml now runs on tags, so the tag is gated -publish-python.yml runs scripts/verify-wheel.py before upload -install-script.yml installs the real pushed tag and exercises bundle priming -``` - -All of these are correct and worth having. None of them constrain what -`foundation@main` will be tomorrow. The install-script smoke test comes closest, -but it proves the install worked at that moment, not that it will keep working. - -## Shape of a fix - -Not decided. The options, roughly: - -``` -pin at release time resolve foundation to a commit sha during the release - process and commit that pin with the version bump. - Reproducible, but adds a step and a coordination burden. - -publish foundation give foundation real versioned releases and depend on a - version range instead of a branch. Correct long-term, - largest change. - -ship the lock make the install path consume uv.lock. Needs a different - install mechanism than `uv tool install --from git+`. -``` - -This needs a decision about how tightly the two repos should be coupled, which -is an architecture call rather than a CI fix. - -## Related - -The same class of problem, smaller blast radius: - -``` -release-notes.yml:38 prerelease: contains(github.ref_name, '-') -``` - -Every `wrapper-v*` tag contains a hyphen, so every TypeScript wrapper release is -marked prerelease. This is load-bearing by accident: it is what stops wrapper -tags from winning the `releases/latest` lookup that `install.sh:25` depends on. -Fixing it "properly" would break the default install path. Leave it alone -until the install path stops depending on `releases/latest`.