Add deep-swe benchmark harness and document it in the evaluation README - #123
Merged
Merged
Conversation
deep-swe is a 113-task agentic SWE benchmark whose tasks are Harbor-format directories pinned to prebuilt Docker images. The pier CLI owns the environment, verification and grading, so this harness is kept separate from the DTU-based matrix harness rather than folded into it: there are no DTUs, graders or task definitions here, only the agent adapters and a runner. Covers the same four arms as the matrix harness (amplifier-agent, amplifier-foundation, opencode-amplifier-agent, opencode-vanilla). Task data is cloned at runtime and never vendored. Adds a high-level section to the evaluation README so the harness is discoverable from the top level, with setup and correctness constraints left in deep-swe/README.md. Also removes the two remaining notes/ files, which were left behind by the earlier move to e2e-first testing. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
.amplifier/evaluation/deep-swe/, a harness for deep-swe (a 113-task agentic SWE benchmark), and documents it at a high level in.amplifier/evaluation/README.md.Why it is separate from the matrix harness
deep-swe tasks are Harbor-format directories pinned to prebuilt Docker images, and the
pierCLI (a Harbor fork) owns the environment, verification and grading. Folding this into the existing DTU-based matrix harness would mean reimplementing what pier already does. So this directory carries no DTUs, graders or task definitions, only the agent adapters and a thin runner.Contents
Covers the same four arms as the matrix harness:
amplifier-agent,amplifier-foundation,opencode-amplifier-agent,opencode-vanilla.Scoring is a binary
reward(all fail-to-pass and pass-to-pass tests pass) plus apartialfraction that is the more useful dev signal. Task data is cloned at runtime into~/.cache/deep-swe/<sha>/and is never vendored into this repo. Results land under<workspace>/evaluation_results/, notruns/.Docs
The evaluation README gets a
deep-swe/line in its Layout block and a high-level section, so the harness is discoverable from the top level. Setup (Docker,pierinstalled from git, agents installed into pier's venv) and the constraints that keep scores comparable stay indeep-swe/README.md.Also
Removes
notes/e2e-coverage-gaps.mdandnotes/foundation-pin-reproducibility.md, left behind by the earlier move to e2e-first testing.