Skip to content

feat(benchmarks): add 25-defect dataset, manifest loader, and Pass@k evaluation harness - #5

Merged
Abdurrafay19 merged 18 commits into
masterfrom
bench/defect-evaluation
Sep 11, 2026
Merged

feat(benchmarks): add 25-defect dataset, manifest loader, and Pass@k evaluation harness#5
Abdurrafay19 merged 18 commits into
masterfrom
bench/defect-evaluation

Conversation

@Abdurrafay19

Copy link
Copy Markdown
Owner

What this adds

Implements the benchmark evaluation harness described in the project spec: a
25-defect synthetic dataset with oracle tests, a typed manifest loader, and a
harness that runs the full LangGraph agent against each defect and grades the
result against a hidden oracle test — never against the SLM's own
self-written tests, since a model that writes weak tests would otherwise
score as "fixed" even when it isn't.

Files added

  • benchmarks/manifest.py — Pydantic schema (DefectRecord, DefectManifest)
    and loaders for the dataset manifest, source files, and oracle tests. Fails
    fast (raises) on missing files or malformed JSON, unlike the tools in
    patchwork/tools/, since this loads a fixed local dataset rather than
    untrusted SLM output or flaky subprocesses. Validates defect IDs are unique
    at load time via a model_validator, not just in tests.
  • benchmarks/dataset/ — 25 buggy Python files across 5 categories (mutable
    default arguments, unhandled NoneType, off-by-one slicing, float precision,
    resource leaks), each with a corresponding oracle test in
    benchmarks/dataset/oracle_tests/.
  • benchmarks/evaluate.pyevaluate_defect() runs one defect through
    build_patchwork_graph() and grades the final patched code against the
    oracle; evaluate_all() runs the full 25-defect set and computes Pass@1 /
    Pass@overall rates plus VRAM/duration telemetry via profile_call.
    Crash-isolated per defect: a transport-level failure
    (RuntimeError/ConnectionError/TimeoutError/OSError) on one defect
    is caught and recorded, not allowed to kill the rest of the batch.
  • tests/test_manifest.py, tests/test_evaluate.py — real-dataset,
    real-subprocess integration tests. structured_llm is mocked everywhere
    (no live Ollama call in CI); grading, sandbox execution, and file I/O are
    real.

Config changes

  • pyproject.toml — added [tool.pytest.ini_options] (testpaths),
    [tool.ruff] (target-version, extend-exclude for
    benchmarks/dataset/, which is intentionally-broken specimen code not
    meant to pass lint/type checks), [tool.mypy] (strict = true made
    permanent, exclude for the same dataset path), and a pynvml
    stub-suppression override consumed by patchwork/telemetry/profiler.py.

Testing

128 tests passing. Coverage: manifest.py 100%, evaluate.py 74% (remaining
gap is main()/save_results()/_print_summary() — CLI glue meant to run
once against a real model, not unit-tested against a mock). Total project
coverage 90%, above the ≥80% target in the project spec.

ruff check, ruff format --check, and mypy --strict all clean across
patchwork/, tests/, benchmarks/.

…andled_nonetype, off_by_one_slicing, float_precision, resource_leaks)
…egories including mutable defaults, float precision, off-by-one errors, and resource leaks
@Abdurrafay19 Abdurrafay19 self-assigned this Sep 11, 2026
@Abdurrafay19 Abdurrafay19 added the enhancement New feature or request label Sep 11, 2026
@Abdurrafay19
Abdurrafay19 merged commit 1e3ca40 into master Sep 11, 2026
4 checks passed
@Abdurrafay19
Abdurrafay19 deleted the bench/defect-evaluation branch September 11, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant