Releases: Connorrmcd6/surface-bench
Releases · Connorrmcd6/surface-bench
Release list
surface-bench v1.0.0 — confirmatory dataset
First public release of the surface-bench dataset and harness.
What's included
- Pre-registered (
prereg-v2-multi, surf 0.6.2), provider-agnostic benchmark of documentation drift in LLM coding agents. - Confirmatory multi-turn matrix: 5 models across Anthropic, OpenAI, and Google; 13 cascade scenarios × 5 conditions × 10 trials = 3,250 graded completions, 0 errors.
- Single-shot pilot: 3 Claude models, 1,320 graded completions.
- Deterministic grading (no LLM judge); raw per-call data released and re-gradeable.
Headline finding
A confident stale doc drives the misled rate to 68–100% on every model — capability does not buy resistance — via three distinct failure modes (blind obedience, suppressed-but-rescuable verification, and verify-then-defer).
Licensing
Harness code MIT; dataset, scenarios, and research docs CC-BY-4.0. See DATASET.md.