Skip to content

Releases: Connorrmcd6/surface-bench

surface-bench v1.0.0 — confirmatory dataset

Choose a tag to compare

@Connorrmcd6 Connorrmcd6 released this 16 Jun 18:19

First public release of the surface-bench dataset and harness.

What's included

  • Pre-registered (prereg-v2-multi, surf 0.6.2), provider-agnostic benchmark of documentation drift in LLM coding agents.
  • Confirmatory multi-turn matrix: 5 models across Anthropic, OpenAI, and Google; 13 cascade scenarios × 5 conditions × 10 trials = 3,250 graded completions, 0 errors.
  • Single-shot pilot: 3 Claude models, 1,320 graded completions.
  • Deterministic grading (no LLM judge); raw per-call data released and re-gradeable.

Headline finding
A confident stale doc drives the misled rate to 68–100% on every model — capability does not buy resistance — via three distinct failure modes (blind obedience, suppressed-but-rescuable verification, and verify-then-defer).

Licensing
Harness code MIT; dataset, scenarios, and research docs CC-BY-4.0. See DATASET.md.