Skip to content

Milestones

List view

  • Surface validation — generalization & blog

    Generalization + publication for the Surface validation study (follow-up to #11 and #12). Establishes that the effect isn't Claude-specific and turns it into the published write-up. 1. **Multi-provider sweep.** The `Model` adapter is provider-agnostic by design; add **OpenAI (GPT)** and **Gemini** adapters (including tool-use for the #12 loop) and run the same scenario suite across families, so "a more capable model is no more rot-resistant" doesn't rest on three Claude models. Report deltas per provider. 2. **External-publication rigor (#117).** Pre-registration (git-tagged before the run), oracle/sanity tripwires, ABC-checklist (arXiv 2507.02825) compliance, Holm–Bonferroni correction, per-scenario breakdown, and a transparent limitations section. 3. **Blog post.** Methodology, the **verification-suppression** headline, the worse-than-nothing finding (C1<C0), the token-cost story, cross-provider results, and the honest "what this does / doesn't assess" framing. **Deferred to stretch / v3:** the real-OSS-repo case study (#109) — highest-effort, hardest-to-grade item; revisit as an external-validity credibility anchor once the core synthetic result holds. Depends on #11 (and #12) for results to publish.

    No due date
    •4/5 issues closed
    80% complete1 open4 closed

  • Surface validation — agentic compounding (T4)

    Multi-turn agentic track for the Surface validation study (follow-up to #11). **Thesis (sharpened in v2):** the single-shot pilot (#11) showed stale docs break agents when the drifted dependency is hidden — but with that dependency knowable *only* from the doc, "follow the stale doc" is partly rational, so the result is attackable as true-by-construction. This milestone gives the agent a **read-only tool surface** (`read_file`/`grep`/`list_dir` + `final_answer`) so it can *choose* to verify the dependency, and measures whether a confident **stale doc suppresses that verification**: - **H4:** verification_rate(C1) < verification_rate(C0) - **H5** (mediation): within C1, agents that read the hidden dependency are correct; those that don't are misled That is the non-tautological core of the published result. **Design.** Read-only tools only — no edit/run: a test runner would let the agent brute-force ground truth and wash out the doc-trust signal we measure (a full edit/run "thrash" loop that quantifies compounding cost is deferred future work). Grading stays **byte-identical** to #11 because `final_answer` carries the `VERDICT:` line / `FILE:` blocks. A per-trial sandbox copies `code/` **including the hidden dependency**, which the prompt still withholds. Conditions C0–C3 carry over plus a new **Cw warning-only control** (#116). Turns/tokens ("compounding cost") are kept as a *secondary* metric. Built on the #11 harness (scenario format, conditions, graders, provider-agnostic models). Out of scope here: multi-provider sweep and real-OSS case study (see #13).

    No due date
    •6/6 issues closed
    100% complete0 open6 closed

  • Empirical validation of Surface

    Not tied to a release. A standardized, reproducible, provider-agnostic benchmark quantifying how much documentation *accuracy* changes agent task performance — the gap Surface exists to protect. **Causal framing (kept honest):** Surface doesn't make an agent smarter; it stops docs silently rotting. Its value = the performance delta between an agent on FRESH docs vs ROTTED docs. We measure that delta using drift of exactly the kind `surf check` catches. **Conditions** (same code+task; only the context block differs): C0 code-only · C1 stale doc (ungoverned world) · C2 fresh doc (Surface-governed world) · C3 stale doc + real `surf check --format json` report. **Hypotheses:** H1 C2>C1 (accuracy beats rot — core value) · H2 C1<C0 (rotted docs worse than nothing — the headline) · H3 C3≈C2 (surfacing drift recovers the loss). Harness: Python under `bench/` (separate top-level dir, outside the Rust workspace; the core stays no-network/deterministic). Scenarios are synthetic curated fixtures with deterministic graders (hidden tests for code-edit, structured-verdict rubric for QA) + a secondary 'misled rate'. If the pilot shows a clear effect, the same harness runs across multiple models/providers for a blog post. Bench may later be extracted to a standalone `surface-bench` repo — it has no inbound deps on the core.

    No due date
    •8/8 issues closed
    100% complete0 open8 closed