docs: research paper + presenter pack (written from recorded results only) - #22
Merged
Conversation
Two new deliverables under docs/paper/, written entirely from numbers already recorded in the repo (docs/figures/*-data.json + the docs/ spine) — no new measurements, no model calls, no figure regeneration: - forge-gap-paper.md — an empirical-methods paper (abstract, intro, background, methods, setup, results, discussion, limitations, conclusion). Reports all five measured deltas with their Wilson/ Newcombe 95% CIs, INCLUDING the two nulls (S6 malformed; S8 retry-nudge control), labels the S4/S6 gaps as injected, embeds the six committed figures with honest captions, and cites only the canonical Wilson (1927) / Newcombe (1998) statistics references — the guardrail primitives are framed as reproduced common practice, per the repo's honesty rule (D1). - forge-gap-presenter-pack.md — claim-by-claim defense, a provenance table (claim -> number -> source file), anticipated hard questions (manufactured gap, null-as-result, Wilson/Newcombe, answer-key objection, un-validatable residual, cross-run capstone gap, roads not taken), and how to trace any figure live. Flagged in-text where a number was never recorded: per-run token/ dollar costs for the main ablations (only the S7 pilot's token counts exist in D20), so neither document makes absolute cost claims. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Two Markdown deliverables under
docs/paper/, produced entirely from numbers already recorded in the repo — no new measurements, no model/API calls, no figure regeneration. This PR is for your review; per the mission it is not merged.docs/paper/forge-gap-paper.md(~4,300 words) — a professional empirical-methods paper: abstract, introduction, background/related work, methods, experimental setup, results (with the six committed figures embedded and a consolidated results table), discussion, limitations, conclusion, provenance note, references.docs/paper/forge-gap-presenter-pack.md(~1,500 words) — the defense companion: one-minute story, claim-by-claim defenses, a provenance table (claim → number → source file), anticipated hard questions with crisp answers, and how to trace any figure live.Headline results, as written in the paper
Honesty framing held throughout: injected gaps labeled injected on every mention; both nulls reported at full prominence; nemo's capstone +100.0 pp [+81.7, +100.0] explicitly labeled cross-run; GLM-4.6 given no guardrail bar (none was ever measured on its clean task); validation presented as a self-consistency check with its fooled-152 proof and measured blind spot.
Sources of truth used
All arm rates, CIs, and mechanism-fire counts:
docs/figures/*-data.json. Pilots, sizing analyses, the S10 residual decomposition, and robustness probes:docs/DECISIONS.md(D12–D24) with ROADMAP/README cross-checks. Design facts:scenario.py,agent.py,faults.py,stats.py. The 77-test count was verified by direct grep. No numbers were taken fromruns/.Citations
The repo names no publications for the guardrail primitives, so the paper frames them as reproduced common practice (stated explicitly in References) and cites only the canonical statistics references the repo names by method: Wilson (1927) and Newcombe (1998, method 10).
Flagged gaps (numbers that don't exist in the repo)
Scope
Only the two new files — no project code, tests, figures, or existing
docs/spine files were touched.🤖 Generated with Claude Code