Skip to content

docs: research paper + presenter pack (written from recorded results only) - #22

Merged
ksdisch merged 1 commit into
mainfrom
docs/paper-and-presenter-pack
Aug 1, 2026
Merged

docs: research paper + presenter pack (written from recorded results only)#22
ksdisch merged 1 commit into
mainfrom
docs/paper-and-presenter-pack

Conversation

@ksdisch

@ksdisch ksdisch commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What this is

Two Markdown deliverables under docs/paper/, produced entirely from numbers already recorded in the repo — no new measurements, no model/API calls, no figure regeneration. This PR is for your review; per the mission it is not merged.

  • docs/paper/forge-gap-paper.md (~4,300 words) — a professional empirical-methods paper: abstract, introduction, background/related work, methods, experimental setup, results (with the six committed figures embedded and a consolidated results table), discussion, limitations, conclusion, provenance note, references.
  • docs/paper/forge-gap-presenter-pack.md (~1,500 words) — the defense companion: one-minute story, claim-by-claim defenses, a provenance table (claim → number → source file), anticipated hard questions with crisp answers, and how to trace any figure live.

Headline results, as written in the paper

Experiment (stage) Result Verdict
Error-recovery on injected transient faults, GLM-4.6, N=40 67.5% → 100%, +32.5 pp, Newcombe 95% [+17.3, +48.0] real
Retry-nudge (and error-recovery) on injected malformed calls, GLM-4.6, N=20 all arms 100%, +0.0 pp [−16.1, +16.1] null — reported as one
Submit-nudge on mistral-nemo's natural no-submit gap, N=20 0% → 75%, +75.0 pp [+47.8, +88.8]; retry-nudge null (0 fires) in the same run real
Validation stacked on submit-nudge, mistral-nemo, N=40 75% → 100%, +25.0 pp [+11.1, +40.2] real
Validation un-stacked on llama-3.1-8b's hallucination gap, N=40 0% → 45%, +45.0 pp [+28.2, +60.2]; 55% residual decomposed (35% / 10% / 7.5% / 2.5%) real

Honesty framing held throughout: injected gaps labeled injected on every mention; both nulls reported at full prominence; nemo's capstone +100.0 pp [+81.7, +100.0] explicitly labeled cross-run; GLM-4.6 given no guardrail bar (none was ever measured on its clean task); validation presented as a self-consistency check with its fooled-152 proof and measured blind spot.

Sources of truth used

All arm rates, CIs, and mechanism-fire counts: docs/figures/*-data.json. Pilots, sizing analyses, the S10 residual decomposition, and robustness probes: docs/DECISIONS.md (D12–D24) with ROADMAP/README cross-checks. Design facts: scenario.py, agent.py, faults.py, stats.py. The 77-test count was verified by direct grep. No numbers were taken from runs/.

Citations

The repo names no publications for the guardrail primitives, so the paper frames them as reproduced common practice (stated explicitly in References) and cites only the canonical statistics references the repo names by method: Wilson (1927) and Newcombe (1998, method 10).

Flagged gaps (numbers that don't exist in the repo)

  • Per-run token/dollar costs for the main ablations were never recorded (only the S7 pilot's token counts exist, in D20). Both documents therefore make no absolute cost claims; this is disclosed in the paper's Limitations.

Scope

Only the two new files — no project code, tests, figures, or existing docs/ spine files were touched.

🤖 Generated with Claude Code

Two new deliverables under docs/paper/, written entirely from numbers
already recorded in the repo (docs/figures/*-data.json + the docs/ spine)
— no new measurements, no model calls, no figure regeneration:

- forge-gap-paper.md — an empirical-methods paper (abstract, intro,
  background, methods, setup, results, discussion, limitations,
  conclusion). Reports all five measured deltas with their Wilson/
  Newcombe 95% CIs, INCLUDING the two nulls (S6 malformed; S8
  retry-nudge control), labels the S4/S6 gaps as injected, embeds the
  six committed figures with honest captions, and cites only the
  canonical Wilson (1927) / Newcombe (1998) statistics references —
  the guardrail primitives are framed as reproduced common practice,
  per the repo's honesty rule (D1).

- forge-gap-presenter-pack.md — claim-by-claim defense, a provenance
  table (claim -> number -> source file), anticipated hard questions
  (manufactured gap, null-as-result, Wilson/Newcombe, answer-key
  objection, un-validatable residual, cross-run capstone gap, roads
  not taken), and how to trace any figure live.

Flagged in-text where a number was never recorded: per-run token/
dollar costs for the main ablations (only the S7 pilot's token counts
exist in D20), so neither document makes absolute cost claims.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@ksdisch
ksdisch merged commit a60c64d into main Aug 1, 2026
1 check passed
@ksdisch
ksdisch deleted the docs/paper-and-presenter-pack branch August 1, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant