feat(s08): surface test-integrity findings in the PR body and the run webhook (default off)

AUDIT_OPEN.md

@@ -17,6 +17,16 @@
   (neutron#6, teploy-cli#18/#19/#20 — links above/below).
 - The integration-check test-quoting defect (spaced checkout paths) was
   fixed in PR #56.
+- **S08 PR-body/webhook surfacing built (PR #58, this wave):** the detector's
+  findings now reach the pull request's Verification section and the
+  run-completion webhook behind `SHIP_TEST_INTEGRITY_SURFACING=off|shadow|on`
+  (default off; invalid values are off with a log). Advisory only — not a
+  gate, not in the delivery record, and no step is added to the durable log
+  in any mode. The one step-sequence effect, under `on` only: a run whose
+  only verification part is an integrity finding now records the existing
+  `verification` step. Clean diffs surface nothing (pinned by test). Not one
+  live run; soak `shadow` on the real worker and read its logs before any
+  `on` flip, the same rule as the other shadows.
 
 ## 2026-10-04 — wave 4 (S07, S13, S15/S17, S19, S21, S22, S24, S26, S27)
 
@@ -123,7 +133,8 @@ detail is in the programme's wave-2 status table. Items that matter for audit:
 - **S05 stack-detect (PR #17):** unwired; readiness still keys on project
   config only, not manifest, lockfile or recipe digest.
 - **S08 test-integrity (PRs #18, #31):** advisory-only on the run page; not a
-  gate; not in the PR body or webhook; false-positive rate unmeasured.
+  gate; PR-body/webhook surfacing landed in wave 5 (PR #58) behind a
+  default-off flag; false-positive rate unmeasured.
 - **S16 schedules (PR #24):** the overlap policy is inert in the worker until
   `sweepWorkflowSchedules` is given `isRunning`.
 - **S25 policy (PR #25):** tested pure core, no enforcement; every entry point

NEXT_SESSION.md

@@ -15,8 +15,8 @@ Wave 4's state ("implemented and checked by automated tests only") is now partly
 ## Next work, in order
 
 1. **Shadow-log review** after soak (placement/policy/budget/tool-manifest/knowledge-provenance JSONL + reports); set `SHIP_MODEL_ROUTING_POLICY` so the routing shadow has something to record. Only then discuss any `on` flips.
-2. **S08 wiring** (approved): findings into PR body + webhook behind a default-off flag; worker-path change, so shadow-first per the standing rule.
-3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S27 real teploy-adapter run against a scratch target; S19 doctor probes against the real store/clock; human observation of dashboard users. (S03's storage migration is done: migration 008 rehearsed on the restored `pre-80c9187` copy, receipt in `evals/receipts/2026-10-04-s03-requirements-migration.json`; the store stays unwired — a write path is separate future work.)
+2. **S08 wiring** (approved): BUILT in wave 5 (this PR) — findings in the PR body + run-completion webhook behind `SHIP_TEST_INTEGRITY_SURFACING=off|shadow|on`, default off. Still to do: soak `shadow` on the real worker, then decide the `on` flip.
+3. **Live proofs still open**: S01 credential proofs on a real sandbox with a private repo; S27 real teploy-adapter run against a scratch target; S19 doctor probes against the real store/clock; human observation of dashboard users. (S03's storage migration is done: migration 008 rehearsed on the restored `pre-80c9187` copy, receipt in `evals/receipts/2026-10-04-s03-requirements-migration.json`; the store stays unwired — a write path is separate future work. S07's plan-park wiring landed in #59.)
 4. **Code still unfinished**: S04 and S12 barely started; most of S07 beyond the grounding check; S10 follow-ups (offline/error states, other roles, screen readers); wiring the inert modules (S15/S17 into delivery+incidents, S19 snapshot producer + restore-check, S18 worker tree provisioning, S07 into the plan-park point).
 5. **Re-grade retained runs where possible**: only transcripts were preserved for pre-batch runs, so regrades are limited to transcript+fixture-verifiable scenarios; write supplementary `regrade-*.json` beside the records, never replacing originals.
 6. **Deferred**: configuration B of the S02 comparison (needs real API spend, owner-gated).

docs/SHIP_RELEASE_PROGRAMME_2026-09-21.md

@@ -505,7 +505,7 @@ This section supersedes the earlier table where rows overlap (S13, S28, S05, S06
 | S09 | #15 | `src/finding-continuity.ts` classifier (still-open, resolved, regressed, new, moved, superseded, unconfirmed-disappearance) | 38 tests, line-only negative controls, 4 mutations; synthetic fixtures | Not wired; thresholds unvalidated on real PRs; forge reads carry no inline comments or thread state |
 | S03 | #16 | Read-only "Task state" panel on the run page from `taskRecord()` | 8 view-model tests + negative controls; web 144; build; Chromium render at 390 and 1440 on seeded data | No real run observed; no storage or write path for criteria, evidence or waivers; runs list unchanged |
 | S05 | #17 | `src/stack-detect.ts` cited stack and service proposals, readiness digest | 32 tests + negative controls (two attempted controls did not fail; redundant by design) | Not wired; no real-repo corpus; regex TOML/YAML parsing; nothing proposed was executed |
-| S08 | #18, #31 | `src/test-integrity.ts` detector; advisory read-time panel in the run Verification view (`SHIP_ORACLE_PATHS` read in the web layer only) | 47 detector tests + 14 mutation controls; 9 projection tests + negative control (a missing diff never renders clean); web 153; build | Not a gate; not in the PR body, webhook or delivery record (would change the worker path, needs a decision); precision unmeasured; component not browser-rendered |
+| S08 | #18, #31 | `src/test-integrity.ts` detector; advisory read-time panel in the run Verification view (`SHIP_ORACLE_PATHS` read in the web layer only) | 47 detector tests + 14 mutation controls; 9 projection tests + negative control (a missing diff never renders clean); web 153; build | Not a gate; PR-body/webhook surfacing approved 2026-10-04 and landed in wave 5 (#58) behind `SHIP_TEST_INTEGRITY_SURFACING`; still not in the delivery record; precision unmeasured; component not browser-rendered |
 | S22 | #19 | `src/mission.ts` mission graph validation, readiness by accepted deliverable at an exact revision, acceptance aggregation | 24 tests + 8 mutation controls | Not wired or persisted; no launching, replanning or cancel; write-grant conflicts checked as strings only |
 | S20 | #20 | `docs/claims/ledger.json` -> `docs/CLAIM_LEDGER.md`, `scripts/claim-ledger.mjs --check` | 14 tests; 7 guards disabled to confirm | Not in CI; 12 seeded claims, none verified; tests are cited by name only |
 | S23 | #21 | `src/model-routing.ts` versioned routing and fallback policy with gates | 16 tests + 6 negative controls | Not wired; failure classification, policy store and the fixed-vs-routed comparison not built |
@@ -558,6 +558,14 @@ Upstream feature requests for the Teploy CLI, **not yet filed**: a deploy dry-ru
 
 Also merged this wave: #9 plan and inventory, #10 devalue 5.9.3 override. S10 and S02 grader sensitivity landed after this table was first written and are the first two rows after S13/S06.
 
+#### Wave 5 (2026-10-04): S08 surfacing
+
+Same rules as waves 3 and 4: off by default, default-off equivalence proven by tests. This row exists because the owner approved the direction on 2026-10-04 (see the wave-5 notes at the top of `AUDIT_OPEN.md`).
+
+| Package | PR | Flag (default) | Implemented | Checked | Still open (specific) |
+| --- | --- | --- | --- | --- | --- |
+| S08 | #58 | `SHIP_TEST_INTEGRITY_SURFACING=off\|shadow\|on` (off) | The detector's findings on the pull request body (a "Test integrity" part inside the existing Verification section, marker-spliced and idempotent like the rest) and as an additive `test_integrity` field on the run-completion webhook payload. No new step, no recorded step result changes, delivery record untouched; clean adds nothing anywhere. Shadow logs what `on` would surface, per the standing worker-path rule | 13 wiring tests: default-off byte-equivalence (section and payload), real-path flag-on tests through `verificationSection`/`spliceVerification`/`runWebhookPayload` and the worker's own terminal composition, negative controls (clean diff, no/`empty`/non-string diff, garbage diff, invalid flag value), cap and middle-cut honesty | Not one live run (no forge, worker or receiver reachable); shadow not soaked on a worker; detector precision still unmeasured (wave-2 open item); the `fix` command's own PR path (`fix-evidence.ts`) does not surface findings — worker paths only |
+
 ### Evidence that was not available
 
 - Private execution receipts (`COMPLETION_2026-09-25/`), the claim ledger and the competitive review are not in this repository or this environment, so the plan's statements that rest on them (local `42af7fb`, last live deployment `dd5b13e`, the 21/13/2 product baseline) were **not re-verified** here. The 21/13/2 figure was later recounted against the retained records under `evals/product-journeys/results/` (2026-10-03): it is not reproducible (28 attempts, not 36); the origin of 36/21/13 is in the private receipts, which remain unavailable.

src/durable.ts

@@ -63,6 +63,7 @@ import {
 import { buildIfDeclared, flowIfPresent, observeIfDeclared, recordLadder, smokeIfDeclared, visualIfDeclared, type AssetSink, type LadderHooks } from "./ladder-steps.js";
 import { compareAroundNow, effectiveTelemetryTarget, telemetryAppliesTo, telemetryRegression, type TelemetryTarget, type TelemetryVerdict } from "./observe.js";
 import { spliceVerification, verificationSection, type Evidence } from "./verification.js";
+import { surfacedTestIntegrity } from "./test-integrity-surface.js";
 import { shortHash, warmClient, warmSlugOf, type WarmState } from "./warm.js";
 import { mergeFact, verificationSummary, type VerificationFacts } from "./verification-summary.js";
 import { preExisting, runTests, testComment, testTargetFromInput, testsFailedNudge, type TestOutcome, type TestTarget } from "./tests.js";
@@ -2766,13 +2767,21 @@ async function publishIfRepoRun(
     if (followUpPreview !== undefined && input.supersedePreviews === true) await supersedeOlderPreviews(ctx, config, input, followUpPreview);
     const legs = await runLadderLegs(ctx, executor, config, input, followUpPreview, co.branch, { baseline, build, tests }, assetSink(ref, token, input.pr, config.artifacts));
     const followUpProof = proofLinks(legs);
+    // S08 surfacing: the detector over the diff this follow-up just published,
+    // rendered into the same Verification section as the rest of the evidence.
+    const followUpIntegrity = surfacedTestIntegrity({
+      diff: push.kind === "pushed" ? push.diff : undefined,
+      where: "the pull request body",
+      runId: ctx.runId,
+    });
     const followUp: Evidence = {
       ...(tests !== undefined ? { tests } : {}),
       ...(baseline !== undefined ? { testsBaseline: baseline } : {}),
       ...(followUpPreview !== undefined ? { preview: followUpPreview } : {}),
       ...(input.telemetry === true ? { telemetry: await telemetryIfAsked(ctx, config, input) } : {}),
       ...(legs.rungs !== undefined ? { rungs: legs.rungs } : {}),
       ...(followUpProof.length > 0 ? { proof: followUpProof } : {}),
+      ...(followUpIntegrity !== undefined ? { testIntegrity: followUpIntegrity } : {}),
     };
     await publishVerification(ctx, ref, token, input.pr, followUp);
     // NO auto-merge and NO rollback watch on this path, deliberately. A review
@@ -2857,6 +2866,11 @@ async function publishIfRepoRun(
   if (legs.flow !== undefined) facts.flow = legs.flow;
   if (legs.observe !== undefined) facts.observeWindow = legs.observe;
   const proof = proofLinks(legs);
+  // S08 surfacing: the detector over the diff this run published (the recorded
+  // repo-push diff — the same bytes the run page's panel analyses), rendered
+  // into the Verification section publishVerification amends onto the body.
+  // Nothing is recorded; off and shadow write nothing at all.
+  const integrity = surfacedTestIntegrity({ diff: push.diff, where: "the pull request body", runId: ctx.runId });
   await publishVerification(ctx, ref, token, pr.number, {
     ...(tests !== undefined ? { tests } : {}),
     ...(baseline !== undefined ? { testsBaseline: baseline } : {}),
@@ -2867,6 +2881,7 @@ async function publishIfRepoRun(
     // So a root-level suite over a change confined to one subtree is called
     // out on the pull request rather than read as a green gate (tests.ts).
     ...(changedList.length > 0 ? { changedPaths: changedList.map((f) => f.path) } : {}),
+    ...(integrity !== undefined ? { testIntegrity: integrity } : {}),
   });
   // Computed ONCE, outside any step, and fed to both gates below. It is a pure
   // function of the `telemetry-check` step's recorded verdict (see

src/notify.ts

@@ -1,6 +1,7 @@
 import type { ScanFinding } from "./findings.js";
 import type { ChangeClass } from "./change-class.js";
 import { rungsForWire, type Rung } from "./ladder.js";
+import type { TestIntegrityRecord } from "./test-integrity-surface.js";
 import { canonicalJson, EVENT_SCHEMA_VERSION } from "./tool-manifest.js";
 import type { EventEnvelope } from "./tool-manifest.js";
 import { createHash, createHmac } from "node:crypto";
@@ -77,6 +78,13 @@ export interface RunNotification {
   merged?: boolean;
   /** The agent's own result status on a terminal run (see RunWebhookPayload.outcome). */
   outcome?: string;
+  /**
+   * S08 surfacing (advisory, behind SHIP_TEST_INTEGRITY_SURFACING=on): the
+   * test-integrity detector's findings over the run's published diff. Absent
+   * when the flag is off or shadow, or when nothing was found — a consumer
+   * treats absent as "nothing surfaced", never as "the tests are sound".
+   */
+  testIntegrity?: TestIntegrityRecord;
 }
 
 /**
@@ -303,6 +311,21 @@ export interface RunWebhookPayload {
    * wire; this is what says it did not finish. Additive: absent on parks.
    */
   outcome?: string;
+  /**
+   * S08 surfacing (additive): present only when SHIP_TEST_INTEGRITY_SURFACING
+   * is on AND the detector found something in the published diff. Advisory —
+   * a receiver shows it, never gates on it; the same caveat the run page's
+   * panel carries (it flags patterns, it does not prove intent).
+   */
+  test_integrity?: {
+    verdict: "suspicious" | "tampered";
+    total: number;
+    omitted?: number;
+    findings: Array<{ kind: string; file: string; line: number; severity: string; confidence: string; evidence: string }>;
+    /** False when the published diff was recorded with a middle cut, so the read is partial. */
+    diff_complete: boolean;
+    oracle_configured: boolean;
+  };
 }
 
 export function runWebhookPayload(event: RunNotification, publicUrl?: string): RunWebhookPayload {
@@ -338,6 +361,18 @@ export function runWebhookPayload(event: RunNotification, publicUrl?: string): R
       ? { verification: { rungs: rungsForWire(event.verification.rungs), summary: event.verification.summary } }
       : {}),
     ...(event.merged !== undefined ? { merged: event.merged } : {}),
+    ...(event.testIntegrity !== undefined
+      ? {
+          test_integrity: {
+            verdict: event.testIntegrity.verdict,
+            total: event.testIntegrity.total,
+            ...(event.testIntegrity.omitted !== undefined ? { omitted: event.testIntegrity.omitted } : {}),
+            findings: event.testIntegrity.findings.map((f) => ({ ...f })),
+            diff_complete: event.testIntegrity.diffComplete,
+            oracle_configured: event.testIntegrity.oracleConfigured,
+          },
+        }
+      : {}),
   };
 }