Motivation
Automated reviews (Opus-class models invoked headlessly, e.g. claude -p from another harness) produce long reviews whose text re-enters the calling context as input. We want reviews that are shorter and still catch what matters, and we want that claim measured rather than asserted.
The contract under test
Compress the per-finding format, never the finding count:
- Each finding is fixed-shape:
file:line, one-sentence claim, severity, confidence, one-sentence failure scenario.
- No diff restatement, no preamble, no praise, no closing summary.
- Never instruct severity filtering. Current Opus-class models follow "only report high-severity" literally and measured recall falls. Report everything, filter downstream.
The study
- Arms: baseline review prompt vs the fixed-shape contract, same PRs, same model.
- Metric: confirmed findings per thousand output tokens.
- Counterweight (Goodhart guard): recall — confirmed findings the compressed arm drops relative to baseline. A shorter review that loses real bugs fails.
- Disposition data already exists and is minable: review threads carry parseable verdict blocks (ACCEPTED / REJECTED_* / OBSOLETE), so "confirmed" needs no new collection.
Runs under the measurement design like any other study: arms carry manifests, figures derive from the scorer, promotion is a reviewed act. Until promoted, every figure is unaudited.
Motivation
Automated reviews (Opus-class models invoked headlessly, e.g.
claude -pfrom another harness) produce long reviews whose text re-enters the calling context as input. We want reviews that are shorter and still catch what matters, and we want that claim measured rather than asserted.The contract under test
Compress the per-finding format, never the finding count:
file:line, one-sentence claim, severity, confidence, one-sentence failure scenario.The study
Runs under the measurement design like any other study: arms carry manifests, figures derive from the scorer, promotion is a reviewed act. Until promoted, every figure is unaudited.