docs: what the pack layer can and cannot do for a line edit, measured against kate bench

[kieranklaassen]

Companion to compound-packs PR #17, round 2 of the kate-bench optimize run (that PR carries the full report and learnings log). Stacked on cursor/compound-packs-7c77; docs only. Three plugin-side hypotheses were built, tested against cw-line-edit unchanged on the same rows with a paired per-opportunity ledger, and none passed the selection rule, so this branch carries the learnings and the arms stay on cursor/kate-bench-pack-integration-arms-cf0e with their tests (I3 811cfa1, fix 07bf43b, I6 e1bbcc2 + f55ba7f, I2 e9a8d77).

What was measured

Writer: this plugin's cw-line-edit at 2490123 reading the kate-bench-writing pack through packs-resolve.py, one subagent per batch of ten paragraphs, whole-document context. Scorer: kate bench's own KatePatch target logic on experimenter-safe rows. Dev = 60 of Kate Lee's edit opportunities stratified to her type mix plus 15 paragraphs she left alone; three never-inspected check slices; a document-disjoint holdout. Selection: net paired flips ≥ +2 on dev and ≥ 0 on the check slice, gates on KEEP abstention, volume, matched-hunk share, serious errors. Comparison arm (plugin unchanged, three round-1 pack rules): 0.400 / 0.417 action recall on dev, 0.294 / 0.353 on holdout.

arm	plugin change	dev	paired vs unchanged	check slice	kept
I2 (e9a8d77)	a matching pack rule may narrow the step's default checklist	0.367	−2 / −3	—	no
I3 (811cfa1, fix 07bf43b)	packs-check.py; packs may carry checks/ run before judgment	0.383	−1 / −2	−2	no
I6 (e1bbcc2, f55ba7f)	packs-retrieve.py; packs may carry examples/ retrieved per paragraph	0.400	0 / −1	0	no
D1 (diagnostic)	Kate's sites marked, fix withheld	0.533	+8, p = 0.021	—	—

Learnings this branch records

1. Rule text changes what the step does at a site, not which sites it finds: every arm raised precision (KEEP 0.47 → 0.73, edits at no editor site 77 → 51, matched share 0.28 → 0.36) and left recall within one of sixty.
2. Input beat instruction: the whole draft instead of three paragraphs was worth +5 of 60 on dev and +2–3 of 34 on holdout, more than any rule or integration change.
3. Deterministic checks have a headroom question first: nineteen house-style checks fire on 3 of 60 paragraphs Kate edited.
4. A check's replace is its test: a pattern that matched correct copy shipped and was caught by the measurement; packs-check.py on the arms branch drops no-op findings.
5. Lexical retrieval of the editor's before/after pairs shows the step moves it already makes; it did not reach the finding headroom.
6. Of 36 baseline misses: ~8 finding, ~16 a different action at a found site (rewording, swap, content), 12 untouched even when marked.

Files

• docs/learnings/2026-09-11-kate-bench-pack-integration.md — the note with the numbers and the arms' commits
• references/packs.md — new section "What a pack changes in an editing step, measured"
• skills/cw-line-edit/SKILL.md — two Lessons

Tests: python3 -m unittest discover -s tests -q passes on both branches (101 on this one; 108 on the arms branch).

Provenance

Built by a Cursor cloud agent for Kieran. The agent's git token has no write access to this repository, so the branches and this PR were created through Kieran's GitHub integration; commits are therefore attributed to that identity. Do not merge on the agent's behalf.