diff --git a/docs/hardening-log.md b/docs/hardening-log.md index 058a469..83abce8 100644 --- a/docs/hardening-log.md +++ b/docs/hardening-log.md @@ -24,3 +24,5 @@ escape `\|`, one line), `source` (gate-a|gate-b|bot|manual), | 2026-07-19 | unverified-enforcement-claim | across four Gate-B rounds on the index-tree documentation task, prose repeatedly claimed more than the hook guarantees, and each fix introduced a subtler version of the same overclaim — "the tree-hash proves what was reviewed", then "what is being committed is what was reviewed", then "what a commit would actually carry", then "everything a commit could carry"; every round searched for the previous PHRASE rather than the CLAIM, so a synonym survived each time | gate-b | major | 1 prose | AGENTS.md Don'ts, "Never describe what a gate proves without checking what it actually compares" — a rule plus a by-meaning grep recipe, sibling to the existing manifest-claims rule. Deliberately NOT another banned phrase: adding one phrase per round is the same rung a fifth time. What it does NOT do, in the rule text as well as here: nothing runs it in CI (a human runs it), and an overclaim phrased without those totality words escapes it. It raises the floor; it does not close the class | | 2026-07-20 | verification-masks-failure | the plan's mutation-verification steps read `sh …codex-gate.test.sh 2>&1 \| grep -cE '^FAIL'`, which reports the GREP's status, not the runner's — a suite that died part-way, or exited non-zero without printing `FAIL`, would read as 0 and be recorded as green. Found by CodeRabbit on PR #8, in the very steps used to prove this branch's guards were load-bearing | bot | major | 1 prose | plan Task 1 Step 3 + the two later mutation steps now capture `exit=$?` separately into a variable and RETURN success only when the status and the FAIL count are both zero (per-run `mktemp` file, not a shared path), with the reason stated inline. An earlier revision of this row said they "assert both" when they only printed both and still returned grep's status — caught by Gate B, in the row recording this very class. Scope of the fix, plainly: it corrects the three sites in this plan and the rationale a future plan author reads — nothing checks new plans for the same shape, so a plan written tomorrow can reintroduce it | | 2026-07-20 | truncated-tool-output-read-as-complete | Codex Gate A responses arrived cut off on long finding lists — on effectively every substantial pass in real project use — and a cut landing between findings is indistinguishable from a short list, so dropped findings read as a clean review; field agents were improvising a write-to-file workaround per session, which means the failure was handled only when someone happened to notice | manual | major | P std | CLAUDE.md §5 "Findings go to a FILE, not the response" + the same block in the workflow-init inline template: Codex writes the full list to `.context/codex-reviews/.md` ending `END OF FINDINGS ( total)`, replies with one line, and the reader accepts only on terminator-present AND count-matching AND nothing-but-finding-lines AND (for Gate-B `full`) both branch files. Gate B needs one file per branch because `reviewType: full` runs two reviewers in parallel off one additionalContext — a shared path lets the second overwrite the first and still pass every check. What it does NOT do: nothing checks the terminator mechanically (instruction-backed by design; a recurrence is the trigger to build the checker); the hook counts on `PostToolUse` and never sees the file, so an incomplete pass still increments the counter — including a FAILED review, because the pinned `mcp-codex-dev@1.0.1` catches its own errors, executor timeouts and aborts included, and returns `{success: false}` as a normal result rather than throwing, so Claude Code reads it as a successful tool call (an earlier draft of this row inferred from the documented `PostToolUseFailure` split that errors increment nothing — Gate B caught it by reading the server; the rule is now stated without reference to event routing: discount every incomplete pass whatever the counter says); and detection misses a model writing a wrong count with matching lines, or a stale file if the pre-call delete is skipped. Secondary mitigation only, in README: `MAX_MCP_OUTPUT_TOKENS` (25,000 default) moves the ceiling, does not remove it, and does not apply to tools declaring `anthropic/maxResultSizeChars`. Which documented mechanism (token limit vs persist-to-disk threshold) caused the field loss was NOT established; the protocol is correct under either. Fingerprint minted rather than filed under `verification-masks-failure`: that would make this a recurrence on a `1 prose` row, and the skill's recurrence rule then demands a mechanical rung or sharpened AGENTS.md wording, neither of which is P — three Gate-A passes each rejected a different attempt to narrate that gap, which is the class being bent to fit the ladder rather than the defect | +| 2026-07-25 | docs-drift | docs/prompt-standards.md said ad-hoc briefs are not reviewed against "all ten items" while its own checklist runs 1-12 and AGENTS.md invariant 11 independently says 12 — a hard-coded count contradicting the list it counts, in the same file | bot | minor | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence of this class, after 2026-07-18 `1 prose` (manifest-claims rule) and 2026-07-18 `P std` (spec-updated-with-fix rule); neither prior rung could reach a stale count, and the ladder's escalation from those is mechanical. The FIX landed in 4f31df2 (PR #12) — what is pending is the HARDENING, not the defect. Deferred by explicit decision: the checker is ~8 lines of shell plus tests, and writing it into a docs-only PR would fire full Gate B on new shell code; it rides the upcoming canvas-findings hardening round instead, which pays that cost anyway. Resolve by appending a rung-2 row referencing this one, never by editing it | +| 2026-07-25 | unverified-enforcement-claim | docs/sparring-briefing.md asserted "This is a prompt artifact and follows docs/prompt-standards.md" while giving its target model as "any capable chat model" — checklist item 1 requires a NAMED executing model plus the author having checked that model's prompting page, and a model class satisfies neither half (there is no single page to check), so the document contradicted its own conformance claim | bot | major | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence, after 2026-07-18 `P std` (prompt-standards item 11) and 2026-07-19 `1 prose` (the "never describe what a gate proves" rule, whose own ref states it is human-run, not CI-run, and "raises the floor; it does not close the class"). The class has now cycled P std → 1 prose → recurrence without ever reaching a deterministic rung, and this instance IS tool-decidable — a file claiming to follow prompt-standards must carry a `Target model:` line — which is why the escalation is mechanical rather than a fourth prose rule. The FIX landed in 4f31df2 (PR #12); the HARDENING is what is pending, deferred to the canvas-findings round on the same scope reasoning as the sibling row. Resolve by appending a rung-2 row referencing this one, never by editing it | diff --git a/docs/prompt-standards.md b/docs/prompt-standards.md index 1c136b7..020771a 100644 --- a/docs/prompt-standards.md +++ b/docs/prompt-standards.md @@ -6,6 +6,14 @@ commands (`plugins/dev-workflow/commands/`), the agent definitions (`plugins/dev-workflow/hooks/codex-gate.sh`), and every template `/workflow-init` writes are all prompt artifacts — they are the product, not documentation of it. +**Ad-hoc task briefs are prompts too.** A brief handed to the coding agent for a +single task — whether written by the human or by an upstream sparring chat +(`docs/sparring-briefing.md`) — steers the same model with the same failure modes, +and two field incidents came from briefs carrying unverified premises. Briefs are +held to this checklist in spirit (success criteria, stop conditions, verified +claims); nobody reviews them against all 12 items per brief, which is exactly why +the checklist's habits have to live in how they are written. + So this checklist is **this repo's own standard**, not only something it hands to other projects: when authoring or changing any of the above, it must pass the checklist below. `/workflow-init` scaffolds a copy of this file into each project it diff --git a/docs/sparring-briefing.md b/docs/sparring-briefing.md new file mode 100644 index 0000000..4a13190 --- /dev/null +++ b/docs/sparring-briefing.md @@ -0,0 +1,89 @@ +# Sparring briefing — the upstream advisor chat + +A briefing for the AI chat that sits **upstream** of the coding workflow: the +sparring partner the human talks to before and between agent runs. It prepares +decisions, writes the prompts the coding agent executes, and validates agent +reports. This document exists because that layer's working knowledge otherwise +lives only in chat history — and chat history does not survive a new session, +a model change, or a different tool. Read time: five minutes; that is the point. + +Target model: Claude in a chat interface, upstream of Claude Code as the coding +agent. This is a prompt artifact and follows `docs/prompt-standards.md`. The role +itself is not Claude-specific — another capable chat model can fill it — but the +wording here is written for Claude, per checklist item 1. + +## The role, in one paragraph + +You are strategist, decision-preparer, prompt author, and validator — **not** a +gate. You help the human decide (options with trade-offs, a recommendation, and +what it costs), you turn decisions into precise briefs for the coding agent, and +you check the agent's reports against the actual repository before advising on +them. You are the same model family as the coding agent, so you share its blind +spots: nothing you approve counts as a review pass, and your prompts enter the +workflow through the same gates as everything else. The chain has corrected this +role's authors repeatedly — treat that as the system working, not as an affront. + +## Onboarding — read these four, in order + +1. `AGENTS.md` — the invariants both gates check against, and § Commands. +2. `todos.md` — what is parked behind which trigger; do not resurrect parked + items without their trigger firing. +3. `docs/hardening-log.md` — what has already gone wrong here and at which rung + it was closed; recurrence of a class is a signal, not a coincidence. +4. `docs/getting-started.md` — the workflow's shape, if you are new to it. + +## Working rules + +- **Evidence before advice.** Claims about the repo are checked against the repo + — read the file, run the test, cite the line. If you cannot verify, say so and + mark the statement as unverified. Why: this project's most frequent defect + class is a plausible claim nobody checked, and the ledger proves it. +- **Advisory, never exempt.** Your prompts and designs go through the normal + workflow (intake → gates → PR). Do not design around the gates, and do not + treat a satisfied human as a substitute for a clean pass. Why: cross-model + independence is the core invariant, and you are not the other model. +- **One decision at a time, with its price.** When the human must choose, + present the options, name what each costs, recommend one, and mark the + recommendation as yours. Why: unpriced recommendations get followed, not + decided — and the human is the only decision-maker in this system. +- **Respect the parking discipline.** Reactive-only means work starts from a + finding, a trigger, or an explicit pull by the human — not from "while we're + at it". Why: this repo measurably drifts into self-referential polishing + without that rule. + +## Prompt conventions — briefs handed to the coding agent + +Every task brief you emit follows `docs/prompt-standards.md`. In practice that +means, learned from field use: + +- **Findings numbered, each with severity and the evidence** that grounds it. +- **"Done when" as checkable outcomes** (commands that exit 0, files that + exist, greps that return nothing) — never "improve X". +- **A "Stop and ask" block** naming the conditions under which the agent + surfaces instead of guessing — especially where a design decision hides + inside an implementation task. +- **Scope guards stated** ("prompt/template changes only; if the design pulls + toward machinery, stop") — the cheapest way to prevent a small fix from + growing a second feature. +- **Branch → PR is implied** (branch protection enforces it), but say it when + the routing matters (separate branch, don't stack on X). +- **Verify-before-claiming carries into the brief:** if your prompt asserts a + fact about the repo or an external system, either you verified it or the + brief instructs the agent to verify before building on it. Why: two field + incidents came from a sparring prompt carrying an unverified premise. + +## Handoff expectations — what comes back + +Agent reports are expected to ground every claim in a tool result (CLAUDE.md +§4). When a report reaches you: validate the load-bearing claims against the +repo before advising the human on them, and prefer "I checked X, it holds" +over "the report says". Decisions the human makes on your recommendation must +end up in the repo (spec decision records, todos triggers, ledger rows) — a +decision that lives only in this chat does not exist. + +## What this document is not + +Not a plugin feature, not scaffolded by `/workflow-init`, and not a template — +it is one project's hand-written instance. If the pattern proves itself across +several projects, promoting it to a scaffolded template is a todos entry with a +trigger, not a reflex. diff --git a/todos.md b/todos.md index c9335cb..ab7a8cd 100644 --- a/todos.md +++ b/todos.md @@ -177,7 +177,34 @@ backlog. ## Tooling revalidation - [ ] Re-check `docs/prompt-standards.md` against the current model-specific prompting pages on every model-generation change (new Claude model in Claude - Code, new Codex model for the gates). + Code, new Codex model for the gates). Include `docs/sparring-briefing.md` in + that pass — it is a prompt artifact for the upstream advisor chat, and a + model change on either side of it (sparring model or coding agent) can shift + what its conventions should say. Concretely pending: the switch of the + coding agent to the new Opus generation fires this row. +- [ ] When `commands/workflow-init.md` is next touched for other reasons, add the + "ad-hoc task briefs are prompts too" paragraph (docs/prompt-standards.md, + repo copy) to the scaffolded prompt-standards template as well — one + sentence of drift between repo copy and template, recorded here instead of + forcing a plugin release for it (same reasoning as the PR #10 scope + decision). **Resolution vehicle: the upcoming canvas-findings hardening + round**, which touches `commands/workflow-init.md` for template changes and + carries a version bump regardless — the one-sentence sync rides there at no + extra release cost. Re-raised by CodeRabbit on PR #12 (Major) and kept + deferred there on the same reasoning; the scaffolded copy carries no false + claim, only one paragraph less. +- [ ] **Prompt-standards conformance checker — resolves two `pending` ledger rows + (2026-07-25).** Extend `scripts/check-invariants.sh` (+ its regression suite, + mutation-verified) with the mechanical rung both classes have never reached: + every file asserting it follows `docs/prompt-standards.md` must carry a + `Target model:` line, and a prose count of the checklist must match the + number of items actually in it. **Resolution vehicle: the same upcoming + canvas-findings hardening round** — it fires full Gate B anyway on its + `commands/` paths, so the ~8-line checker plus tests land at proportionate + cost instead of turning a docs PR into a code PR. Resolve the two `pending` + rows by **appending** rung-2 rows referencing them (2026-07-25 `docs-drift` + and 2026-07-25 `unverified-enforcement-claim`) — never by editing the + pending rows. *Trigger: that round starting.* - [ ] **Escalation trigger for the invariant checker — read this before patching it.** The checker asserts only the spellings its fixtures cover. Adding one more regex arm per newly-discovered spelling is *not* the ladder working; it is the same