Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/hardening-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,5 @@ escape `\|`, one line), `source` (gate-a|gate-b|bot|manual),
| 2026-07-19 | unverified-enforcement-claim | across four Gate-B rounds on the index-tree documentation task, prose repeatedly claimed more than the hook guarantees, and each fix introduced a subtler version of the same overclaim — "the tree-hash proves what was reviewed", then "what is being committed is what was reviewed", then "what a commit would actually carry", then "everything a commit could carry"; every round searched for the previous PHRASE rather than the CLAIM, so a synonym survived each time | gate-b | major | 1 prose | AGENTS.md Don'ts, "Never describe what a gate proves without checking what it actually compares" — a rule plus a by-meaning grep recipe, sibling to the existing manifest-claims rule. Deliberately NOT another banned phrase: adding one phrase per round is the same rung a fifth time. What it does NOT do, in the rule text as well as here: nothing runs it in CI (a human runs it), and an overclaim phrased without those totality words escapes it. It raises the floor; it does not close the class |
| 2026-07-20 | verification-masks-failure | the plan's mutation-verification steps read `sh …codex-gate.test.sh 2>&1 \| grep -cE '^FAIL'`, which reports the GREP's status, not the runner's — a suite that died part-way, or exited non-zero without printing `FAIL`, would read as 0 and be recorded as green. Found by CodeRabbit on PR #8, in the very steps used to prove this branch's guards were load-bearing | bot | major | 1 prose | plan Task 1 Step 3 + the two later mutation steps now capture `exit=$?` separately into a variable and RETURN success only when the status and the FAIL count are both zero (per-run `mktemp` file, not a shared path), with the reason stated inline. An earlier revision of this row said they "assert both" when they only printed both and still returned grep's status — caught by Gate B, in the row recording this very class. Scope of the fix, plainly: it corrects the three sites in this plan and the rationale a future plan author reads — nothing checks new plans for the same shape, so a plan written tomorrow can reintroduce it |
| 2026-07-20 | truncated-tool-output-read-as-complete | Codex Gate A responses arrived cut off on long finding lists — on effectively every substantial pass in real project use — and a cut landing between findings is indistinguishable from a short list, so dropped findings read as a clean review; field agents were improvising a write-to-file workaround per session, which means the failure was handled only when someone happened to notice | manual | major | P std | CLAUDE.md §5 "Findings go to a FILE, not the response" + the same block in the workflow-init inline template: Codex writes the full list to `.context/codex-reviews/<slot>.md` ending `END OF FINDINGS (<n> total)`, replies with one line, and the reader accepts only on terminator-present AND count-matching AND nothing-but-finding-lines AND (for Gate-B `full`) both branch files. Gate B needs one file per branch because `reviewType: full` runs two reviewers in parallel off one additionalContext — a shared path lets the second overwrite the first and still pass every check. What it does NOT do: nothing checks the terminator mechanically (instruction-backed by design; a recurrence is the trigger to build the checker); the hook counts on `PostToolUse` and never sees the file, so an incomplete pass still increments the counter — including a FAILED review, because the pinned `mcp-codex-dev@1.0.1` catches its own errors, executor timeouts and aborts included, and returns `{success: false}` as a normal result rather than throwing, so Claude Code reads it as a successful tool call (an earlier draft of this row inferred from the documented `PostToolUseFailure` split that errors increment nothing — Gate B caught it by reading the server; the rule is now stated without reference to event routing: discount every incomplete pass whatever the counter says); and detection misses a model writing a wrong count with matching lines, or a stale file if the pre-call delete is skipped. Secondary mitigation only, in README: `MAX_MCP_OUTPUT_TOKENS` (25,000 default) moves the ceiling, does not remove it, and does not apply to tools declaring `anthropic/maxResultSizeChars`. Which documented mechanism (token limit vs persist-to-disk threshold) caused the field loss was NOT established; the protocol is correct under either. Fingerprint minted rather than filed under `verification-masks-failure`: that would make this a recurrence on a `1 prose` row, and the skill's recurrence rule then demands a mechanical rung or sharpened AGENTS.md wording, neither of which is P — three Gate-A passes each rejected a different attempt to narrate that gap, which is the class being bent to fit the ladder rather than the defect |
| 2026-07-25 | docs-drift | docs/prompt-standards.md said ad-hoc briefs are not reviewed against "all ten items" while its own checklist runs 1-12 and AGENTS.md invariant 11 independently says 12 — a hard-coded count contradicting the list it counts, in the same file | bot | minor | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence of this class, after 2026-07-18 `1 prose` (manifest-claims rule) and 2026-07-18 `P std` (spec-updated-with-fix rule); neither prior rung could reach a stale count, and the ladder's escalation from those is mechanical. The FIX landed in 4f31df2 (PR #12) — what is pending is the HARDENING, not the defect. Deferred by explicit decision: the checker is ~8 lines of shell plus tests, and writing it into a docs-only PR would fire full Gate B on new shell code; it rides the upcoming canvas-findings hardening round instead, which pays that cost anyway. Resolve by appending a rung-2 row referencing this one, never by editing it |
| 2026-07-25 | unverified-enforcement-claim | docs/sparring-briefing.md asserted "This is a prompt artifact and follows docs/prompt-standards.md" while giving its target model as "any capable chat model" — checklist item 1 requires a NAMED executing model plus the author having checked that model's prompting page, and a model class satisfies neither half (there is no single page to check), so the document contradicted its own conformance claim | bot | major | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence, after 2026-07-18 `P std` (prompt-standards item 11) and 2026-07-19 `1 prose` (the "never describe what a gate proves" rule, whose own ref states it is human-run, not CI-run, and "raises the floor; it does not close the class"). The class has now cycled P std → 1 prose → recurrence without ever reaching a deterministic rung, and this instance IS tool-decidable — a file claiming to follow prompt-standards must carry a `Target model:` line — which is why the escalation is mechanical rather than a fourth prose rule. The FIX landed in 4f31df2 (PR #12); the HARDENING is what is pending, deferred to the canvas-findings round on the same scope reasoning as the sibling row. Resolve by appending a rung-2 row referencing this one, never by editing it |
8 changes: 8 additions & 0 deletions docs/prompt-standards.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ commands (`plugins/dev-workflow/commands/`), the agent definitions
(`plugins/dev-workflow/hooks/codex-gate.sh`), and every template `/workflow-init`
writes are all prompt artifacts — they are the product, not documentation of it.

**Ad-hoc task briefs are prompts too.** A brief handed to the coding agent for a
single task — whether written by the human or by an upstream sparring chat
(`docs/sparring-briefing.md`) — steers the same model with the same failure modes,
and two field incidents came from briefs carrying unverified premises. Briefs are
held to this checklist in spirit (success criteria, stop conditions, verified
claims); nobody reviews them against all 12 items per brief, which is exactly why
the checklist's habits have to live in how they are written.

Comment thread
coderabbitai[bot] marked this conversation as resolved.
So this checklist is **this repo's own standard**, not only something it hands to
other projects: when authoring or changing any of the above, it must pass the
checklist below. `/workflow-init` scaffolds a copy of this file into each project it
Expand Down
89 changes: 89 additions & 0 deletions docs/sparring-briefing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Sparring briefing — the upstream advisor chat

A briefing for the AI chat that sits **upstream** of the coding workflow: the
sparring partner the human talks to before and between agent runs. It prepares
decisions, writes the prompts the coding agent executes, and validates agent
reports. This document exists because that layer's working knowledge otherwise
lives only in chat history — and chat history does not survive a new session,
a model change, or a different tool. Read time: five minutes; that is the point.

Target model: Claude in a chat interface, upstream of Claude Code as the coding
agent. This is a prompt artifact and follows `docs/prompt-standards.md`. The role
itself is not Claude-specific — another capable chat model can fill it — but the
wording here is written for Claude, per checklist item 1.

## The role, in one paragraph

You are strategist, decision-preparer, prompt author, and validator — **not** a
gate. You help the human decide (options with trade-offs, a recommendation, and
what it costs), you turn decisions into precise briefs for the coding agent, and
you check the agent's reports against the actual repository before advising on
them. You are the same model family as the coding agent, so you share its blind
spots: nothing you approve counts as a review pass, and your prompts enter the
workflow through the same gates as everything else. The chain has corrected this
role's authors repeatedly — treat that as the system working, not as an affront.

## Onboarding — read these four, in order

1. `AGENTS.md` — the invariants both gates check against, and § Commands.
2. `todos.md` — what is parked behind which trigger; do not resurrect parked
items without their trigger firing.
3. `docs/hardening-log.md` — what has already gone wrong here and at which rung
it was closed; recurrence of a class is a signal, not a coincidence.
4. `docs/getting-started.md` — the workflow's shape, if you are new to it.

## Working rules

- **Evidence before advice.** Claims about the repo are checked against the repo
— read the file, run the test, cite the line. If you cannot verify, say so and
mark the statement as unverified. Why: this project's most frequent defect
class is a plausible claim nobody checked, and the ledger proves it.
- **Advisory, never exempt.** Your prompts and designs go through the normal
workflow (intake → gates → PR). Do not design around the gates, and do not
treat a satisfied human as a substitute for a clean pass. Why: cross-model
independence is the core invariant, and you are not the other model.
- **One decision at a time, with its price.** When the human must choose,
present the options, name what each costs, recommend one, and mark the
recommendation as yours. Why: unpriced recommendations get followed, not
decided — and the human is the only decision-maker in this system.
- **Respect the parking discipline.** Reactive-only means work starts from a
finding, a trigger, or an explicit pull by the human — not from "while we're
at it". Why: this repo measurably drifts into self-referential polishing
without that rule.

## Prompt conventions — briefs handed to the coding agent

Every task brief you emit follows `docs/prompt-standards.md`. In practice that
means, learned from field use:

- **Findings numbered, each with severity and the evidence** that grounds it.
- **"Done when" as checkable outcomes** (commands that exit 0, files that
exist, greps that return nothing) — never "improve X".
- **A "Stop and ask" block** naming the conditions under which the agent
surfaces instead of guessing — especially where a design decision hides
inside an implementation task.
- **Scope guards stated** ("prompt/template changes only; if the design pulls
toward machinery, stop") — the cheapest way to prevent a small fix from
growing a second feature.
- **Branch → PR is implied** (branch protection enforces it), but say it when
the routing matters (separate branch, don't stack on X).
Comment on lines +68 to +69

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Branch routing is not enforced

When branch protection has not been manually configured, this guidance tells the advisor that branch-to-PR routing is implied even though direct pushes remain available, causing work to bypass PR review and checks that run only for pull_request events.

Prompt To Fix With AI
This is a comment left during a code review.
Path: docs/sparring-briefing.md
Line: 67-68

Comment:
**Branch routing is not enforced**

When branch protection has not been manually configured, this guidance tells the advisor that branch-to-PR routing is implied even though direct pushes remain available, causing work to bypass PR review and checks that run only for `pull_request` events.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dismissed — the factual premise is wrong for this repository.

The claim is that branch protection "has not been manually configured", so direct pushes remain available and can bypass pull_request-only checks. Verified against the GitHub rulesets API rather than the repo's own prose:

name: protect-main.   enforcement: active   target: branch
conditions.include: ["~DEFAULT_BRANCH"]
rules: non_fast_forward, pull_request, required_status_checks
bypass_actors: []   # empty — admins included

So main accepts changes only through a PR with a green quality check, with no bypass actors. That is exactly the premise docs/sparring-briefing.md:67 relies on, and the ruleset was added deliberately to close this gap (README.md:132-139; commit e52d9b6, "the version-bump PR-only gap is closed by the main branch ruleset", #10).

The narrower true statement nearby: .github/workflows/ci.yml and AGENTS.md invariant 12 record that the version-bump checker is pull_request-only, so a direct push would bypass that checker. The ruleset is what answers it — direct pushes to main are not available in the first place.

One limitation worth stating, since it cut the other way here: a ruleset lives in repo settings, not in the tree, so nothing readable in the checkout re-verifies it. That is why this was confirmed against the API.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and the correction is well-documented. The ruleset (protect-main, enforcement active, pull_request + required_status_checks, no bypass actors) directly contradicts the premise of my comment. I was wrong to flag this without verifying the actual branch protection configuration.

The narrower point you surface — that the version-bump checker is pull_request-only and a direct push would bypass it — is answered by the same ruleset making direct pushes to main unavailable entirely. The guidance at line 67 stands.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

- **Verify-before-claiming carries into the brief:** if your prompt asserts a
fact about the repo or an external system, either you verified it or the
brief instructs the agent to verify before building on it. Why: two field
incidents came from a sparring prompt carrying an unverified premise.

## Handoff expectations — what comes back

Agent reports are expected to ground every claim in a tool result (CLAUDE.md
§4). When a report reaches you: validate the load-bearing claims against the
repo before advising the human on them, and prefer "I checked X, it holds"
over "the report says". Decisions the human makes on your recommendation must
end up in the repo (spec decision records, todos triggers, ledger rows) — a
decision that lives only in this chat does not exist.

## What this document is not

Not a plugin feature, not scaffolded by `/workflow-init`, and not a template —
it is one project's hand-written instance. If the pattern proves itself across
several projects, promoting it to a scaffolded template is a todos entry with a
trigger, not a reflex.
29 changes: 28 additions & 1 deletion todos.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,34 @@ backlog.
## Tooling revalidation
- [ ] Re-check `docs/prompt-standards.md` against the current model-specific
prompting pages on every model-generation change (new Claude model in Claude
Code, new Codex model for the gates).
Code, new Codex model for the gates). Include `docs/sparring-briefing.md` in
that pass — it is a prompt artifact for the upstream advisor chat, and a
model change on either side of it (sparring model or coding agent) can shift
what its conventions should say. Concretely pending: the switch of the
coding agent to the new Opus generation fires this row.
- [ ] When `commands/workflow-init.md` is next touched for other reasons, add the
"ad-hoc task briefs are prompts too" paragraph (docs/prompt-standards.md,
repo copy) to the scaffolded prompt-standards template as well — one
sentence of drift between repo copy and template, recorded here instead of
forcing a plugin release for it (same reasoning as the PR #10 scope
decision). **Resolution vehicle: the upcoming canvas-findings hardening
round**, which touches `commands/workflow-init.md` for template changes and
carries a version bump regardless — the one-sentence sync rides there at no
extra release cost. Re-raised by CodeRabbit on PR #12 (Major) and kept
deferred there on the same reasoning; the scaffolded copy carries no false
claim, only one paragraph less.
- [ ] **Prompt-standards conformance checker — resolves two `pending` ledger rows
(2026-07-25).** Extend `scripts/check-invariants.sh` (+ its regression suite,
mutation-verified) with the mechanical rung both classes have never reached:
every file asserting it follows `docs/prompt-standards.md` must carry a
`Target model:` line, and a prose count of the checklist must match the
number of items actually in it. **Resolution vehicle: the same upcoming
canvas-findings hardening round** — it fires full Gate B anyway on its
`commands/` paths, so the ~8-line checker plus tests land at proportionate
cost instead of turning a docs PR into a code PR. Resolve the two `pending`
rows by **appending** rung-2 rows referencing them (2026-07-25 `docs-drift`
and 2026-07-25 `unverified-enforcement-claim`) — never by editing the
pending rows. *Trigger: that round starting.*
- [ ] **Escalation trigger for the invariant checker — read this before patching it.**
The checker asserts only the spellings its fixtures cover. Adding one more regex
arm per newly-discovered spelling is *not* the ladder working; it is the same
Expand Down