From eed589cea2bf99f4588e7302ec22845888b5fbd8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sun, 26 Jul 2026 10:51:32 +0200 Subject: [PATCH 1/2] Give two recurring prompt-conformance classes their mechanical rung MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both classes reached a third occurrence and sat at rung `pending` in the ledger, naming this work as their resolution vehicle. Two checks in check-invariants.sh now guard one spelling each, and the appended rung-2 rows say exactly which. Check 4a — a file asserting it follows docs/prompt-standards.md must carry exactly one column-zero `Target model:` line whose value BEGINS with a recognized token and names exactly one distinct model. Both halves are load bearing: presence alone accepts "any capable chat model", and a token found anywhere accepts it too, because the real PR #12 defect line mentioned Claude as provenance. Measured: token-anywhere matched it, token-at-start did not. Check 4b — a prose count claim must equal the checklist it counts. Word forms one..twenty are in scope BECAUSE the motivating occurrence was one; a digit-only check would have sailed past the very defect the row claims to harden. Claims are recognized in two stages so a malformed `all 012 items` fires rather than being invisible. The ledger is excluded from both checks. A ledger that quotes defects self-rejects the checks that detect them: docs/hardening-log.md carries the historical count claim as evidence, and scanning it would fail the repository forever on rows that exist to record the fix. Also here: one shared fixture initializer, without which 25 of the 61 pre-existing assertions failed on a missing checklist before reaching their own assertion; the docs sweep across seven locations in five files that describe this checker; and AGENTS.md invariant 11 narrowed, since "nothing mechanical checks them" stopped being true. Suite 61 -> 123 assertions. Gate B: 8 passes, 13 findings, final spec and quality passes both clean against an unchanged tree. No plugins/** path is touched, so invariant 12 requires no version bump; the §5 nudge, the companion files and the template sync land in PR 2 with the bump. --- .github/workflows/ci.yml | 5 +- AGENTS.md | 12 +- README.md | 2 +- docs/architecture.md | 2 +- docs/hardening-log.md | 2 + .../2026-07-26-prompt-conformance-checks.md | 537 ++++++++++++++++++ scripts/check-invariants.sh | 233 +++++++- scripts/check-invariants.test.sh | 334 ++++++++++- todos.md | 29 +- 9 files changed, 1134 insertions(+), 22 deletions(-) create mode 100644 docs/superpowers/plans/2026-07-26-prompt-conformance-checks.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8f17a2b..aaaeb86 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,14 +73,15 @@ jobs: - name: Hook state-machine tests run: sh plugins/dev-workflow/hooks/codex-gate.test.sh - # Invariants 5 and 6 mechanically, plus BOTH checkers' regression suites. The + # Invariants 5 and 6 plus two prompt-conformance checks, mechanically, and BOTH + # checkers' regression suites. The # invariant-12 checker itself is not here — it needs a PR base and runs in the # step below, so naming this step "version bump" would show a green version-bump # label on a push that was never version-bump checked. # Each rule was prose first and each was violated anyway — a floating action ref # shipped in this very workflow, a duplicate hooks manifest key stopped the plugin # loading in 0.2.1, and the plugin shipped changes without a bump twice. - - name: Invariant checks (pinning, manifest) + both checker suites + - name: Invariant checks (pinning, manifest, prompt conformance) + both checker suites run: | sh scripts/check-invariants.test.sh sh scripts/check-invariants.sh diff --git a/AGENTS.md b/AGENTS.md index e585554..b3bcac4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,7 @@ todos.md # backlog; `pending` ledger rows point here by .mcp.json # the Codex reviewer, pinned .claude-plugin/marketplace.json .github/workflows/ci.yml # lint + hook tests + invariant checks + validate -scripts/check-invariants.sh # invariants 5 and 6, mechanically (rung 2) +scripts/check-invariants.sh # invariants 5, 6 + prompt conformance (rung 2) scripts/check-invariants.test.sh # its regression suite — reject/accept pairs scripts/check-version-bump.sh # invariant 12, mechanically — PR-only (rung 2) scripts/check-version-bump.test.sh # its regression suite — policy/operational/accept @@ -167,8 +167,12 @@ reader can judge whether it still holds. `docs/hardening-taxonomy.md`, never into the `harden-finding` skill. Otherwise one project leaks into every other. 11. **Prompt changes pass `docs/prompt-standards.md`** — all 12 checklist items, for - any skill, command, agent definition, hook message, or scaffolded template. The prompts are the - product and nothing mechanical checks them. + any skill, command, agent definition, hook message, or scaffolded template. The + prompts are the product, and **no comprehensive mechanical checker exists for them**: + review is the gate. Two narrow checks in `scripts/check-invariants.sh` cover one + spelling each — a `Target model:` line naming exactly one recognized model in files + claiming conformance, and a prose checklist-count claim matching the checklist — and + they are a floor, not coverage. Every other item is judged by a reader. ## Don'ts @@ -231,7 +235,7 @@ Every command below was run in this session and observed to exit 0. | typecheck | n/a — no typed sources (shell + markdown) | | lint | `shellcheck --shell=sh plugins/dev-workflow/hooks/codex-gate.sh && shellcheck --shell=sh --exclude=SC2015 plugins/dev-workflow/hooks/codex-gate.test.sh && shellcheck --shell=sh scripts/check-invariants.sh && shellcheck --shell=sh --exclude=SC2015 scripts/check-invariants.test.sh && shellcheck --shell=sh scripts/check-version-bump.sh && shellcheck --shell=sh scripts/check-version-bump.test.sh` | | test | `sh plugins/dev-workflow/hooks/codex-gate.test.sh` | -| invariant checks (5 pinning, 6 manifest) | `sh scripts/check-invariants.test.sh && sh scripts/check-invariants.sh` | +| invariant checks (5 pinning, 6 manifest, prompt conformance) | `sh scripts/check-invariants.test.sh && sh scripts/check-invariants.sh` | | invariant check (12 version bump) | `sh scripts/check-version-bump.test.sh && sh scripts/check-version-bump.sh main` | | build | n/a — nothing is compiled or bundled | diff --git a/README.md b/README.md index 5ec6a52..e721142 100644 --- a/README.md +++ b/README.md @@ -123,7 +123,7 @@ honest gap ([reasoning](docs/coding-workflow.md#adapting-it-to-another-project)) CI ([`.github/workflows/ci.yml`](.github/workflows/ci.yml)) runs four checks on every PR and push to main: `shellcheck --shell=sh` over all three executables and their test files, the hook's test suite, -[`scripts/check-invariants.sh`](scripts/check-invariants.sh) (invariants 5 and 6) plus +[`scripts/check-invariants.sh`](scripts/check-invariants.sh) (invariants 5 and 6, plus two prompt-conformance checks) plus both checkers' regression suites, and `claude plugin validate . --strict`. A fifth check runs **on pull requests only**: diff --git a/docs/architecture.md b/docs/architecture.md index 6757337..4e9a8c5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -15,7 +15,7 @@ README.md, MANIFEST.md, AGENTS.md, CLAUDE.md, todos.md .mcp.json # the Codex reviewer, pinned .claude-plugin/marketplace.json .github/workflows/ci.yml # lint + hook tests + invariant checks + validate -scripts/check-invariants.sh # invariants 5 and 6, mechanically (+ .test.sh) +scripts/check-invariants.sh # invariants 5, 6 + prompt conformance (+ .test.sh) scripts/check-version-bump.sh # invariant 12, PR-only, mechanically (+ .test.sh) plugins/dev-workflow/ .claude-plugin/plugin.json diff --git a/docs/hardening-log.md b/docs/hardening-log.md index 83abce8..1c57175 100644 --- a/docs/hardening-log.md +++ b/docs/hardening-log.md @@ -26,3 +26,5 @@ escape `\|`, one line), `source` (gate-a|gate-b|bot|manual), | 2026-07-20 | truncated-tool-output-read-as-complete | Codex Gate A responses arrived cut off on long finding lists — on effectively every substantial pass in real project use — and a cut landing between findings is indistinguishable from a short list, so dropped findings read as a clean review; field agents were improvising a write-to-file workaround per session, which means the failure was handled only when someone happened to notice | manual | major | P std | CLAUDE.md §5 "Findings go to a FILE, not the response" + the same block in the workflow-init inline template: Codex writes the full list to `.context/codex-reviews/.md` ending `END OF FINDINGS ( total)`, replies with one line, and the reader accepts only on terminator-present AND count-matching AND nothing-but-finding-lines AND (for Gate-B `full`) both branch files. Gate B needs one file per branch because `reviewType: full` runs two reviewers in parallel off one additionalContext — a shared path lets the second overwrite the first and still pass every check. What it does NOT do: nothing checks the terminator mechanically (instruction-backed by design; a recurrence is the trigger to build the checker); the hook counts on `PostToolUse` and never sees the file, so an incomplete pass still increments the counter — including a FAILED review, because the pinned `mcp-codex-dev@1.0.1` catches its own errors, executor timeouts and aborts included, and returns `{success: false}` as a normal result rather than throwing, so Claude Code reads it as a successful tool call (an earlier draft of this row inferred from the documented `PostToolUseFailure` split that errors increment nothing — Gate B caught it by reading the server; the rule is now stated without reference to event routing: discount every incomplete pass whatever the counter says); and detection misses a model writing a wrong count with matching lines, or a stale file if the pre-call delete is skipped. Secondary mitigation only, in README: `MAX_MCP_OUTPUT_TOKENS` (25,000 default) moves the ceiling, does not remove it, and does not apply to tools declaring `anthropic/maxResultSizeChars`. Which documented mechanism (token limit vs persist-to-disk threshold) caused the field loss was NOT established; the protocol is correct under either. Fingerprint minted rather than filed under `verification-masks-failure`: that would make this a recurrence on a `1 prose` row, and the skill's recurrence rule then demands a mechanical rung or sharpened AGENTS.md wording, neither of which is P — three Gate-A passes each rejected a different attempt to narrate that gap, which is the class being bent to fit the ladder rather than the defect | | 2026-07-25 | docs-drift | docs/prompt-standards.md said ad-hoc briefs are not reviewed against "all ten items" while its own checklist runs 1-12 and AGENTS.md invariant 11 independently says 12 — a hard-coded count contradicting the list it counts, in the same file | bot | minor | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence of this class, after 2026-07-18 `1 prose` (manifest-claims rule) and 2026-07-18 `P std` (spec-updated-with-fix rule); neither prior rung could reach a stale count, and the ladder's escalation from those is mechanical. The FIX landed in 4f31df2 (PR #12) — what is pending is the HARDENING, not the defect. Deferred by explicit decision: the checker is ~8 lines of shell plus tests, and writing it into a docs-only PR would fire full Gate B on new shell code; it rides the upcoming canvas-findings hardening round instead, which pays that cost anyway. Resolve by appending a rung-2 row referencing this one, never by editing it | | 2026-07-25 | unverified-enforcement-claim | docs/sparring-briefing.md asserted "This is a prompt artifact and follows docs/prompt-standards.md" while giving its target model as "any capable chat model" — checklist item 1 requires a NAMED executing model plus the author having checked that model's prompting page, and a model class satisfies neither half (there is no single page to check), so the document contradicted its own conformance claim | bot | major | pending | todos.md § Tooling revalidation, "Prompt-standards conformance checker — resolves two `pending` ledger rows (2026-07-25)". THIRD occurrence, after 2026-07-18 `P std` (prompt-standards item 11) and 2026-07-19 `1 prose` (the "never describe what a gate proves" rule, whose own ref states it is human-run, not CI-run, and "raises the floor; it does not close the class"). The class has now cycled P std → 1 prose → recurrence without ever reaching a deterministic rung, and this instance IS tool-decidable — a file claiming to follow prompt-standards must carry a `Target model:` line — which is why the escalation is mechanical rather than a fourth prose rule. The FIX landed in 4f31df2 (PR #12); the HARDENING is what is pending, deferred to the canvas-findings round on the same scope reasoning as the sibling row. Resolve by appending a rung-2 row referencing this one, never by editing it | +| 2026-07-26 | unverified-enforcement-claim | third occurrence, resolved: a prompt asserting it follows docs/prompt-standards.md while naming no executing model — the PR #12 sparring-briefing said `Target model: any capable chat model` | bot | major | 2 lint | scripts/check-invariants.sh check 4a + fixtures in .test.sh. **Resolves the 2026-07-25 `pending` row** (same fingerprint, appended not edited). GUARDED SPELLING, exactly: in a `*.md` file containing the literal `prompt artifact and follows`, the column-zero `^Target model:` line must appear exactly once, its value must BEGIN with a tested token (`Claude\|Codex\|GPT`, portable ERE boundary — token-anywhere is not enough, since the real defect line contained "Claude" as provenance), and it must name exactly one DISTINCT token. STILL INSTRUCTION-BACKED: a bare `Target model: Claude` naming no execution surface passes; a value whose prose is wrong in a way no token test sees passes; a column-zero declaration inside a fenced block counts (no fence awareness in POSIX grep); an unknown future model fails until the token list is extended. BLIND SPOT, stated because it is structural: this ledger is EXCLUDED from the check it records — a ledger that quotes defects self-rejects the checks that detect them, and scanning it would fail the repo forever on rows like this one. OVER-ESCALATION WARNING: `harden-finding` compares fingerprints only, so a later in-class defect OUTSIDE the guarded spelling will be proposed for a stronger rung than anything justifies; that is a proposal a human validates, and this row names the guard so they can judge it. The skill's scope-blind recurrence rule is parked in todos.md | +| 2026-07-26 | docs-drift | third occurrence, resolved: a prose count contradicting the checklist it counts — PR #12's `docs/prompt-standards.md` said "all ten items" while its own checklist ran 1-12 | bot | minor | 2 lint | scripts/check-invariants.sh check 4b + fixtures in .test.sh. **Resolves the 2026-07-25 `pending` row** (same fingerprint, appended not edited). GUARDED SPELLING, exactly: two-stage claim recognition over `*.md` — stage 1 matches `all ( checklist)? items` with outer word boundaries, stage 2 requires canonical decimal, so `all 012 items` fires as malformed rather than being invisible; every recognised claim must equal N, where N comes from exactly one `^## Checklist([[:space:]].*)?$` heading whose body holds labels forming the contiguous canonical sequence 1..N, in BOTH docs/prompt-standards.md and the workflow-init template, which must agree. Word forms are in scope BECAUSE the motivating occurrence was one — a digit-only check would have sailed past the very defect this row claims to harden. STILL INSTRUCTION-BACKED: counts above twenty in word form, ordinals, hyphenated compounds (`all twenty-one items`), split-line claims, and any count of something other than this checklist. Same ledger blind spot and over-escalation warning as the row above | diff --git a/docs/superpowers/plans/2026-07-26-prompt-conformance-checks.md b/docs/superpowers/plans/2026-07-26-prompt-conformance-checks.md new file mode 100644 index 0000000..78b5a95 --- /dev/null +++ b/docs/superpowers/plans/2026-07-26-prompt-conformance-checks.md @@ -0,0 +1,537 @@ +# Plan — PR 1: mechanical rung for two prompt-conformance classes + +Layer 1 of a two-PR split. **This PR is `scripts/` + the doc sites those checks +directly describe.** It touches no `plugins/**` path, so invariant 12 requires no +version bump here. A's §5 nudge, C's companions, the template sync and the rest of the +housekeeping are PR 2. + +Two ledger rows dated 2026-07-25 sit at rung `pending` and name this work as their +resolution vehicle. This PR resolves both. + +--- + +## Task 1 — Check 4a: target model named + +**Scope.** Hardens the third `unverified-enforcement-claim` occurrence: a prompt +asserting it follows `docs/prompt-standards.md` while naming no executing model. + +**Rule.** For each in-scope file containing the tested assertion spelling +`prompt artifact and follows`: +1. **Count all** `Target model:` declarations; require **exactly one**. Counting all — + not "exactly one non-empty" — is deliberate: the latter accepts a valid line beside + any number of empty ones, which is malformed state reported as conforming. + **One declaration grammar, used for both counting and value validation:** the + anchored `^Target model:` at column zero. Indented lines, blockquoted ones + (`> Target model:`) and mid-sentence prose mentions do not count — the column-zero + anchor excludes them for free. Using one pattern for both counting and validation is + the point: doing it with two patterns is how a file passes cardinality on one set of + lines and value-validation on another. Near-miss fixtures: indented, blockquoted, + prose mention. + **Fenced blocks are a stated limitation, not a claim.** A column-zero + `Target model:` inside a ``` fence **does** count — POSIX grep has no fence + awareness, and specifying one would mean a fence-length/indentation/unclosed-fence + parser that no other rule here needs. Verified with `/usr/bin/grep`: all four claiming + files currently carry exactly **one** column-zero declaration each, so nothing in the + tree depends on the distinction today. If a future template embeds a fenced example + declaration, the cardinality rule fires — a false positive whose fix is obvious, which + is the safe direction. +2. That sole value must be non-empty and **begin with** a tested token, matched in + **ERE (`grep -E`)** with a **portable** boundary — `\b` is a GNU/PCRE extension, not + POSIX, and this repo has already been bitten once by assuming a non-POSIX regex + construct (the BSD-sed `\|` alternation documented in `unquote()`): + `^Target model: *(Claude|Codex|GPT)([^[:alnum:]_]|$)` + Without the boundary, `ClaudeX`, `Codex2` and `GPTfoo` all pass. Stated + implementation-neutrally on purpose: the checker runs under `sh` against whatever + `grep` is on `PATH` — BSD grep locally, GNU grep on `ubuntu-24.04` CI — and only POSIX + ERE constructs may be assumed. A `\b` that happens to work in both is still the wrong + thing to write, for the same reason the BSD-sed `\|` alternation in `unquote()` was. +3. **Count distinct recognized tokens in the value; require exactly one.** No separator + grammar at all. Each of `Claude`, `Codex`, `GPT` is tested for whole-word presence + with the portable boundary `(^|[^[:alnum:]_])TOKEN([^[:alnum:]_]|$)`; the count of + *distinct* tokens present must be 1. + This replaces an earlier separator-enumeration rule (` or `, ` and `, `/`, `,`) and + its generated 24-case matrix. Enumerating separators was both incomplete — it missed + GPT-leading values and several orderings — and fragile, since a boundary group + consumes the separator's leading space and breaks the very pattern meant to find it. + Counting distinct tokens is mechanically complete over the stated rule instead of + approximating it. Verified across nine values: + + | Value | distinct | verdict | + |---|---|---| + | `Claude via Claude Code` | 1 | accept (same token twice is one model) | + | `Claude in a chat interface, upstream of Claude Code` | 1 | accept | + | `Claude via Claude Code or the API` | 1 | accept — one model, two surfaces | + | `Claude or Codex` / `GPT/Codex` / `Codex, Claude` / `Claude and Codex` | 2 | reject, every separator, no separator rule needed | + | `Claude or Codex2` | 1 | accept — `Codex2` is not a recognized model | + | `any capable chat model` | 0 | reject | + + Rules 2 and 3 are both required and neither subsumes the other: the historical defect + `any capable chat model (… developed with Claude …)` has **one** distinct token and is + caught only by rule 2's begins-with anchor. + +**Why begins-with and not contains.** Presence alone accepts +`Target model: any capable chat model` — the exact PR #12 defect. Token-*anywhere* also +fails: the real defect line read `any capable chat model (the role was developed with +Claude …)` and **contains** `Claude`. Measured: anywhere → 1 match (misses it), +at-start → 0 (catches it). + +**Stays instruction-backed, and the ledger row must say so.** A bare +`Target model: Claude` naming no execution surface passes; so does any value whose prose +is wrong in a way no token test sees. An unknown future model fails until the list is +extended — a false positive, the safe direction. + +**Files.** `scripts/check-invariants.sh`. + +--- + +## Task 2 — Check 4b: checklist count claims + +**Scope.** Hardens the third `docs-drift` occurrence: a prose count contradicting the +checklist it counts. + +**Rule — definitions.** Exactly **two** files define a checklist: +`docs/prompt-standards.md` and the template inside +`plugins/dev-workflow/commands/workflow-init.md` (read only — not edited by this PR). +Each must have **exactly one** checklist heading matching the anchored ERE +`^## Checklist([[:space:]].*)?$` — **not** the literal string `## Checklist`, which +matches neither real file: both actually read +`## Checklist (each item must be verifiably true)`, so an exact-literal implementation +rejects the real repository before comparing a single item. Near-miss fixtures must pin +the grammar (`## Checklists`, `### Checklist`, `## Checklist-ish`, and the real +parenthesised form). Its bounded body (that heading → +the next `## `) holds **at least one** item line matching `^[1-9][0-9]*\. \*\*` — a +**canonical decimal** label, so `0.` and leading-zero forms like `01.` are malformed and +fire rather than being silently renumbered into `1..N`. Claims are recognized in **two stages**, because one canonical pattern alone would make +a malformed claim *invisible* rather than rejected: first a **near-claim detector** +`(^|[^[:alnum:]_])all [0-9]+( checklist)? items([^[:alnum:]_]|$)` matches any digit run; +then the matched digits must be canonical `[1-9][0-9]*` or the claim **fails as +malformed**. Without that first stage, `all 012 items` matches no canonical claim, is +silently ignored, and its required reject fixture cannot pass — which is the state an +earlier draft specified. Comparison is done as **strings after canonicalization**, never +by shell arithmetic on an unbounded digit run, so a very long digit string cannot +overflow or error its way into a wrong verdict; uncertainty fires, per invariant 2. +Fixtures: `0.`-labelled, `01.`-labelled, `all 012 items`, and a 40-digit claim. Each +checklist section's labels must additionally form exactly the **contiguous canonical +sequence 1 through N** — so `1, 1, 3` is malformed rather than silently authoritative at +N=3. Zero, empty, duplicate or multiple sections **fail**; +never a silent 0-vs-0 compare. The two counts must be equal, giving one authoritative `N`. + +**Rule — claims. Word forms are IN scope, and this is the correction that makes the +resolution honest.** An earlier draft scoped this check to digit spellings only. That +would have been a false hardening: the actual 2026-07-25 `docs-drift` occurrence this +check resolves was **`all ten items`** — a word form (verified in git: +`4f31df2` removed the line "nobody reviews them against all ten items per brief"). A +digit-only check sails straight past the defect it claims to guard, which is the same +mistake made once already on 4a, where a presence-only test accepted +`any capable chat model`. + +So the recognized claim grammar is +`(^|[^[:alnum:]_])all (|)( checklist)? items([^[:alnum:]_]|$)`, +where `` is the bounded list **one … twenty** mapped to digits. **The outer +boundaries are required, not decorative:** without them the same ERE also matches +`small ten items` and `all ten itemsized` — measured, 3 matches unbounded versus 1 +bounded — so an unbounded implementation would reject unrelated prose while the ledger +row claimed a bounded guard. **Every** claim must equal `N`; comparing every +claim to `N` makes conflicting duplicates fail by construction, so no separate +claim-cardinality rule is needed. Fixtures must include the historical +`all ten items` against a 12-item checklist. + +Out of scope, stated rather than implied: numbers above twenty in word form, ordinals, +hyphenated compounds (`all twenty-one items`), and claims split across lines. The ledger +row names exactly this boundary. + +**Files.** `scripts/check-invariants.sh`. + +--- + +## Task 3 — Scan domains and exclusions, stated per check + +**Scope.** Pass 6 found the single exclusion table ambiguous: it was labelled "claim scan +only" while one row said "4a only", leaving no derivable authoritative domain. + +**Rule — positive domain first.** Both checks scan **`--include='*.md'` only**, rooted at +the repository root, via `grep -r` (which does **not** follow symlinks; `-R` would — the +non-following form is the intended one, so a checked-in symlink cannot drag an +out-of-tree file into scope). Markdown-only is the right domain because both rules are +about prompt text; the existing invariant-5 scan's wider +`*.yml/*.yaml/*.json/*.toml` set is deliberately **not** reused, since a `Target model:` +line in a JSON fixture is not a prompt claim. Hidden directories are in scope by default, +which is exactly why `.context/` needs an explicit exclusion below rather than relying on +its leading dot. + +Then the exclusions — two separate domains. + +| | 4a (assertion scan) | 4b (claim scan) | +|---|---|---| +| `source-files/` | excluded — frozen extraction archive, never edited | excluded — same | +| `docs/superpowers/` | excluded — historical artifacts | excluded — its plans legitimately say "all 11 checklist items"; **load-bearing, not tidy** | +| `.context/` | excluded — generated review artifacts; the quality command must not depend on ephemeral review wording | excluded — same | +| `docs/hardening-log.md` | **excluded** — the ledger *quotes* finding text, so a row describing this very defect trips its own check (verified: it matches the phrase and has no `Target model:` line) | **excluded — and this one is a BLOCKER if missed**, see below | + +**The ledger exclusion is the one that would otherwise break the build**, and it is the +only self-referential trap left once the domain is Markdown-only. Every exclusion carries +a paired control (Task 5). + +**Path-form rule — prefix AND suffix, because the two scans emit different shapes.** +Prefix form varies by `grep` implementation and must not be assumed: write exclusions +prefix-independently, e.g. `(^|/)source-files/` rather than `^\./source-files/`. + +The suffix matters just as much, and getting it wrong silently disables an exclusion. +4a's file scan (`grep -rl`) emits **bare paths**, so `(^|/)hardening-log\.md$` works. +4b's claim scan (`grep -rno`) emits **`path:line:match`**, so that same `$`-anchored +pattern matches nothing and the ledger is scanned anyway — reproducing the BLOCKER while +looking excluded. Verified by simulating both checks against the real tree: with the +`$`-anchored pattern 4b still returned 3 claims including the ledger's `all ten items`; +with `(^|/)hardening-log\.md($|:)` it returns 2, both `12`, and the same pattern still +excludes correctly in the bare-path form. **Use the `($|:)` form for every exact-file +exclusion**, and let the paired controls (Task 5) prove each one. + +**Real-repo simulation is a required pre-implementation step, not a nicety.** Both +checks were simulated against the actual tree while writing this plan; that is what +caught the anchor defect above. 4a currently passes on all four in-scope files +(1 declaration, 1 valid value each); 4b agrees at N=12 across both definitions. + +**Why the ledger must be excluded from 4b as well — the append-only trap.** Once word +forms entered 4b's grammar, the ledger became self-rejecting: `docs/hardening-log.md:27` +is the 2026-07-25 `pending` row, and it *quotes* the historical defect — +`… not reviewed against "all ten items" while its own checklist …`. So 4b sees a claim of +`ten` against a 12-item checklist and fails. The ledger is **append-only** and that row +must stay byte-unchanged, so this would reject the real repository **forever**, making +success criterion 3 unreachable and the resolution row a lie. Measured with +`/usr/bin/grep`: **3** in-domain claims (`hardening-log.md:27` `all ten items`, +`prompt-standards.md:14` `all 12 items`, `AGENTS.md:169` `all 12 checklist items`) → +**2** after the exclusion, both `12`. +The general shape, worth stating because it will recur: **a ledger that quotes defects +cannot be scanned by checks that detect those defects.** Both checks exclude it for the +same reason, and the new ledger rows must record that quoted-evidence blind spot as +instruction-backed. + +**No `.sh` self-exclusion is needed, and requiring one was a contradiction.** An earlier +draft excluded `check-invariants.sh` and `.test.sh` explicitly and demanded fixtures like +`scripts/check-invariants-helper.sh`. Impossible: the scan is `--include='*.md'`, so no +`.sh` file can enter scope at all (verified: 0 `.sh` matches). The domain restriction +already does that job; the explicit exclusion added only unsatisfiable fixtures. + +**Measure with `/usr/bin/grep`, not the shell's `grep` — this invalidated four earlier +measurements.** An interactive shell here defines `grep` as a **function shimming to +ugrep** with `--ignore-files`, i.e. it honours `.gitignore`. The checker runs under `sh` +and gets the real `grep`. The two disagree on exactly the things this task depends on: + +| | shell `grep` (ugrep shim) | `/usr/bin/grep` (what runs) | +|---|---|---| +| path prefix | no `./` | `./` | +| `.gitignore`d paths | skipped | **scanned** | +| `.context/` review artifacts | invisible | **visible** | + +Consequence, and it flips an earlier conclusion: a probe through the shim reported "no +hits" in `.context/`, suggesting that exclusion was merely prudent. With the real grep, +`.context/codex-reviews/gate-a-plan-pass-6.md` **does** match the 4a trigger phrase. The +`.context/` exclusion is load-bearing, not speculative. Verified totals with +`/usr/bin/grep`: 10 candidate files before exclusions, **4** after (`docs/sparring-briefing.md`, +`workflow-init.md`, `harden-finding/SKILL.md`, `intake/SKILL.md`); 4b finds 2 claims, +both `12`. + +**Files.** `scripts/check-invariants.sh`. + +--- + +## Task 4 — Mutation evidence: a one-time run, recorded + +**Scope.** Show each new check is load-bearing — that its fixtures fail when the check +is removed — as **documented evidence from a development-time run**, not as permanent +machinery. + +**Why this shape.** It is the house pattern: every mutation verification in this +repository so far (the hook suite, `check-version-bump`) was an evidenced run recorded as +"reverting X fails exactly assertions Y". Three earlier drafts tried to build an +automated oracle instead, and each attempt produced a defect the next review pass caught — +a test-only path override that was a no-op, a fixed-destination fix that was a no-op one +level down, an expected-delta enumeration that could not include accept fixtures, a +sentinel that would have required sentinel-only behaviour in the *production* checker. +**No path parameterization is needed at all:** a one-time run copies the repo to scratch +and neuters the copy in place, so the suite resolves its sibling checker normally. + +**Procedure** (per check; `4a` shown): +```sh +TMP=$(mktemp -d) || exit 1 +[ -n "$TMP" ] && [ -d "$TMP" ] || { echo "no scratch dir" >&2; exit 1; } +trap 'rm -rf "$TMP"' EXIT HUP INT TERM # not just fall-through: an abort otherwise + # leaves a whole repo copy (incl. .git) behind +cp -R . "$TMP/repo" +sed '/BEGIN check 4a/,/END check 4a/d' scripts/check-invariants.sh > "$TMP/repo/scripts/check-invariants.sh" + +sh scripts/check-invariants.test.sh > "$TMP/before" 2>&1; base=$? +( cd "$TMP/repo" && sh scripts/check-invariants.test.sh ) > "$TMP/after" 2>&1; mut=$? + +# Every status is ASSERTED, not merely captured — an unasserted status is the +# verification-masks-failure class this procedure claims to avoid. +[ "$base" -eq 0 ] || { echo "VOID: baseline suite not green ($base)" >&2; exit 1; } +[ "$mut" -ne 0 ] || { echo "VOID: mutant suite passed — the check is not load-bearing" >&2; exit 1; } +diff "$TMP/before" "$TMP/after" | grep '^> FAIL' | sed 's/^> FAIL - //; s/ (.*)$//' > "$TMP/flipped" +[ -s "$TMP/flipped" ] || { echo "VOID: no assertions flipped" >&2; exit 1; } +cat "$TMP/flipped" # <- this list is the evidence that gets recorded +``` +**What the script asserts, and what it does not — stated precisely, because overclaiming +here is the failure this whole procedure exists to avoid.** The commands reject exactly +three states: a baseline that was already red, a mutant that passed (so the check was not +load-bearing), and an empty flipped set. They do **not** establish that the mutant failed +for the *right* reason: a syntax error in the neutered copy could break accept fixtures +and produce a non-empty flipped list that these checks happily record. + +That last validation is a **human step, and it is mandatory**: compare the printed +flipped set against the reject cases listed for that check in Task 5's fixture table — +exact set equality, no extras, nothing missing — and confirm the mutant's output carries +no checker abort or syntax diagnostic. The dry-run's three flips were confirmed that way, +by reading them, not by the script proving them. Recording evidence without that +comparison is not permitted, and no wording anywhere may imply the script performs it. + +**Validated before implementation.** The procedure was dry-run against the *existing* +invariant-6 block: `base=0`, `mut=1`, and exactly three assertions flipped — +`hooks key rejected`, `skills key rejected`, `key/colon split across lines rejected`. +That also supplies Task 5's load-bearing evidence for the pre-existing checks, so one +procedure serves both purposes. + +**Recording the evidence — three places, none of them optional.** +1. **PR body** — each mutant and the exact assertion names it flipped. +2. **A short comment block in `scripts/check-invariants.test.sh`** — the same + mutant → flipped-assertions mapping, so it travels with the file rather than living + only in a merged PR. +3. **The checker's header comment** — the re-run procedure and its trigger. The trigger + must be **broader than "the scan logic"**: the recorded mapping is invalidated by a + change to either marked check, to the markers themselves, to any of those checks' + fixtures or assertion names, or to the harness that runs them. A narrow trigger would + let the evidence comment go stale while claiming no re-run was due — which is the + dishonesty prompt-standards item 11 forbids, in the very block that documents it. + After any such run, **both** recorded mappings (PR body and test-file comment) are + updated together. + +**Honest phrasing is part of the deliverable (invariant 11, prompt-standards item 11).** +None of these three may describe the evidence as automated enforcement. Nothing re-runs +it; it is a documented manual step with a stated trigger, and the comment must say so. +Calling it a guard would be precisely the `unverified-enforcement-claim` this PR hardens. + +**What this deliberately does not do, and why that is acceptable now.** It does not catch +a *future* checker edit that silently kills fixtures. That guard already exists at the +process level — any `scripts/` change fires full Gate B, and the documented procedure +makes re-running cheap. Permanent machinery to replace a documented manual step needs a +recurrence to justify it: a checker edit that shipped with silently-dead fixtures. None +has happened. If one does, that recurrence is the trigger to build the oracle. + +**Files.** `scripts/check-invariants.sh` (header comment + `BEGIN`/`END` markers), +`scripts/check-invariants.test.sh` (evidence comment block). + +--- + +## Task 5 — Prerequisite: one shared fixture initializer, then the fixtures + +**Scope.** 4b's cardinality rule is mandatory, so every fixture repo needs a valid +checklist or it fails before reaching its own assertion. + +**Rule.** **Four** call sites `cp "$CHECKER"` and build a repo directly — `run_with`, +`sh_case`, and two inline blocks (verified by grep). None creates +`docs/prompt-standards.md` or `workflow-init.md`. Extending `run_with` alone leaves three +broken: accept cases turn red and reject cases start passing for the wrong reason — +exactly the diagnostic-isolation failure the suite's `$5`-substring guard exists to +prevent. Extract **one** initializer installing a valid agreeing checklist pair plus a +matching claim; call it from all four; **then** add enforcement. + +**Fixtures.** Following the suite's `expect_reject`/`expect_accept` pattern — every +reject case names a substring of its expected diagnostic, so it cannot pass on an +unrelated violation. + +**Multi-model fixtures follow the distinct-token rule, so the case list is small and +complete.** Because rule 3 counts distinct tokens rather than matching separators, the +fixtures need only pin the rule's boundaries, not a separator matrix: one reject per +distinct pair using a *different* separator each (`Claude or Codex`, `GPT/Codex`, +`Codex, Claude`, `Claude and Codex`) to show the verdict is separator-independent; and +the **four** accepts that define the edges — same token repeated +(`Claude via Claude Code`), the chat-interface form +(`Claude in a chat interface, upstream of Claude Code`), one model with two surfaces +(`Claude via Claude Code or the API`), and a token-prefix non-model +(`Claude or Codex2`, which must **accept** because `Codex2` is not a recognized model). +All **eight** verdicts verified before implementation. + +| Check | Reject cases | Accept cases | +|---|---|---| +| 4a | no `Target model:` line; two declarations; one valid + one empty declaration; empty value; `any capable chat model (…Claude…)`; token-prefix near-misses `ClaudeX`, `Codex2`, `GPTfoo` as the *first* token; four two-token rejects, each with a **different** separator to show the verdict is separator-independent — `Claude or Codex`, `GPT/Codex`, `Codex, Claude`, `Claude and Codex` | `Claude via Claude Code` (same token twice = one model); `Claude in a chat interface, upstream of Claude Code`; `Claude via Claude Code or the API` (one model, two surfaces); **`Claude or Codex2`** (a token-prefix in *second* position is not a second model — the settled boundary) | +| 4b | digit claim ≠ N; **word-form claim ≠ N (`all ten items` vs a 12-item checklist — the historical defect)**; checklist section absent; section present but empty; duplicate `## Checklist` heading; non-contiguous labels `1, 1, 3`; the two definitions disagreeing; near-miss headings `## Checklists`, `### Checklist`, `## Checklist-ish` | one valid digit claim matching N; a word-form claim matching N (`all twelve items`); two agreeing claims; the real parenthesised heading `## Checklist (each item must be verifiably true)` | +**Three fixtures are mandatory because they lock this plan's hardest-won lessons into +code rather than into an appendix nobody re-reads.** Each was a BLOCKER found at Gate A, +and each was introduced by the fix for the previous one: +| Lesson | Fixture | +|---|---| +| a ledger that quotes defects self-rejects the check that detects them | a fixture repo whose `docs/hardening-log.md` carries `all ten items` and a quoted assertion phrase — must **accept** | +| an expected-delta set cannot include accept fixtures, which stay `ok` when a check is disabled | an accept fixture that stays `ok` under the 4a mutation, asserted explicitly | +| a canonical-only grammar makes malformed input *invisible* rather than rejected | `all 012 items` — must **reject** as malformed, via the two-stage detector | + +Exclusion fixtures are a **per-check matrix**, not one row — 4a and 4b have different +exclusion sets, some entries are directory prefixes and two are exact files: + +| Excluded path | Applies to | Excluded fixture (must ACCEPT) | Neighbour (must REJECT) | +|---|---|---|---| +| `source-files/` | 4a, 4b | `source-files/x.md` | `source-filesX/x.md` | +| `docs/superpowers/` | 4a, 4b | `docs/superpowers/x.md` | `docs/superpowersX/x.md` | +| `.context/` | 4a, 4b | `.context/x.md` | `contextX/x.md` | +| `docs/hardening-log.md` | **4a and 4b** | that exact file, carrying both a quoted assertion phrase and `all ten items` | `docs/hardening-log-notes.md` | + +Each row carries the **same** violating content on both sides. That pairing is what +proves an exclusion is load-bearing rather than an overbroad filter — or a fixture that +never matched the rule at all. The neighbour column also pins the path-form rule: a +prefix-anchored exclusion that accidentally matches `source-filesX/` fails its own row. + +**Mutation evidence lives in Task 4** — a one-time, recorded run, not an oracle built +here. The only thing this task owes it is the **seam**: each new check's body in +`check-invariants.sh` is wrapped in exact marker comments +(`# --- BEGIN check 4a ---` / `# --- END check 4a ---`, likewise 4b) so the `sed` range +delete has something stable to cut. Nothing else keys on the markers. + +**Files.** `scripts/check-invariants.test.sh`. + +**Verify.** Suite green. And the pre-existing checks must be shown still load-bearing +after the initializer — but **not** by "re-running the invariant-5/6 mutation": no such +procedure exists in the suite today, so that instruction was unexecutable. Instead, name +concrete pre-existing reject cases and re-verify each still fails **with its own +diagnostic** (the `$5` substring), so a case that survives only because its check was +accidentally disabled is distinguishable from one that genuinely rejects: +- invariant 5: `unpinned npx in a shell script rejected`, `major-only ref rejected`, + `branch ref rejected`, `ubuntu-latest rejected`; +- invariant 6: `hooks key rejected` (diagnostic substring + `re-declares a convention-loaded`). + +**These names are grep-verified, and two earlier ones were not.** A prior draft of this +task named `unpinned action ref rejected` and `ubuntu-latest runner rejected`; neither +exists (`grep -cF` → 0). The real cases are the ones above. An unverified fixture name is +the same defect class this PR hardens, one layer up — so the implementer must verify each +name before relying on it. + +**Verify with the quoted argument, expecting exactly 1.** A bare +`grep -cF 'ubuntu-latest rejected'` returns **3** — the substring also occurs inside +`quoted ubuntu-latest rejected` and `matrix ubuntu-latest rejected`, so a bare +substring count cannot identify which case is meant. Search the quoted assertion +argument (`grep -cF '"ubuntu-latest rejected"'`) and require a count of exactly 1 for +every selected case. + +--- + +## Task 6 — Docs-drift sweep, by claim not phrase + +**Scope.** A third and fourth check makes existing descriptions of this script stale. + +**Rule.** Verify every sentence describing `check-invariants.sh` against what it now +implements. Known sites — the fourth was missed by the previous sweep list and is why +this searches by claim: + +**Files.** `scripts/check-invariants.sh` header ("Both checks"); `AGENTS.md:37` +(layout tree) and `AGENTS.md:234` (Commands row) — both say "invariants 5 and 6"; +`docs/architecture.md:18`; `README.md:126` — "(invariants 5 and 6)", contributor-facing; +`.github/workflows/ci.yml:76` (the comment "Invariants 5 and 6 mechanically") **and** +`ci.yml:83` (the step name "Invariant checks (pinning, manifest) + both checker suites") +— two distinct sites in that file, not one. + +**Seven locations across five files — and every single pass found one more.** Three in +the previous plan → `docs/architecture.md` (pass 6) → `README.md` (PR1 pass 1) → +`AGENTS.md`'s second site (pass 2) → `ci.yml`'s second site (pass 3). That monotone +escalation *is* the `docs-drift` class this PR hardens, demonstrating itself inside the +plan that hardens it. Which is exactly why the rule searches **by claim** and the final +verify **greps** — this table is a starting point, and its own history says it will be +incomplete again. + +**Verify.** `grep -rn` by claim (`check-invariants`, "both checks", "invariant checks", +CI step text) returns no sentence contradicting the implemented checks. + +--- + +## Task 7 — Ledger: two appended rung-2 rows + +**Scope.** Resolve the two `pending` rows. + +**Rule.** + +| fingerprint | source | severity | rung | ref | +|---|---|---|---|---| +| `unverified-enforcement-claim` | `bot` | major | 2 lint | check 4a; **resolves** the 2026-07-25 `pending` row by reference | +| `docs-drift` | `bot` | minor | 2 lint | check 4b; **resolves** the 2026-07-25 `pending` row by reference — and the row must state that the guard covers digit **and** bounded word-form (one…twenty) count claims, because the occurrence being resolved was the word form `all ten items`; above twenty, ordinals, hyphenated compounds and split-line claims stay instruction-backed | + +`source` stays `bot` — it records the surfaced finding's provenance, and both pending +rows record `bot`; switching to `manual` would make one finding read as a fresh manual +occurrence. Each row states the exact spelling its check guards, what stays +instruction-backed, **and** the over-escalation warning: `harden-finding` compares +fingerprints only, so a later in-class defect *outside* the guarded spelling will be +proposed for a stronger rung than anything justifies. Accepted deliberately — escalation +is a proposal a human validates, and the row names the guarded spelling for that human. +No row is ever edited; resolution is by appending. + +**Files.** `docs/hardening-log.md`. + +**Verify.** Anchored column-2 grep counts; both `pending` rows byte-unchanged; both new +rows read `bot`. + +--- + +## Task 8 — todos.md: mark the vehicle consumed + +**Scope.** The PR #12 entry naming this round as the resolution vehicle. + +**Rule.** There is no checker/template "half" — they are **two separate unchecked items**. +Precisely: the standalone item **"Prompt-standards conformance checker — resolves two +`pending` ledger rows (2026-07-25)"** becomes checked, and its now-obsolete +resolution-vehicle text (which names the canvas round and a trigger) is rewritten to name +this PR as what resolved it. The **separate, preceding** "ad-hoc task briefs are prompts +too" template-sync item stays **byte-unchanged** — it is PR 2's. **Edit only.** + +**Files.** `todos.md`. + +--- + +## Invariants touched + +**2** (directional — every ambiguous checklist shape fires rather than passing), **5** +and **6** (the checks this script already enforces must keep working — evidenced by +re-running the exact named pre-existing reject cases in Task 5 and confirming **each +case's own diagnostic**, *not* by "re-running their mutation": no invariant-5/6 mutation +procedure exists in the suite, so that instruction was unexecutable and is gone from this +plan entirely), **11** (below). **12 does not apply: no `plugins/**` path is modified**, +so no version bump. Hooks are not touched. + +## Success criteria + +1. **Prompt-standards self-review (invariant 11) — required, and an earlier draft of + this plan got it wrong.** That draft claimed "no prompt artifact is edited by this PR". + False: Task 6 edits `AGENTS.md`, and `CLAUDE.md:221-222` classes `CLAUDE.md` and + `AGENTS.md` themselves as **prompts, not prose**. So the changed instruction text gets + a full 12-item `docs/prompt-standards.md` review before Gate B, and the `AGENTS.md` + edit fires full Gate B rather than the prose exemption. (`README.md`, by the same rule, + *is* prose — it describes the product rather than instructing a model.) A false scope + claim here would have routed changed model instructions around this repo's only + prompt-quality gate. +2. The **exact** quality command from `AGENTS.md § Commands`, run verbatim and green — + not a subset: shellcheck over all six scripts, **all three** regression suites (hook, + invariant-checker, version-bump-checker), and `claude plugin validate . --strict`. +3. `sh scripts/check-invariants.sh` exits 0 **on the real repository**, not only in + fixtures — the assertion pass 6's F4 showed a fixture-only check would miss. +4. The mutation run yields **both**: the script's validity assertions passing (baseline + green, mutant non-zero, non-empty flipped set) **and** the retained exact flipped-assertion + output, human-compared to the fixture table per Task 4. A prose attestation + unsupported by that retained output is not evidence and is not acceptable. + +--- + +# Appendix — decisions + +**Where plan review stops.** Plan-level review converges on **what** a test proves; +**how** it proves it is code, and code is reviewed at Gate B. Nine Gate-A passes +rediscovered this — three consecutive BLOCKERs, each introduced by the fix for the +previous one (word-forms → ledger self-reject; `CASES_*` derivation → impossible delta; +canonical decimal → `all 012 items` invisible), with finding totals flat across the run: +7, 7, 5, 6, 6, 4, 5, 5, 5. Everything load-bearing here was settled by *running* it +against the real repository; what kept breaking was prose specifying a test procedure. +If this lesson recurs in a later round it graduates to `docs/prompt-standards.md`; until +then it is a decision, not a rule. + +**Mutation evidence is a recorded run, not an oracle.** House pattern (the hook suite and +`check-version-bump` were verified the same way). Four attempts to specify an automated +oracle each produced a defect the next pass caught; the recorded run worked first try. + +**Both 4a rules are required.** The begins-with anchor and the distinct-token count do +not subsume each other: the historical defect `any capable chat model (… Claude …)` has +exactly one distinct token and is caught only by the anchor. diff --git a/scripts/check-invariants.sh b/scripts/check-invariants.sh index 96be1f8..bbd5808 100755 --- a/scripts/check-invariants.sh +++ b/scripts/check-invariants.sh @@ -1,7 +1,7 @@ #!/bin/sh # Mechanical checks for the AGENTS.md invariants that a tool can decide. # -# Both checks exist because prose alone did not hold. Invariant 5 was written down +# These checks exist because prose alone did not hold. Invariant 5 was written down # and this repo's own CI still shipped `actions/checkout@v4` and `ubuntu-latest`; # invariant 6 was believed to say the opposite of what it says, and the resulting # duplicate-hooks manifest key stopped the plugin loading entirely (0.2.1). A rule a @@ -20,6 +20,37 @@ # checker — it raises the floor, it is not a proof. # # TESTED SPELLINGS ONLY: extend the fixtures before extending the regex. +# +# MUTATION RE-RUN PROCEDURE (manual; nothing automates it). The two prompt-conformance +# checks below are bracketed by `# --- BEGIN check 4a ---` / `# --- END check 4a ---` +# markers so a scratch copy can be neutered cleanly: +# +# TMP=$(mktemp -d) || exit 1 +# [ -n "$TMP" ] && [ -d "$TMP" ] || exit 1 # else the copy below targets /repo +# trap 'rm -rf "$TMP"' EXIT HUP INT TERM +# mkdir -p "$TMP/repo"; tar cf - --exclude=.git . | (cd "$TMP/repo" && tar xf -) +# sed '/BEGIN check 4a/,/END check 4a/d' scripts/check-invariants.sh \ +# > "$TMP/repo/scripts/check-invariants.sh" +# sh scripts/check-invariants.test.sh > "$TMP/before" 2>&1; base=$? +# ( cd "$TMP/repo" && sh scripts/check-invariants.test.sh ) > "$TMP/after" 2>&1; mut=$? +# [ "$base" -eq 0 ] || { echo "VOID: baseline not green" >&2; exit 1; } +# [ "$mut" -ne 0 ] || { echo "VOID: check is not load-bearing" >&2; exit 1; } +# diff "$TMP/before" "$TMP/after" | grep '^> FAIL' | sed 's/^> FAIL - //; s/ (.*)$//' \ +# > "$TMP/flipped" +# [ -s "$TMP/flipped" ] || { echo "VOID: nothing flipped" >&2; exit 1; } +# cat "$TMP/flipped" +# +# Each validity check EXITS rather than warning: a check that prints and continues lets +# a red baseline, a passing mutant or an empty flip set be recorded as evidence, which +# is the failure this procedure exists to prevent. +# +# The script's checks establish only that the baseline was green, the mutant failed, and +# something flipped. They do NOT establish the mutant failed for the right reason — a +# syntax error in the neutered copy would also flip cases. Comparing the flipped set +# against the fixture list, and confirming no accept case moved, is a HUMAN step and is +# mandatory. The recorded result lives in check-invariants.test.sh; re-run and update it +# when changing either marked check, its markers, its fixtures or assertion names, or +# the harness. set -u root=$(cd "$(dirname "$0")/.." && pwd) @@ -227,5 +258,205 @@ for manifest in plugins/*/.claude-plugin/plugin.json; do fail "Invariant 6: $manifest re-declares a convention-loaded component." "$bad_keys" done +# Scan domain for the two prompt-conformance checks below: Markdown only, because both +# rules are about prompt text. The wider yml/json/toml domain used by invariant 5 is +# deliberately NOT reused — a `Target model:` line in a JSON fixture is not a prompt +# claim. `grep -r` does not follow symlinks (`-R` would), which is the intended form. +# +# Exclusions, each for its own reason. Anchored `($|:)` at the end, NOT `$`: the file +# scan emits bare paths while the claim scan emits `path:line:match`, and a `$`-anchored +# pattern silently matches nothing in the second form — an exclusion that looks applied +# and is not. +# source-files/ frozen extraction archive, never edited (MANIFEST.md) +# docs/superpowers/ historical artifacts; its plans legitimately say "all 11 checklist +# items", so excluding it is load-bearing, not tidy +# .context/ generated Gate A/B review artifacts; the quality command must not +# depend on ephemeral review wording +# hardening-log.md the ledger QUOTES defects, so a row describing either defect below +# trips the very check that row records. It carries both a quoted +# assertion phrase and the historical `all ten items`. +# No .sh exclusion is needed: `--include='*.md'` already puts this script and its suite +# out of scope. +PROMPT_EXCL='(^|/)source-files/|(^|/)docs/superpowers/|(^|/)\.context/|(^|/)hardening-log\.md($|:)' + +# --- BEGIN check 4a --- +# A file asserting it follows docs/prompt-standards.md must name exactly one executing +# model. Prose alone did not hold: a doc shipped claiming conformance while giving its +# target model as "any capable chat model", which names no model at all. +# +# WHAT THIS CATCHES, exactly — the rest of the class stays instruction-backed: +# the missing/duplicated/unnamed/multi-model spellings of the `Target model:` line, in +# files carrying the tested assertion spelling `prompt artifact and follows`. A bare +# `Target model: Claude` naming no execution surface PASSES, as does any value whose +# prose is wrong in a way no token test can see. +# +# Two independent rules, and neither subsumes the other: the value must BEGIN with a +# recognized token (a token merely present accepts "any capable chat model (… developed +# with Claude …)", which is the exact defect this exists for), and it must contain +# exactly one DISTINCT recognized token (which rejects "Claude or Codex" without needing +# a separator grammar). +# The offenders are COLLECTED and reported once, matching this file's existing idiom +# (`bad_npx`, `bad_keys`). Calling `fail` inside the loop would not work: a `while read` +# fed by a pipeline runs in a subshell, so the `rc=1` it sets is discarded and the +# checker would print every violation and still exit 0. +# The scan's status is captured BEFORE filtering. `grep` exits 0 on a match and 1 on no +# match, but >=2 on a real error (unreadable path, I/O failure, bad option). Piping +# straight into the filter would report the FILTER's status and turn any traversal +# failure into an empty offender set — the checker would print success without having +# looked, which is the fail-open direction invariant 2 forbids. +model_scan=$(grep -rl 'prompt artifact and follows' --include='*.md' . 2>/dev/null) +model_scan_st=$? +[ "$model_scan_st" -le 1 ] || + fail "Prompt standards: the 4a scan failed; results are not trustworthy." \ + "grep exited $model_scan_st" +# The FILTER gets its own status check too. Capturing only the scan's status closed +# traversal errors but left the next stage open: a `grep -vE` failure also yields an +# empty offender set, which reads as clean. +model_files=$(printf '%s\n' "$model_scan" | grep -vE "$PROMPT_EXCL"); model_filter_st=$? +[ "$model_filter_st" -le 1 ] || + fail "Prompt standards: the 4a exclusion filter failed; results are not trustworthy." \ + "grep -v exited $model_filter_st" +bad_model=$(printf '%s\n' "$model_files" | + while IFS= read -r f; do + [ -n "$f" ] || continue + # Per-file statuses are checked too, and an operational failure is emitted as an + # `ERROR:` sentinel line rather than swallowed. The loop runs inside `$( )`, so it + # cannot set rc directly; without the sentinel a `grep` that printed a partial + # count and exited >=2 would leave the offender set empty and the gate would pass. + decls=$(grep -c '^Target model:' "$f"); decls_st=$? + if [ "$decls_st" -gt 1 ]; then + printf 'ERROR: %s: counting declarations failed (grep exited %s)\n' "$f" "$decls_st" + continue + fi + if [ "$decls" -ne 1 ]; then + printf '%s: %s "Target model:" declarations, need exactly 1\n' "$f" "$decls" + continue + fi + # ONE status-bearing command, deliberately not a pipeline. `grep | head | sed` put + # only sed's status in $?, so a grep that printed a valid line and THEN failed + # (status 2) left the file looking conformant — a fail-open path that survived three + # rounds of status-checking because the pipeline's shape hid it. awk also replaces + # `head -1` (`-m` is not POSIX) via `exit` after the first match. + value=$(awk '/^Target model:/ { # extract-target-model + sub(/^Target model:[[:space:]]*/, ""); print; exit }' "$f") + value_st=$? + if [ "$value_st" -ne 0 ]; then + printf 'ERROR: %s: extracting the value failed (pipeline exited %s)\n' "$f" "$value_st" + continue + fi + if ! printf '%s\n' "$value" | grep -qE '^(Claude|Codex|GPT)([^[:alnum:]_]|$)'; then + printf '%s: names no executing model -> Target model: %s\n' "$f" "$value" + continue + fi + distinct=0; tok_err= + for tok in Claude Codex GPT; do + printf '%s\n' "$value" | grep -qE "(^|[^[:alnum:]_])$tok([^[:alnum:]_]|\$)"; tst=$? + if [ "$tst" -eq 0 ]; then distinct=$((distinct + 1)) + elif [ "$tst" -gt 1 ]; then tok_err="grep exited $tst on $tok"; fi + done + if [ -n "$tok_err" ]; then + printf 'ERROR: %s: token matching failed (%s)\n' "$f" "$tok_err" + continue + fi + [ "$distinct" -eq 1 ] || + printf '%s: names %s models, exactly one executes it -> Target model: %s\n' \ + "$f" "$distinct" "$value" + done) +# An operational failure and a real violation are different diagnoses and must not +# share one message: the first means the check did not complete, the second means it did. +if printf '%s\n' "$bad_model" | grep -q '^ERROR: '; then + fail "Prompt standards: the 4a per-file checks failed; results are not trustworthy." \ + "$bad_model" +elif [ -n "$bad_model" ]; then + fail "Prompt standards item 1: a file claiming conformance does not name one executing model." \ + "$bad_model" +fi +# --- END check 4a --- + +# --- BEGIN check 4b --- +# A prose count of the prompt-standards checklist must equal the number of items in it. +# The motivating occurrence was the WORD form "all ten items" against a 12-item list, so +# word forms one..twenty are in scope; above twenty, ordinals, hyphenated compounds and +# split-line claims are not, and stay instruction-backed. +# +# Claims are recognized in TWO stages on purpose. A canonical-only pattern would make a +# malformed claim invisible rather than rejected: `all 012 items` matches no canonical +# claim and would be silently ignored. So stage 1 matches any digit run, and stage 2 +# requires it to be canonical decimal. +# Prints the item count, or 'BAD' for a malformed definition. Returns 2 if the PARSER +# itself failed, which is not the same thing: an awk that cannot run yields empty output, +# and empty matches neither 'BAD' nor a number, so the caller's comparison merely errors +# into a false condition and execution continues with rc still 0 — the checker reporting +# success without having parsed either checklist. +prompt_checklist_count() { # $1 = file + # `grep -c` exits 1 when the count is ZERO, which is a valid answer here (a file with + # no checklist heading is malformed, not unreadable). Only >=2 is a real error, so the + # status is captured and compared rather than used as a bare `||`. + heads=$(grep -cE '^## Checklist([[:space:]].*)?$' "$1"); heads_st=$? + [ "$heads_st" -le 1 ] || return 2 + [ "$heads" -eq 1 ] || { printf 'BAD'; return 0; } + # Any numbered label inside the section is CONSIDERED, not only canonically-formatted + # ones. Matching `^[0-9]+\. \*\*` as the guard skipped a non-bold `13. item` + # entirely, so appending one to both definitions left N at 12 and let a now-stale + # `all 12 items` claim pass — failing open exactly when the checklist changes. + awk ' + /^## Checklist([[:space:]].*)?$/ { inlist = 1; next } + inlist && /^## / { inlist = 0 } + inlist && /^[0-9]+\./ { + if ($0 !~ /^[1-9][0-9]*\. \*\*/) { bad = 1; next } # 0., leading zero, or non-bold + sub(/\..*/, "", $0); n += 1 + if ($0 "" != n "") bad = 1 # string compare: an oversized label must not overflow + } + END { if (bad || n == 0) print "BAD"; else print n } + ' "$1" || return 2 +} +n_repo=$(prompt_checklist_count docs/prompt-standards.md); st_repo=$? +n_tmpl=$(prompt_checklist_count plugins/dev-workflow/commands/workflow-init.md); st_tmpl=$? +if [ "$st_repo" -ne 0 ] || [ "$st_tmpl" -ne 0 ]; then + fail "Prompt standards: the checklist parser failed; results are not trustworthy." \ + "parser exited $st_repo (repo) / $st_tmpl (template)" +elif [ "$n_repo" = BAD ] || [ "$n_tmpl" = BAD ]; then + fail "Prompt standards: a checklist definition is missing, empty, duplicated or misnumbered." \ + "docs/prompt-standards.md=$n_repo workflow-init.md=$n_tmpl" +elif [ "$n_repo" -ne "$n_tmpl" ]; then + fail "Prompt standards: the repo checklist and the scaffolded template disagree." \ + "docs/prompt-standards.md=$n_repo workflow-init.md=$n_tmpl" +else + # One scan for both spellings, one awk to judge them. Digit comparison is done as + # STRINGS after canonicalisation, never `+0`, so a 40-digit claim cannot overflow its + # way to a wrong verdict. + words='one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty' + # Same status capture as the 4a scan, and for the same reason. + claim_scan=$(grep -rnoE "(^|[^[:alnum:]_])all ([0-9]+|$words)( checklist)? items([^[:alnum:]_]|\$)" \ + --include='*.md' . 2>/dev/null) + claim_scan_st=$? + [ "$claim_scan_st" -le 1 ] || + fail "Prompt standards: the 4b claim scan failed; results are not trustworthy." \ + "grep exited $claim_scan_st" + # Filter and validator each get their own status, for the same reason as 4a's. + claim_filtered=$(printf '%s\n' "$claim_scan" | grep -vE "$PROMPT_EXCL"); claim_filter_st=$? + [ "$claim_filter_st" -le 1 ] || + fail "Prompt standards: the 4b exclusion filter failed; results are not trustworthy." \ + "grep -v exited $claim_filter_st" + bad_claims=$(printf '%s\n' "$claim_filtered" | + awk -v n="$n_repo" -v words="$words" ' + BEGIN { c = split(words, w, "|"); for (i = 1; i <= c; i++) val[w[i]] = i } + { + tok = $0; sub(/.*all /, "", tok); sub(/[^0-9a-zA-Z].*/, "", tok) + if (tok ~ /^[0-9]+$/) { + if (tok !~ /^[1-9][0-9]*$/) { print $0 " <- non-canonical number"; next } + if (tok != n) print $0 " <- checklist has " n + } else if (tok in val) { + if (val[tok] != n + 0) print $0 " <- checklist has " n + } + }'); claim_awk_st=$? + [ "$claim_awk_st" -eq 0 ] || + fail "Prompt standards: the 4b claim validator failed; results are not trustworthy." \ + "awk exited $claim_awk_st" + [ -n "$bad_claims" ] && + fail "Prompt standards: a checklist count claim disagrees with the checklist." "$bad_claims" +fi +# --- END check 4b --- + [ "$rc" -eq 0 ] && printf 'invariant checks: ok\n' exit "$rc" diff --git a/scripts/check-invariants.test.sh b/scripts/check-invariants.test.sh index c7657a1..ed5dbf6 100755 --- a/scripts/check-invariants.test.sh +++ b/scripts/check-invariants.test.sh @@ -12,6 +12,27 @@ pass_n=0; fail_n=0 pass() { pass_n=$((pass_n + 1)); printf 'ok - %s\n' "$1"; } fail() { fail_n=$((fail_n + 1)); printf 'FAIL - %s\n' "$1"; } +# Every fixture repo needs a valid, agreeing pair of checklist definitions, because the +# prompt-conformance check treats a missing/empty/duplicated checklist as malformed and +# fires. Without this, all four fixture builders below would produce repos that fail on +# a missing checklist BEFORE reaching their own invariant-5/6 assertion: accept cases +# turn red and reject cases start passing for the wrong reason — the exact +# diagnostic-isolation failure the $5-substring guard exists to prevent. Measured before +# this existed: 25 of 61 assertions failed. +# +# One initializer, called from all four builders. Extending only `run_with` would leave +# `sh_case` and the two inline blocks broken. +init_prompt_fixtures() { # $1 = fixture repo root + mkdir -p "$1/docs" "$1/plugins/dev-workflow/commands" + for pf in "$1/docs/prompt-standards.md" "$1/plugins/dev-workflow/commands/workflow-init.md"; do + { printf '# Prompt Standards\n\n## Checklist (each item must be verifiably true)\n\n' + i=1 + while [ "$i" -le 12 ]; do printf '%s. **item %s**\n' "$i" "$i"; i=$((i + 1)); done + printf '\n## After\n\nReviewed against all 12 items.\n' + } > "$pf" + done +} + work=$(mktemp -d) || work='' # Abort rather than continue with an empty $work: every path below is built as # "$work/r", so an empty value turns the fixture reset into `rm -rf /r`. @@ -19,7 +40,11 @@ if [ -z "$work" ] || [ ! -d "$work" ]; then printf 'FAIL - could not create a temporary directory; refusing to run\n' >&2 exit 1 fi -trap 'rm -rf "$work"' EXIT +# Restore any mode-000 fixture directory BEFORE removing the tree: the scan-error +# fixture chmods one to 000, and an abort between that and its restore would leave a +# directory this cleanup cannot traverse as a non-root user. +cleanup() { [ -d "$work/r/docs/locked" ] && chmod 755 "$work/r/docs/locked" 2>/dev/null; rm -rf "$work"; } +trap cleanup EXIT HUP INT TERM # Build a minimal repo whose only content is $1 (a workflow file body), run the # checker in it, and report its exit status. @@ -27,6 +52,7 @@ run_with() { # $1 = workflow body, $2 = optional manifest body, $3 = optional .m rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ "$work/r/plugins/p/.claude-plugin" cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" printf '%s\n' "$1" > "$work/r/.github/workflows/ci.yml" # Single-quoted default: inside double quotes `\{` stays a literal backslash, which # produced an invalid-JSON fixture. Harmless against a grep, but a fixture that is @@ -235,6 +261,7 @@ sh_case() { # $1 = script body, $2 = name, $3 = expect_reject? (1/0) rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ "$work/r/plugins/p/.claude-plugin" cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" printf '%s\n' '{"name": "p"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" printf '#!/bin/sh\n%s\n' "$1" > "$work/r/run.sh" @@ -266,6 +293,7 @@ sh_case '# never run npx -y floating-example' "npx inside a shell comment accept rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ "$work/r/plugins/p/.claude-plugin" cp "$CHECKER" "$work/r/scripts/" +init_prompt_fixtures "$work/r" printf '%s\n' '{"name": "p"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" printf '#!/bin/sh\nnpx -y floating-extra\n' > "$work/r/scripts/check-invariants-extra.sh" @@ -282,6 +310,7 @@ for badname in 'a|b' 'a&b' 'a\b'; do rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ "$work/r/plugins/p/.claude-plugin" cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" printf '%s\n' '{"name": "p"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" printf 'jobs:\n q:\n steps:\n - uses: actions/setup-node@v4\n' \ @@ -293,6 +322,309 @@ for badname in 'a|b' 'a&b' 'a\b'; do else pass "violation in filename '$badname.yml' still rejected"; fi done +# --- Prompt conformance: checks 4a and 4b ------------------------------------------ +# +# MUTATION EVIDENCE (recorded 2026-07-26). This is a DOCUMENTED DEVELOPMENT-TIME RUN, +# not automated enforcement: nothing re-runs it, and nothing here fails if it goes stale. +# The procedure lives in the checker's header comment. +# +# Deleting the `BEGIN check 4a`/`END check 4a` block flipped exactly 20 assertions +# (baseline exit 0, mutant exit 1): every `4a:` reject fixture (15), the four +# `exclusion: neighbouring …` controls, which depend on 4a because they carry the +# assertion phrase, and `4a value extraction failure fires`, whose stage 4a alone reaches. +# Deleting the `4b` block flipped exactly 22: every `4b:` reject fixture (16), the four +# `4b exclusion: neighbouring …` controls, and the two stage-failure fixtures that reach +# their stage only through 4b — `checklist parser failure fires` and +# `4b claim validator failure fires`, since deleting the block means neither the parser +# nor the validator is ever called. +# Two fixtures flip in NEITHER mutation, and that is the expected result rather than an +# omission: `scan error fires` and `4a/4b exclusion filter failure fires` break a stage +# that BOTH checks use, so the surviving check still fires when its sibling is deleted. +# In both runs no accept case moved and no unrelated case moved — the second half of the +# check, and the one a non-empty flip set alone does not establish. +# +# An earlier version of this block recorded 17 and 11. Those were true when written and +# went stale the moment fixtures were added — which is exactly the failure the trigger +# below exists to prevent, and it was caught at Gate B rather than by the trigger. If you +# add a fixture, you are changing this mapping. +# +# RE-RUN TRIGGER — broader than "the scan logic", because the mapping above is +# invalidated by more than that: re-run and update BOTH this block and the PR record +# after changing either marked check, its markers, any of these fixtures or their +# assertion names, or the harness that runs them. +# Diagnostics these cases must name, so none can pass on an unrelated violation. +MODEL='name one executing model' +CLAIM='count claim disagrees' +DEFN='checklist definition is missing' +MISMATCH='scaffolded template disagree' + +# Build a fixture repo, drop $4 at path $3, run the checker. $2=1 expects rejection. +prompt_case() { # $1 = name, $2 = 1|0, $3 = relative path, $4 = body, $5 = diagnostic + rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ + "$work/r/plugins/p/.claude-plugin" + cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" + printf '%s\n' '{"name": "p", "version": "1.0.0"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" + printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" + mkdir -p "$work/r/$(dirname "$3")" + printf '%s\n' "$4" > "$work/r/$3" + out=$( cd "$work/r" && sh scripts/check-invariants.sh 2>&1 ); st=$? + if [ "$2" -eq 1 ]; then + if [ "$st" -eq 0 ]; then fail "$1 (exited 0)" + elif ! printf '%s' "$out" | grep -q "$5"; then + fail "$1 (wrong diagnostic: $(printf '%s' "$out" | tr '\n' ' '))" + else pass "$1"; fi + else + if [ "$st" -eq 0 ]; then pass "$1" + else fail "$1 ($(printf '%s' "$out" | tr '\n' ' '))"; fi + fi +} + +# 4a — a file claiming conformance must name exactly one executing model. +A='prompt artifact and follows docs/prompt-standards.md' +prompt_case "4a: no Target model line rejected" 1 docs/a.md "$A" "$MODEL" +prompt_case "4a: two declarations rejected" 1 docs/a.md \ + "Target model: Claude via Claude Code +Target model: Codex via mcp__codex__* +$A" "$MODEL" +prompt_case "4a: a valid plus an empty declaration rejected" 1 docs/a.md \ + "Target model: Claude via Claude Code +Target model: +$A" "$MODEL" +prompt_case "4a: empty value rejected" 1 docs/a.md "Target model: +$A" "$MODEL" +# The exact PR #12 defect: names a model CLASS, and mentions Claude only as provenance. +prompt_case "4a: unnamed model naming Claude only as provenance rejected" 1 docs/a.md \ + "Target model: any capable chat model (developed with Claude as the sparring partner) +$A" "$MODEL" +# Token prefixes are not the token. Without a portable boundary these all pass. +prompt_case "4a: token-prefix ClaudeX rejected" 1 docs/a.md "Target model: ClaudeX +$A" "$MODEL" +prompt_case "4a: token-prefix Codex2 rejected" 1 docs/a.md "Target model: Codex2 +$A" "$MODEL" +prompt_case "4a: token-prefix GPTfoo rejected" 1 docs/a.md "Target model: GPTfoo +$A" "$MODEL" +# Two distinct models, one per separator: the verdict is separator-independent. +prompt_case "4a: two models via or rejected" 1 docs/a.md "Target model: Claude or Codex +$A" "$MODEL" +prompt_case "4a: two models via slash rejected" 1 docs/a.md "Target model: GPT/Codex +$A" "$MODEL" +prompt_case "4a: two models via comma rejected" 1 docs/a.md "Target model: Codex, Claude +$A" "$MODEL" +prompt_case "4a: two models via and rejected" 1 docs/a.md "Target model: Claude and Codex +$A" "$MODEL" +prompt_case "4a: repeated same token accepted" 0 docs/a.md \ + "Target model: Claude via Claude Code +$A" "" +prompt_case "4a: chat-interface form accepted" 0 docs/a.md \ + "Target model: Claude in a chat interface, upstream of Claude Code +$A" "" +prompt_case "4a: one model with two surfaces accepted" 0 docs/a.md \ + "Target model: Claude via Claude Code or the API +$A" "" +# The settled boundary: a token PREFIX in second position is not a second model. +prompt_case "4a: second-position token prefix accepted" 0 docs/a.md \ + "Target model: Claude or Codex2 +$A" "" +prompt_case "4a: indented declaration does not count" 1 docs/a.md " Target model: Claude +$A" "$MODEL" +prompt_case "4a: blockquoted declaration does not count" 1 docs/a.md "> Target model: Claude via Claude Code +$A" "$MODEL" +prompt_case "4a: mid-sentence mention does not count" 1 docs/a.md \ + "The Target model: field is set elsewhere. +$A" "$MODEL" +prompt_case "4a: file without the assertion is not scanned" 0 docs/a.md \ + "just prose, no conformance claim" "" + +# 4b — a prose count claim must equal the checklist it counts (N = 12 in fixtures). +prompt_case "4b: wrong digit claim rejected" 1 docs/c.md "all 11 items" "$CLAIM" +# The motivating 2026-07-25 occurrence was the WORD form. +prompt_case "4b: word-form claim disagreeing rejected" 1 docs/c.md "all ten items" "$CLAIM" +# LESSON LOCK: a canonical-only grammar makes malformed input invisible, not rejected. +prompt_case "4b: non-canonical 012 rejected as malformed" 1 docs/c.md "all 012 items" "$CLAIM" +prompt_case "4b: 40-digit claim rejected" 1 docs/c.md \ + "all 1234567890123456789012345678901234567890 items" "$CLAIM" +prompt_case "4b: correct digit claim accepted" 0 docs/c.md "all 12 items" "" +prompt_case "4b: correct word claim accepted" 0 docs/c.md "all twelve items" "" +prompt_case "4b: two agreeing claims accepted" 0 docs/c.md \ + "all 12 items and later all 12 checklist items" "" +# Outer boundaries: without them these read as claims and reject unrelated prose. +prompt_case "4b: embedded-prefix near-miss accepted" 0 docs/c.md "small ten items" "" +prompt_case "4b: embedded-suffix near-miss accepted" 0 docs/c.md "all ten itemsized" "" + +# 4b — malformed checklist DEFINITIONS fire rather than comparing 0 to 0. +prompt_case "4b: absent checklist section rejected" 1 docs/prompt-standards.md \ + "# X +no checklist here" "$DEFN" +prompt_case "4b: empty checklist section rejected" 1 docs/prompt-standards.md \ + "## Checklist (each item must be verifiably true) + +## After" "$DEFN" +prompt_case "4b: duplicate Checklist heading rejected" 1 docs/prompt-standards.md \ + "## Checklist (a) +1. **x** +## Checklist (b) +2. **y**" "$DEFN" +prompt_case "4b: non-contiguous labels rejected" 1 docs/prompt-standards.md \ + "## Checklist +1. **a** +1. **b** +3. **c** +## After" "$DEFN" +prompt_case "4b: leading-zero label rejected" 1 docs/prompt-standards.md \ + "## Checklist +01. **a** +## After" "$DEFN" +prompt_case "4b: near-miss heading ## Checklists is not a definition" 1 docs/prompt-standards.md \ + "## Checklists +1. **a** +## After" "$DEFN" +prompt_case "4b: near-miss heading ### Checklist is not a definition" 1 docs/prompt-standards.md \ + "### Checklist +1. **a** +## After" "$DEFN" +prompt_case "4b: near-miss heading ## Checklist-ish is not a definition" 1 docs/prompt-standards.md \ + "## Checklist-ish +1. **a** +## After" "$DEFN" +prompt_case "4b: zero label rejected" 1 docs/prompt-standards.md \ + "## Checklist +0. **a** +## After" "$DEFN" +# A numbered line that is not a canonical item must make the definition BAD, not be +# skipped: skipping it left N unchanged, so adding an item to BOTH definitions kept a +# now-stale count claim passing — a fail-open path exactly when the checklist changes. +prompt_case "4b: trailing non-bold numbered item rejected" 1 docs/prompt-standards.md \ + "## Checklist +1. **a** +2. item two +## After" "$DEFN" +prompt_case "4b: interspersed non-bold numbered item rejected" 1 docs/prompt-standards.md \ + "## Checklist +1. **a** +2. plain +3. **c** +## After" "$DEFN" +prompt_case "4b: definitions disagreeing rejected" 1 docs/prompt-standards.md \ + "## Checklist +1. **a** +## After" "$MISMATCH" + +# Exclusions. Each pair uses the SAME violating content inside and outside the excluded +# path, which is what proves the exclusion is load-bearing rather than an overbroad +# filter — or a fixture that never matched the rule at all. +prompt_case "exclusion: source-files/ accepted" 0 source-files/x.md "$A" "" +prompt_case "exclusion: neighbouring source-filesX/ still scanned" 1 source-filesX/x.md "$A" "$MODEL" +prompt_case "exclusion: docs/superpowers/ accepted" 0 docs/superpowers/x.md "$A" "" +prompt_case "exclusion: neighbouring docs/superpowersX/ still scanned" 1 docs/superpowersX/x.md "$A" "$MODEL" +prompt_case "exclusion: .context/ accepted" 0 .context/x.md "$A" "" +prompt_case "exclusion: neighbouring contextX/ still scanned" 1 contextX/x.md "$A" "$MODEL" +# LESSON LOCK: the ledger QUOTES defects, so scanning it self-rejects forever. It carries +# both a quoted assertion phrase and the historical `all ten items`. +prompt_case "exclusion: hardening-log.md quoting both defects accepted" 0 docs/hardening-log.md \ + "| 2026-07-25 | docs-drift | said \"all ten items\" while $A | bot | minor | 2 lint | x |" "" +prompt_case "exclusion: neighbouring hardening-log-notes.md still scanned" 1 docs/hardening-log-notes.md \ + "$A" "$MODEL" + +# The pairs above carry 4a's assertion phrase and therefore prove only 4a's filtering. +# A broken or overbroad 4b exclusion would ship with every other assertion green, so +# each excluded path gets a second pair carrying a DISAGREEING count claim instead. +# The hardening-log pair matters most: 4b scanning the ledger is what would reject the +# real repository forever, since the ledger quotes `all ten items` as evidence. +D='all ten items' +prompt_case "4b exclusion: source-files/ accepted" 0 source-files/n.md "$D" "" +prompt_case "4b exclusion: neighbouring source-filesX/ still scanned" 1 source-filesX/n.md "$D" "$CLAIM" +prompt_case "4b exclusion: docs/superpowers/ accepted" 0 docs/superpowers/n.md "$D" "" +prompt_case "4b exclusion: neighbouring docs/superpowersX/ still scanned" 1 docs/superpowersX/n.md "$D" "$CLAIM" +prompt_case "4b exclusion: .context/ accepted" 0 .context/n.md "$D" "" +prompt_case "4b exclusion: neighbouring contextX/ still scanned" 1 contextX/n.md "$D" "$CLAIM" +prompt_case "4b exclusion: hardening-log.md quoting a stale count accepted" 0 docs/hardening-log.md \ + "| 2026-07-25 | docs-drift | said \"$D\" while its checklist ran 1-12 | bot | minor | pending | x |" "" +prompt_case "4b exclusion: neighbouring hardening-log-notes.md still scanned" 1 docs/hardening-log-notes.md \ + "$D" "$CLAIM" + +# A scan that ERRORS must not read as "no offenders". grep exits >=2 on a traversal +# failure, and piping it straight into the exclusion filter would report the filter's +# status instead — the checker would print success without having looked. +# Skipped as root, where an unreadable directory is still readable and the fixture +# cannot produce the condition it tests. +if [ "$(id -u)" -eq 0 ]; then + pass "scan error fires (skipped: running as root)" +else + rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ + "$work/r/plugins/p/.claude-plugin" + cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" + printf '%s\n' '{"name": "p", "version": "1.0.0"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" + printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" + mkdir -p "$work/r/docs/locked"; printf 'x\n' > "$work/r/docs/locked/x.md" + chmod 000 "$work/r/docs/locked" + out=$( cd "$work/r" && sh scripts/check-invariants.sh 2>&1 ); st=$? + chmod 755 "$work/r/docs/locked" + if [ "$st" -eq 0 ]; then fail "scan error fires (exited 0 - scan failure read as clean)" + elif ! printf '%s' "$out" | grep -q 'scan failed'; then + fail "scan error fires (wrong diagnostic: $(printf '%s' "$out" | tr '\n' ' '))" + else pass "scan error fires"; fi +fi + +# A parser that cannot RUN must fire, not fall through. With awk broken, the count +# substitution yields empty, which matches neither 'BAD' nor a number, so the comparison +# merely errors into a false condition and execution continues with rc still 0 — the +# checker reporting success without having parsed either checklist. Injected via PATH +# because the checker calls `awk` unqualified. +rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ + "$work/r/plugins/p/.claude-plugin" "$work/r/fakebin" +cp "$CHECKER" "$work/r/scripts/" +init_prompt_fixtures "$work/r" +printf '%s\n' '{"name": "p", "version": "1.0.0"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" +printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" +printf '#!/bin/sh\nexit 2\n' > "$work/r/fakebin/awk" +chmod +x "$work/r/fakebin/awk" +out=$( cd "$work/r" && PATH="$work/r/fakebin:$PATH" sh scripts/check-invariants.sh 2>&1 ); st=$? +if [ "$st" -eq 0 ]; then fail "checklist parser failure fires (exited 0 - parser failure read as clean)" +elif ! printf '%s' "$out" | grep -q 'parser failed'; then + fail "checklist parser failure fires (wrong diagnostic: $(printf '%s' "$out" | tr '\n' ' '))" +else pass "checklist parser failure fires"; fi + +# Selective PATH wrappers. The fake `awk` above exits on the FIRST call, which is the +# checklist parser, so execution never reaches the claim validator or the exclusion +# filters — meaning those newly-checked statuses had no fixture and could be deleted +# while the suite stayed green. Each wrapper below delegates to the real tool and fails +# only for the one invocation under test, identified by an argument unique to it. +inject_case() { # $1 = name, $2 = tool, $3 = match-arg, $4 = expected diagnostic + rm -rf "$work/r"; mkdir -p "$work/r/scripts" "$work/r/.github/workflows" \ + "$work/r/plugins/p/.claude-plugin" "$work/r/fakebin" + cp "$CHECKER" "$work/r/scripts/" + init_prompt_fixtures "$work/r" + printf '%s\n' '{"name": "p", "version": "1.0.0"}' > "$work/r/plugins/p/.claude-plugin/plugin.json" + printf '%s\n' "$PINNED" > "$work/r/.github/workflows/ci.yml" + # A VALID claiming file, so check 4a's per-file path actually executes. Without one + # the 4a scan matches nothing, the loop body never runs, and a wrapper aimed at the + # extraction step would never be reached — the fixture would pass by not testing. + printf 'Target model: Claude via Claude Code\nprompt artifact and follows docs/prompt-standards.md\n' \ + > "$work/r/docs/claiming.md" + real=$(command -v "$2") + { printf '#!/bin/sh\n' + # shellcheck disable=SC2016 # deliberate: $@ must stay literal in the GENERATED script + printf 'for a in "$@"; do case "$a" in %s) exit 2 ;; esac; done\n' "$3" + printf 'exec %s "$@"\n' "$real" + } > "$work/r/fakebin/$2" + chmod +x "$work/r/fakebin/$2" + out=$( cd "$work/r" && PATH="$work/r/fakebin:$PATH" sh scripts/check-invariants.sh 2>&1 ); st=$? + if [ "$st" -eq 0 ]; then fail "$1 (exited 0 - stage failure read as clean)" + elif ! printf '%s' "$out" | grep -q "$4"; then + fail "$1 (wrong diagnostic: $(printf '%s' "$out" | tr '\n' ' '))" + else pass "$1"; fi +} +# `-vE` is used only by the two exclusion filters; invariant 5's filter uses plain -v. +inject_case "4a/4b exclusion filter failure fires" grep '-vE' 'exclusion filter failed' +# `-v words=...` is unique to the 4b claim validator; the checklist parser takes none. +inject_case "4b claim validator failure fires" awk 'words=*' 'claim validator failed' +# The 4a value extraction is identified by a marker comment inside its awk program, so +# this wrapper fails ONLY that invocation and not the parser or the claim validator. +inject_case "4a value extraction failure fires" awk '*extract-target-model*' \ + 'per-file checks failed' + printf '\n---\n' if [ "$fail_n" -eq 0 ]; then printf 'all passed (%s assertions)\n' "$pass_n"; else printf '%s passed, %s FAILED\n' "$pass_n" "$fail_n"; exit 1 diff --git a/todos.md b/todos.md index ab7a8cd..54ab35e 100644 --- a/todos.md +++ b/todos.md @@ -193,18 +193,23 @@ backlog. extra release cost. Re-raised by CodeRabbit on PR #12 (Major) and kept deferred there on the same reasoning; the scaffolded copy carries no false claim, only one paragraph less. -- [ ] **Prompt-standards conformance checker — resolves two `pending` ledger rows - (2026-07-25).** Extend `scripts/check-invariants.sh` (+ its regression suite, - mutation-verified) with the mechanical rung both classes have never reached: - every file asserting it follows `docs/prompt-standards.md` must carry a - `Target model:` line, and a prose count of the checklist must match the - number of items actually in it. **Resolution vehicle: the same upcoming - canvas-findings hardening round** — it fires full Gate B anyway on its - `commands/` paths, so the ~8-line checker plus tests land at proportionate - cost instead of turning a docs PR into a code PR. Resolve the two `pending` - rows by **appending** rung-2 rows referencing them (2026-07-25 `docs-drift` - and 2026-07-25 `unverified-enforcement-claim`) — never by editing the - pending rows. *Trigger: that round starting.* +- [x] **Prompt-standards conformance checker — resolved the two `pending` ledger rows + (2026-07-25).** Landed as checks 4a and 4b in `scripts/check-invariants.sh`, with + fixtures in its regression suite, in the canvas-findings round's **PR 1**. (No + fixture count is quoted here on purpose: an earlier draft said "42", the Gate-B + fixes took it past that, and a hard-coded total at a doc site describing this + checker is the very drift class the checker exists for.) Both `pending` + rows were resolved by appending rung-2 rows dated 2026-07-26, never edited. Each + new row states the exact spelling its check guards and what stays + instruction-backed. Two things turned out differently than this entry assumed: + word forms had to be in scope, because the motivating `docs-drift` occurrence + spelled its count as a **word** rather than a digit, and a digit-only check would + have missed it entirely. (The exact phrase is deliberately not quoted here: check + 4b reads a live count claim in any scanned `*.md`, and it caught this very entry + when it was first written. The ledger may quote it because the ledger is excluded; + editable prose should reword instead.) And the + ledger itself had to be **excluded** from both checks, because a ledger that + quotes defects self-rejects the checks that detect them. - [ ] **Escalation trigger for the invariant checker — read this before patching it.** The checker asserts only the spellings its fixtures cover. Adding one more regex arm per newly-discovered spelling is *not* the ladder working; it is the same From 92de0d2ca2bf50b0806cb22f505b31f56cfe5e1e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Daniel=20S=C3=A4nger?= <20968534+dsnger@users.noreply.github.com> Date: Sun, 26 Jul 2026 14:20:16 +0200 Subject: [PATCH 2/2] Force the string compare in 4b's digit branch, matching its sibling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit nitpick on #13: `prompt_checklist_count` writes `$0 "" != n ""` to force a string compare, and the header credits that idiom for keeping a long digit run from overflowing into a wrong verdict — but the claim validator did the equivalent comparison as a bare `tok != n`. Validated by hand rather than by subagent: the PR touches AGENTS.md, so process-pr-review's instruction-path precheck bars triage. The claim is true as a consistency point and not a live bug — `n` arrives via `-v` (a strnum) and `tok` comes from `sub()` (a string), so POSIX already resolves it as a string compare, and the 40-digit fixture passes either way on BSD awk 20200816. The point stands that it relied on type inference the rest of the file deliberately does not. Gate B triviality skip, documented per CLAUDE.md §5: one-token, behaviour- preserving change inside an existing awk expression, covered by the existing `4b: 40-digit claim rejected` and `4b: non-canonical 012 rejected` fixtures. Not hardened: no ledger class fits a style-consistency nit whose behaviour was already correct, and minting one would dilute the recurrence signal. (First attempt at this comment broke the script — an apostrophe in `awk's` terminated the single-quoted awk program. The suite caught it; the comment now says so.) --- scripts/check-invariants.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/scripts/check-invariants.sh b/scripts/check-invariants.sh index bbd5808..dfcc7ab 100755 --- a/scripts/check-invariants.sh +++ b/scripts/check-invariants.sh @@ -445,7 +445,13 @@ else tok = $0; sub(/.*all /, "", tok); sub(/[^0-9a-zA-Z].*/, "", tok) if (tok ~ /^[0-9]+$/) { if (tok !~ /^[1-9][0-9]*$/) { print $0 " <- non-canonical number"; next } - if (tok != n) print $0 " <- checklist has " n + # Same forced-string idiom as prompt_checklist_count above. n arrives via -v, + # which makes it a strnum, so a bare tok != n leans on awk type inference to + # stay a string compare. It does today on every awk tested, but the header + # credits this idiom precisely so overflow on a long digit run cannot depend + # on that inference. (No apostrophes in here: this program is inside a + # single-quoted shell string, and one terminated it.) + if (tok "" != n "") print $0 " <- checklist has " n } else if (tok in val) { if (val[tok] != n + 0) print $0 " <- checklist has " n }