Study: pre-push verification harness for autonomous code evolution #152

Intake phase 2 (study & framing), per ADR-0014. This body is the qualified spec and states present state (refreshed 2026-08-18; GitHub keeps the edit history). Staged go 1–2: PR 1a is merged (#160), PR 2 is merged (#167) — its decision reversed in review, see P1 — PRs 3–5 stay behind a second gate; PR 4 got its go 2026-09-19 (ordered first) and is delivered, PRs 3 and 5 remain gated.
2026-09-20 (review binding): four oracle/CI candidates joined P2's remaining list; P3.3's comment-permanence claim is narrowed in place.
2026-09-21: the key-expiry candidate is broadened to every key under security/ and is delivering; one new candidate joined P2.

Problem statement

The repository has a mature governance layer (AGENTS.md, docs/conventions.md, the ADRs, the #106 status SSOT, the ADR-0014 intake pipeline) and a post-publication outcome oracle (verify_release, #149). This study specs the layer between the two: a pre-push verification harness that lets an agent, or a human, prove a change is good before CI is asked, plus the deterministic capabilities that produce such changes.

Its first two passes are live (ADR-0016 #160, ADR-0020 #167). What remains: an enforced floor under the ADR-0012 autonomy (PR 3), deterministic structured writes (PR 4), and the oracle's later passes.

Current state

The three problems

P1. The PR gate — cost question closed, coverage question answered the other way

This section is rewritten (2026-08-18). Its original framing was overtaken by its own dependencies, and the conclusion it proposed was rejected in review.

The study opened P1 as a cost problem: build-test ran, per version combination, an amd64 build plus a full multi-arch build. Raw cost fell by an order of magnitude from outside this study's scope: 16 × (1 + 4) ≈ 80 platform builds → 3 × (1 + 2) = 9, via ADR-0015 (#158) and ADR-0019 (#164). #150 (GHA cache limits) was a symptom of the old size.

By the time PR 2 was authored, the cost that justified trimming was therefore already gone. The proposal — a representative subset plus one multi-arch canary — was implemented, measured, and declined:

• measured 14.2 → 8.6 runner-minutes at an unchanged wall clock, because the canary is the critical path and the multi-arch builds it replaced already ran in parallel;
• runner minutes are free on a public repository, so the saving was cache churn only;
• it was paid for in arm64 coverage — the wrong currency, because of what the measurement surfaced next.

The finding that redirected the decision: nothing in CI had ever executed an arm64 image. build-test ran container-structure-test on amd64 only; push-edge and release-please build multi-arch and test nothing. A green build proves the image assembles, not that it works — which is how #161 shipped a non-executable aws for years behind green multi-arch builds, found by a human running --full on Apple Silicon rather than by any gate.

ADR-0020 records the opposite of the original proposal and is merged in #167: the gate builds and structure-tests every supported combination on every published platform, one job per pair, arm64 under emulation. The emulated assertions are proven to execute (aws-cli/2.36.6 … exe/aarch64.debian.13). Cost rose (~14 → ~20 runner-minutes, comparable wall clock); the blind spot closed. Bounded: a second AWS CLI line takes the gate to twelve jobs, the trigger to revisit deliberately.

P2. The pre-push oracle is live but partial

Scope is bounded by the ADR-0016 principle: the oracle checks only invariants no purpose-built tool covers, and calls tools rather than re-implementing them.

Live (#160): supported_versions.json ↔ the files in security/**, both directions, extended to both architectures by ADR-0017; ADR index ↔ the files in docs/adr/.

Remaining candidates:

• version policy: semver, sorted, one patch per minor, within the ADR-0015 EOL window
• workflow paths: filters against files that actually exist
• the L5 dash rule
• HashiCorp GPG key expiry — delivering in ci(validate): fail on an expired key in security/ #211, and broadened in the process: the candidate named one key, the check covers every key under security/, because awscliv2.asc turned out to have been expired since 2023-09-17 as well. Root cause recorded there and in ADR-0016 (amended): gpg --verify exits 0 for a good signature made by an expired key, so neither lapse was visible to anything. Blocked on the AWS key itself, which no host reachable from a session serves (see fix(security): refresh the expired vendor signing keys #210)
• (added 2026-09-21, found while writing the above) a failing check truncates --fast: every check function ends on [ "$ok" = 1 ] && pass …, which returns 1 under set -euo pipefail and aborts run_fast, so the first failure swallows every later check and the validate: FAILED line never prints. Exit codes are unaffected, so no gate is wrong — the output is just far less useful than it looks, and an agent reading it sees one finding where there may be four. Fix is one line per check; it touches them all, hence a pass of its own
• one publisher per declared tag — ADR-0018 defines the matrix, the check asserts no second writer appears
• the published architectures agreeing across workflows — the shapes differ: push-edge.yml and release-please.yml carry the literal linux/amd64,linux/arm64, build-test.yml carries ["amd64","arm64"] and composes linux/<arch> per job (ADR-0020)
• documentation coherence: resolvable @-imports, resolvable relative links, every docs/ file claimed exactly once by the documentation map, no rule defined outside docs/conventions.md. The oracle owns the checking; Study: documentation single-source-of-truth map and agent context loading #170 owns the map they check, so this pass lands after it
• --simulate-publish, learnings graduation, and seam 2 from the CI-duplication review
• (added 2026-09-20, review binding) shellcheck on scripts/**, local-if-present in --fast (the hadolint pattern) — two real bug classes hit on 2026-09-19
• (added 2026-09-20) one container-structure-test path: the PR gate calls the oracle's CST step instead of the third-party plexsystems action — two version sources today, and the only third-party action on the gate
• (added 2026-09-20) one hadolint path: fold lint-dockerfile.yml into validate.yml (action-bundled hadolint vs the oracle's pinned 2.12.0)
• (added 2026-09-20) a pre-agreed split criterion for validate.sh (e.g. ~600 lines → scripts/lib/), decided before Monorepo consolidation — merge terraform-aws-cli + terraform-azure-cli (draft plan) #131 needs scripts/ product-agnostic

Delivered by PR 2 (#167): seam 1 — rendering the container-structure-test config moved into validate.sh --render-tests, called by the workflow.

Deliberately not an oracle check: Dockerfile apt pins against the test-template assertions — container-structure-test owns detection, and PR 4's atomic writes remove the drift class at the source.

P3. Autonomy without an enforced floor

Three defects now, the third with evidence:

1. Every session rule (never merge a PR, never delete a branch or tag, never force push or amend, never bypass hooks, never push master) exists only as prose in AGENTS.md and docs/conventions.md. Nothing blocks the action.
2. agentRoles in .claude/settings.json is inert. It is not a Claude Code settings key and nothing reads it. ADR-0006 states that swapping a model is a one line config edit; physically, that mechanism does not exist. The real vehicle is .claude/agents/*.md with model: in frontmatter.
3. Conventions L1 (English only for repo-facing content) has been violated twice by agent sessions, both times while the exchange with the maintainer was in French: a full comment on ci: lint commits and PR title with commitlint (Phase 1) #121 (2026-06-18) and a comment on ci(build-test): test every combination on every published platform #167 (2026-08-18). A repository-wide scan on 2026-08-18 found the tracked files and the entire commit history clean; the French is confined to GitHub artefacts — those two comments, plus the pre-conventions pull-request titles of 2026-04-22 (ci: conventional commits et semantic release (epic #101) #107–ci: amélioration des workflows - concurrency, pr triggers, cache (epic #103) #114). The rule existed both times and stopped nothing, which is this problem in one sentence. Two aggravating facts: no CI gate reads pull-request comment bodies, and a session cannot edit or delete a comment (no such tool in the MCP surface), so a slip is permanent and can only be superseded. (Narrowed 2026-09-20: the MCP surface now carries update_issue_comment — a session can edit its own comments; others' comments, deletion, and the enforcement gap stand.) Maintainer decision 2026-08-18: the historical artefacts are left as they are — closed, never merged, and written before the conventions existed. The work is forward-looking only.

Target architecture

Four verification tiers, one script, docked into ADR-0014 phase 5 (realisation). The tier numbering is this study's shorthand and is defined here; ADR-0016 deliberately does not use it.

ADR-0014 intake:  1 dedup -> 2 go/nogo -> 3 prioritise -> 4 plan -> 5 REALISATION
                                                                        |
                    +---------------------------------------------------+
                    v
   T0  scripts/validate.sh --fast    no Docker, <10s    human, agent loop, every PR
   T1  scripts/validate.sh --full    Docker, 1 combo    before push
   T2  CI build-test                 full matrix x published arches, built and tested
   T3  CI release-please             publication + verify_release (authoritative)
                    |
   +----------------+--------------+
   |  capabilities (scripts/)      |   guardrails             feedback
   |  bump-version, refresh-pins   |   PreToolUse deny-list   L4/L5 graduation
   |  skills are thin wrappers     |   real .claude/agents/   simulate-then-fix
   +-------------------------------+

Core invariant: one oracle, three callers — and one entry point. scripts/validate.sh is the same script whether run by the maintainer, by an agent mid-loop, or by the validate.yml job.

Verification tiers

Acceptance criteria

The harness is working, not merely merged, when:

• T0 discipline: validate.sh --fast runs in under ~10s without Docker and is always green on master.
• Drift classes retired: zero recurrences in the following quarter — versions↔security/** and ADR-index via the oracle; pins↔template via its owners.
• Gate honesty (after PR 2, met): a green build-test means every published combination was built and executed on every published architecture.
• Guardrails proven (after PR 3): every NEVER in AGENTS.md, and conventions L1, have a deny rule with a test demonstrating the block fires, on both the Bash and MCP paths.

Automation ladder, and the arbitration of #20

#20 is therefore not an alternative to a bump script, it is its trigger.

Guardrails

Each binding NEVER becomes a PreToolUse deny rule in a single .claude/hooks/deny-destructive.sh:

Bash patterns alone miss the real path. Agents act through the GitHub MCP tools as much as through Bash: a gh pr merge pattern never fires when the call is mcp__github__merge_pull_request. The deny layer needs two matchers: the Bash patterns above, plus PreToolUse rules on the MCP tool names themselves — at minimum mcp__github__merge_pull_request, mcp__github__enable_pr_auto_merge, and any branch/ref deletion or force-update API call.

A third matcher, on content, not on verbs — maintainer decision 2026-08-18, from P3.3. The same hook gains a predicate over the body of outbound GitHub text (mcp__github__add_issue_comment, issue_write, create_pull_request, update_pull_request, add_reply_to_pull_request_comment, pull_request_review_write) and over git commit messages: French accented letters, or French stop-words on word boundaries, denies the call with "rewrite in English before sending". Two signals required, to keep accented proper nouns from tripping it. This is the only mechanism that can enforce L1 — no CI gate reads comment bodies, and prose has now failed twice.

Escape hatch for every rule: overrides live only in settings.local.json (git-ignored, human-owned), so a misfiring rule can never lock the maintainer out. Limit to state in the ADR: hooks bind Claude Code sessions only (the ADR-0009 adapter layer); another tool's adapter must carry its own equivalent, and the agent-agnostic obligation stays in AGENTS.md.

Separately: remove the inert agentRoles block and create real .claude/agents/{orchestrator,executor,reviewer}.md carrying model: in frontmatter — an amendment to ADR-0006 rather than a new decision.

Session lifecycle: plan before edit

Proposed session rule (AGENTS.md, tier-1 per the awareness-tiers doctrine in #106): non-trivial work begins with an explored, stated plan before the first edit; iterating an open PR to green needs no new plan while the fix stays within the PR's declared scope. The escape hatch is unchanged: an ambiguous, architecturally significant or out-of-scope fix exits the delivery loop and returns to planning.

Rejected option: a global defaultMode: "plan" in .claude/settings.json. Plan mode is a per-lifecycle-phase discipline, not a per-session default — a static default would stall every ADR-0012 CI-fix session in plan mode.

Ships with PR 3, carried by an amendment to ADR-0012.

Feedback loop (closing it)

Capture stays manual and cheap (L4: session learnings land in the #106 body). The missing half is graduation: nothing triggers it, nothing owns it, no target is defined. Four additions close the loop:

1. Graduation rule: a learning graduates when it recurs or is binary; preferred target order oracle check > conventions / AGENTS.md rule > ADR amendment > skill. The highest reachable target wins — a check never needs to be remembered.
2. Owner — a PR 1 definition of done: ripe 📌 Tracking & resume — modernization + agent framework #106 learnings ship as part of the oracle. The three writing learnings share one root and are candidates for a single docs/conventions.md L-rule.
3. Cadence — mechanical nudge only: the Phase 7 monthly health-check workflow flags 📌 Tracking & resume — modernization + agent framework #106 learnings older than about a month as "graduate or drop". The human decides what each becomes.
4. Capture rule — bind the input: a surprising failure ships with a learnings entry in the 📌 Tracking & resume — modernization + agent framework #106 body, in the same gesture as the fix.

Candidate rule surfaced by PR 2 (2026-08-18): a queued item restates why it is still worth doing before it is delivered, not only what it does. P1 is the case study.

Counter-example that sizes the loop (2026-08-18): the L1 violation graduated straight past prose — the learning existed, in this very repository, and the second occurrence happened anyway. When a learning is binary and machine-checkable, prose is not a target; it goes to the highest rung or it does not count as graduated.

Proposed sequencing

Order matters. A skill written before the oracle exists re-implements logic in prose, which is exactly what conventions L2 forbids.

Decisions requiring an ADR

Open findings

Tracking and upkeep

• ADR numbers are allocated at write time — claimed at merge, not at authoring.
• Keep the 📌 Tracking & resume — modernization + agent framework #106 body in sync (conventions D5) — including when a decision here reverses, not only when a PR opens or merges.

Next step

1. Second gate decision on PRs 3 and 5 — PR 4 is delivered; PR 3 carries the L1 enforcement, which has a live defect behind it (P3.3).
2. Oracle later passes (the P2 remaining candidates + seam 2); the documentation-coherence pass waits on Study: documentation single-source-of-truth map and agent context loading #170.