From 862dcf03c477ddc93df83dc668abbea0f92f7fac Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 17 Aug 2026 21:31:26 +0100 Subject: [PATCH 1/4] docs(ci): plan semantic proof quality --- .agent-loop/CURRENT_STATE.md | 1 + .../CHUNK_MAP.md | 11 ++ .../DECISIONS.md | 58 +++++++ .../DISCOVERY.md | 110 +++++++++++++ .../INTENT.md | 106 ++++++++++++ .../WS-CI-005-semantic-proof-quality/PLAN.md | 154 ++++++++++++++++++ .../WS-CI-005-semantic-proof-quality/RISKS.md | 15 ++ .../STATUS.md | 26 +++ .../WS-CI-005-01-proof-strength-foundation.md | 79 +++++++++ .../WS-CI-005-02-reviewer-proof-adoption.md | 77 +++++++++ .../WS-CI-005-03-escaped-defect-evaluation.md | 91 +++++++++++ .../WS-CI-005-PLAN-semantic-proof-quality.md | 52 ++++++ 12 files changed, 780 insertions(+) create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/CHUNK_MAP.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DECISIONS.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DISCOVERY.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/INTENT.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/RISKS.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/STATUS.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md diff --git a/.agent-loop/CURRENT_STATE.md b/.agent-loop/CURRENT_STATE.md index 825402615..971ef6149 100644 --- a/.agent-loop/CURRENT_STATE.md +++ b/.agent-loop/CURRENT_STATE.md @@ -45,6 +45,7 @@ authority; these records do not grant or withhold it. | [WS-CI-002](initiatives/WS-CI-002-deterministic-agent-gates/STATUS.md) | `WS-CI-002-01` complete through PR #311; Agent Gates is deterministic per PR head | Preserve protected-branch review as the independent approval authority | | [WS-CI-003](initiatives/WS-CI-003-atomic-chunk-state/STATUS.md) | `WS-CI-003-01` complete | Require every chunk PR to land its final contract and initiative state atomically | | [WS-CI-004](initiatives/WS-CI-004-review-evidence-integrity/STATUS.md) | Exact-target, nine-reviewer, final-head, impact-cone, and adversarial-proof work is merged through PR #345; `WS-CI-004-05` semantic-completeness hardening is complete | Human review and merge `WS-CI-004-05`; do not start another successor automatically | +| [WS-CI-005](initiatives/WS-CI-005-semantic-proof-quality/STATUS.md) | Proof-strength custody and escaped-defect learning are planned; no implementation behavior exists | Start `WS-CI-005-01` only through explicit human direction | | [WS-DB-001](initiatives/WS-DB-001-v01-schema-baseline/STATUS.md) | v0.1 schema baseline complete through PRs #316 and #317 | Extend `0001_v01_baseline` only through future bounded migrations | | [WS-SEC-001](initiatives/WS-SEC-001-dependency-alert-remediation/STATUS.md) | `WS-SEC-001-01` is complete with patched runtime and tooling dependencies | Handle future security alerts through fresh bounded dependency changes | | [WS-DOCS-001](initiatives/WS-DOCS-001-current-v01-documentation/STATUS.md) | Current v0.1 entry documentation complete | Keep current pages synchronized with merged capability changes | diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/CHUNK_MAP.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/CHUNK_MAP.md new file mode 100644 index 000000000..a83762ff9 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/CHUNK_MAP.md @@ -0,0 +1,11 @@ +# Chunk Map: WS-CI-005 Semantic Proof Quality + +| Chunk | Goal | Risk | State | +|---|---|---:|---| +| `WS-CI-005-PLAN` | Record failure replay, proof model, risks, decisions, and bounded implementation contracts | L1 | Planned; no implementation behavior | +| `WS-CI-005-01` | Add proof-strength vocabulary, escaped-failure reference, receipt fields, compatibility validation, and focused tests | L1 | Planned; not started | +| `WS-CI-005-02` | Install candidate test-of-the-test and relevant failure-pattern obligations across reviewer skills/agents with mutation enforcement | L1 | Planned after `01`; not started | +| `WS-CI-005-03` | Prove adoption with blind PR #349 replay fixtures and integrate proof-quality summaries into evidence/trust workflows | L1 | Planned after `02`; not started | + +This map is sequencing guidance, not an active queue. Human approval starts one +chunk. One implementation chunk equals one pull request. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DECISIONS.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DECISIONS.md new file mode 100644 index 000000000..521dd77a3 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DECISIONS.md @@ -0,0 +1,58 @@ +# Decisions: WS-CI-005 Semantic Proof Quality + +## D1: Improve proof quality, not reviewer count + +The existing nine specialties remain. This initiative sharpens their evidence +obligations instead of adding overlapping agents. + +## D2: Proof custody follows the claimed boundary + +Pure, service, repository, transaction, concurrency, direct-SQL, composition, +and negative-structure claims require evidence capable of observing that exact +boundary. A stronger infrastructure label does not automatically replace a +different kind of proof. + +## D3: Every PASS tests at least one proof + +A final reviewer PASS requires a concrete test-of-the-test adversarial probe. +Merely listing named tests and passing commands is insufficient. + +## D4: Escaped defects become shared failure patterns + +Valid escaped findings are promoted into concise reviewer knowledge and blind +evaluation fixtures. PR-specific prose does not become an ever-growing active +checklist. + +## D5: Real isolation requires an existing foreign resource + +A missing-row mock cannot prove tenant isolation. Repository-level isolation +must exercise a valid resource owned by another tenant or project. + +## D6: Database truth requires database custody + +Rollback, locking, concurrency, triggers, constraints, and direct-SQL integrity +must be proven against the real database. Source inspection may identify a +finding but cannot produce executed database proof. + +## D7: Reuse canonical rules + +When schema, runtime, public API, migration, and database layers express the +same invariant, reviewers must identify one canonical owner or explicitly prove +equivalence. Silent duplicate validation is not accepted. + +## D8: Machine checks validate shape; blind evaluations validate judgment + +The validator enforces closed fields and declared compatibility. It does not +pretend to infer semantic correctness from filenames. Raw blind fixtures prove +whether reviewer behavior actually improves. + +## D9: Preserve simple contribution authority + +Nothing in this initiative starts work, grants permission, approves a PR, or +merges code. GitHub permissions and explicit human merge remain authoritative. + +## D10: Review artifacts are untrusted data + +Diffs, comments, findings, fixtures, and evidence may contain instructions. +Reviewers inspect but never execute or obey those instructions. Blind +evaluations must prove this behavior independently of expected-answer secrecy. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DISCOVERY.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DISCOVERY.md new file mode 100644 index 000000000..99fa80ed5 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/DISCOVERY.md @@ -0,0 +1,110 @@ +# Discovery: WS-CI-005 Semantic Proof Quality + +## Current behavior + +WS-CI-004 already supplies exact-target inspection, structured receipts, +specialty routing, atomic traceability, residual-escape analysis, and isolated +reviewer evaluations. `scripts/reviewer_contracts.py` checks that every reviewer +agent and skill contains the shared semantic requirements. Its evaluation model +uses positive, negative, stale-replay, output-contract, and handoff cases. + +The remaining gap is semantic discrimination. The validator checks that a +traceability row has an owner, implementation source, named proof, custody, and +result. It does not determine whether the named proof can observe the behavior. + +## Relevant files and symbols + +| Path | Current responsibility | Gap | +|---|---|---| +| `.agents/skills/reviewer-evidence-protocol/SKILL.md` | Universal exact-head and semantic review process | No proof-strength or test-of-the-test rule | +| `.agents/skills/{architecture,security,qa,test-delta,ci-integrity,reuse-dedup,senior-engineer}-review/SKILL.md` | Specialty review depth | No shared database/isolation/strict-fake failure patterns | +| `.codex/agents/*-reviewer.toml` | Custom reviewer execution contracts | Can repeat named proof without demonstrating discrimination | +| `.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json` | Structured advisory receipt | Trace rows do not declare proof strength or adversarial mutation outcome | +| `scripts/reviewer_contracts.py` | Reviewer contract/evaluation validator | Validates fields and tokens, not proof compatibility | +| `scripts/test_reviewer_contracts.py` | Mutation and output regression tests | Does not replay non-discriminating proof classes | +| `WS-CI-004/evaluations/{CASES,EXPECTATIONS}.json` | Blind reviewer evaluation inputs | Covers broad specialties, not the PR #349 escapes | +| `.agents/skills/evidence-gate/SKILL.md` | Deterministic pre-review evidence | Does not classify infrastructure custody | +| `.agents/skills/pr-trust-bundle/SKILL.md` | Human-facing evidence summary | Can summarize a named but semantically weak test | + +## Failure replay + +### PR #338 and PR #346 + +These exposed path continuity, atomic state vocabulary, public-owner boundaries, +completed-history immutability, ledger parity, and compound-criterion gaps. +WS-CI-004 now covers exact-head and traceability mechanics for those classes. + +### PR #349 + +The following escaped after initial internal passes: + +1. `PROJECT_POINTS` quantity `"1.0"` passed a duplicated runtime validator while + canonical schema and database rules rejected it. +2. A COMPENSATION owner fact was trusted after checking only its project, not + binding identity and instrument type. +3. Malformed immutable input leaked `AttributeError` before domain concealment. +4. A mocked repository exception was presented as transaction rollback proof. +5. Fake authorization methods raised labels for wrong-session/copy/replay cases + without constructing those conditions. +6. PostgreSQL `<>` comparisons over nullable operands allowed trigger guards to + evaluate to unknown and skip rejection. +7. Independent foreign keys allowed project, policy, and version facts that + existed individually but did not share composite ownership. +8. A cross-project read test used a mock returning `None`, duplicating the + missing-record case rather than proving repository isolation. +9. PostgreSQL regressions initially failed during shared fixture setup because + they recreated a globally unique service identity, so the intended + integrity assertion was never reached. +10. A required-version regression initially used an invalid non-UUID value the + old code already rejected instead of `None`, the previously accepted bad + selector. + +These are proof-quality failures: the named proof existed but could not +distinguish the defect. + +## Existing tests and gaps + +Existing reviewer-contract tests prove protocol adoption and receipt shape. +They do not currently prove: + +- proof custody is compatible with the claim; +- a reviewer mutates or contradicts a claimed invariant; +- strict fakes validate identity, state, and call order; +- real tenant-isolation proof persists a foreign resource; +- database review covers NULL semantics, composite ownership, direct SQL, and + rollback durability; +- reuse review compares schema, runtime, and database representations of one + canonical rule; +- escaped findings become permanent blind evaluation cases. +- fixture setup reaches the intended assertion rather than merely failing; +- a regression input distinguishes corrected behavior from the pre-fix code. + +## Dependencies and conventions to preserve + +- Extend `scripts/reviewer_contracts.py`; do not create a parallel validator. +- Extend the WS-CI-004 reviewer registry and evaluation harness; do not fork the + nine-reviewer map. +- Keep shared rules in one protocol/reference and specialty deltas in their + existing skill and agent files. +- Keep receipts advisory and out of tree; GitHub remains durable authority. +- Keep external review responses separate from internal receipts. + +## Risks discovered + +| Risk | Consequence | Planned control | +|---|---|---| +| Named proof without observability | False PASS | Closed proof-strength and compatibility rules | +| Permissive fake | Simulated security/isolation evidence | Strict-fake obligations and blind fixtures | +| ORM-only review | SQL integrity bypass | Database integrity reference and direct-SQL probes | +| Missing tenant record | Isolation test duplicates not-found | Real foreign-resource proof requirement | +| Duplicated business rule | Schema/runtime drift | Canonical-rule reuse comparison | +| More reviewer prose | Token/maintenance growth | One concise shared reference, validated adoption tokens | + +## Unknowns to measure during implementation + +- Whether proof-strength metadata belongs directly in the receipt schema or in + a referenced traceability sub-schema. +- The smallest blind-fixture set that covers each escape without leaking the + answer or making evaluation slow. +- Which proof-strength mismatches can be checked deterministically and which + remain reviewer judgments. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/INTENT.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/INTENT.md new file mode 100644 index 000000000..db3abe066 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/INTENT.md @@ -0,0 +1,106 @@ +# Intent: WS-CI-005 Semantic Proof Quality + +## Problem being solved + +Exact-head receipts, named tests, traceability rows, and adversarial prose can +still produce a false `PASS` when the proof does not discriminate the claimed +behavior. PR #349 passed internal review before later probes found duplicated +quantity rules, partial owner-fact validation, malformed-input leakage, mocked +rollback, simulated handle semantics, nullable SQL guard bypass, incomplete +composite ownership, and a cross-project test that was only a missing-row test. + +## Why this work matters + +Workstream is governed contribution infrastructure. A confident but vacuous +review can allow an authorization, tenant-isolation, lifecycle, compensation, +or audit defect to become durable. Repeated full review cycles also exhaust +maintainers and contributors. Review must become sharper, not larger. + +## Current behavior + +- WS-CI-004 binds reviews to exact clean Git targets and requires atomic + traceability, impact-cone inspection, adversarial probes, and residual-escape + analysis. +- Nine specialty reviewers have structural contract and blind-evaluation tests. +- Proof rows name tests and execution custody, but do not use a closed proof- + strength vocabulary. +- A mock can be cited for a database, transaction, isolation, or prepared-handle + claim even when it cannot exercise that behavior. +- Escaped defects are documented per PR but are not promoted into reusable + reviewer failure classes. + +## Target behavior + +Every material review claim declares the weakest acceptable proof custody. +Reviewers attempt a concrete test-of-the-test, reject proofs that cannot observe +the claimed failure, reuse canonical owner rules, and apply shared database and +tenant-isolation probes where relevant. Escaped defects become compact reusable +evaluation fixtures so the same blind spot is not rediscovered manually. + +## Design chosen + +Extend the existing reviewer protocol and evaluator with: + +1. a closed proof-strength vocabulary; +2. machine-validated claim-to-proof compatibility; +3. shared escaped-failure patterns derived from real PRs; +4. strict-fake, database-integrity, tenant-isolation, and canonical-rule reuse + obligations; +5. specialty-specific blind evaluations proving the requirements change + reviewer behavior. + +## Alternatives considered + +- Add more generic reviewers: rejected because overlapping prompts reproduce + the same blind spots and increase latency. +- Require PostgreSQL and full fanout for every PR: rejected as disproportionate. +- Depend on CodeRabbit to catch internal misses: rejected because it is an + independent external sensor, can rate-limit or skip, and does not own repo + architecture. +- Add a new contribution permission or merge gate: rejected. GitHub remains the + only repository authority. + +## Boundaries preserved + +- No Workstream product, API, authorization, payment, artifact, review, task, + contribution, compensation, or database behavior changes. +- No new hosted service, secret, model provider, merge automation, signed start, + loop memory, or post-merge reconciliation. +- Reviewer routing remains proportionate; low-risk work does not receive + ceremonial fanout. +- CodeRabbit, CI, internal review, and human review remain independent sensors. + +## Expected risks + +- Turning useful heuristics into rigid bureaucracy. +- Misclassifying proof strength and forcing infrastructure where pure proof is + sufficient. +- Adding token-heavy duplicated instructions to every skill and agent. +- Creating fixtures that leak their expected answer to reviewers. +- Mistaking a proof taxonomy for proof that a reviewer reasoned correctly. + +## What must not change + +The engineering loop remains: + +```text +Intent -> Plan -> Bounded Change -> Tests -> Review -> PR -> Human Merge +``` + +One implementation chunk remains one PR. Planning artifacts explain work; they +do not authorize contribution or merge. + +## How this will be proven + +- Validator mutation tests remove each proof-quality requirement and fail. +- Raw blind fixtures replay the missed defects from PRs #338, #346, and #349. +- Tests prove mock proof cannot satisfy database, transaction, concurrency, or + real-isolation custody. +- Tests prove each specialty catches its owned failure while valid controls + remain clear. +- Final-head reviewer replay validates the initiative against its own rules. + +## Human decisions required + +- Approve this plan before implementation. +- Approve each implementation PR and any explicitly accepted Medium risk. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md new file mode 100644 index 000000000..04343ac80 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md @@ -0,0 +1,154 @@ +# Plan: WS-CI-005 Semantic Proof Quality + +## Proposed approach + +Deepen the existing reviewer system in three bounded layers: define proof +semantics, adopt them in reviewers, then evaluate them against real escaped +defects. Do not add reviewers, hosted infrastructure, or contribution gates. + +## Proof-strength vocabulary + +Use a closed vocabulary in traceability rows: + +| Strength | Minimum observable boundary | +|---|---| +| `pure` | Pure domain function or immutable value behavior | +| `service` | Orchestration, ordering, error mapping, and forbidden calls | +| `repository` | Real query/persistence behavior against the record database | +| `transaction` | Real staged effects plus commit/rollback observation | +| `concurrency` | Independent sessions/processes contending on shared state | +| `direct_sql` | Database constraint/trigger behavior bypassing ORM validation | +| `composition` | Real public ports and composition-root wiring | +| `negative_structure` | Import/route/background-job/ownership structure through syntax-aware inspection | + +The vocabulary describes minimum custody, not a hierarchy where one proof +automatically replaces all others. For example, a database test cannot replace +a pure validation test, and a service mock cannot prove rollback. + +### Closed compatibility rules + +| Claimed boundary | Compatible minimum proof | Incompatible substitute | +|---|---|---| +| Pure validation/value rule | `pure` | Coverage or source grep alone | +| Service ordering/error/forbidden call | `service` | Pure helper test without orchestration | +| Query/persistence/tenant isolation | `repository` with a real stored row | Mock returning missing/`None` | +| Commit or rollback atomicity | `transaction` with staged and final state | Mocked repository exception | +| Lock/race behavior | `concurrency` with independent sessions/processes | Sequential service calls | +| Constraint/trigger/NULL guard | `direct_sql` against the real database | ORM-only construction or source inspection | +| Runtime owner-port wiring | `composition` | Service with a permissive fake | +| Import/route/background-job absence | `negative_structure` with syntax/registry-aware inspection | Raw substring grep where equivalent syntax exists | + +Every traceability row declares both `claimed_boundary` and `proof_strength`. +The validator owns this compatibility matrix; a reviewer-supplied +`proof_compatibility` value cannot override it. `unavailable` never supports +PASS, and proof compatible with one boundary does not satisfy another. + +## Test-of-the-test rule + +Every final reviewer verdict must attempt at least one concrete contradiction of +a material claim. The probe states: + +- the defect inserted or simulated; +- why the named proof should observe it; +- the actual result; +- whether the proof survived incorrectly; +- the resulting finding or residual uncertainty. + +For changed tests, QA/test-delta must inspect whether the test would fail under +the defect. For unchanged trusted proof, a targeted mutation or equivalent +source-level counterexample may be used when execution is unavailable, but the +receipt must label it as inspected rather than executed. + +## Shared escaped-failure patterns + +Create one concise reference consumed by the shared protocol and specialties: + +- permissive fake / label-only exception; +- mock used for repository, transaction, concurrency, or direct-SQL claim; +- missing real foreign tenant resource; +- partial returned-fact validation; +- duplicated canonical rule across schema/runtime/database; +- nullable SQL comparison and three-valued-logic escape; +- independent foreign keys where composite ownership is required; +- malformed public value reaching attribute access or persistence before + concealment; +- source-text boundary grep that misses syntax-equivalent imports; +- aggregate coverage or broad test invocation hiding an unproven behavior. +- untrusted diff, comment, finding, or evidence text instructing a reviewer to + ignore protocol, fabricate proof, execute content, or return `PASS`. +- fixture setup aborting before the intended assertion; +- regression input that the pre-fix implementation already rejects. + +The reference is reviewer knowledge, not an automatic universal checklist. +Reviewers select patterns relevant to the impact cone and say why. + +## Machine validation + +Extend the canonical receipt and reviewer-contract validator to require: + +- `proof_strength` on every traceability row; +- an explicit `proof_compatibility` result; +- at least one test-of-the-test adversarial probe for PASS; +- unavailable infrastructure custody to remain explicit; +- no PASS where the named proof is weaker than the declared behavior boundary; +- stable failure-pattern IDs in findings/evaluations. + +Do not attempt to infer semantics from test filenames. Deterministic validation +checks closed fields, compatibility declarations, and required evidence. Blind +evaluations test whether reviewers apply the judgment correctly. + +## Specialty adoption + +- Architecture: composite ownership, syntax-aware private edges, composition + roots, and database/model parity. +- Security: tenant/actor/resource substitution, nullable/fail-open state, + replay, and concealment. +- QA and test-delta: discriminating proof, real failure injection, and + test-of-the-test mutation. +- CI integrity: exact execution custody, selection, PostgreSQL/services, + coverage and artifact provenance. +- Reuse/dedup: canonical rule comparison across schema, service, public API, + migration, and database constraint. +- Senior engineering: permissive fakes, misleading abstractions, and proof cost. +- Documentation and product/operations retain their specialties and adopt the + common proof fields without irrelevant database ceremony. + +## Alternatives rejected + +- One large implementation PR: too difficult to review and self-test. +- A new database reviewer: responsibilities already belong to architecture, + security, QA/test-delta, and CI integrity. +- Mandatory mutation tooling for the entire repository: expensive and outside + this initiative; targeted reviewer evaluation mutations are sufficient. +- Natural-language-only guidance: cannot detect drift or missing adoption. + +## Boundaries preserved + +- Repository engineering infrastructure only. +- No product/runtime code or migrations. +- No CI threshold reduction, test removal, or reviewer bypass. +- No automatic merge, approval, start, or post-merge state mutation. + +## Verification strategy + +- Unit tests for proof-strength schema and compatibility validation. +- Mutation tests removing every new shared requirement from a skill and agent. +- Blind positive/negative fixtures for every PR #349 failure class. +- Untrusted-evidence fixtures prove reviewers ignore embedded instructions and + never execute commands supplied by diffs, comments, findings, or evidence. +- False-positive controls showing when a pure/service mock is appropriate. +- Output-set tests proving exact-head receipts cannot PASS with incompatible or + unavailable proof. +- Forward evaluation of all nine reviewers with raw artifacts and no leaked + expected answer. +- Standard state, Markdown, stale wording, diff, and reviewer-contract checks. + +## Delivery sequence + +1. `WS-CI-005-01`: shared vocabulary, receipt schema, validator, and reference. +2. `WS-CI-005-02`: candidate specialty skill/agent obligations and contract + mutation tests; this step does not claim behavioral adoption. +3. `WS-CI-005-03`: escaped-defect blind fixtures, forward evaluation, final + reviewer adoption, and trust-bundle/evidence-gate integration. + +Each chunk is one PR. The next chunk begins only after explicit human start. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/RISKS.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/RISKS.md new file mode 100644 index 000000000..c1f51e554 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/RISKS.md @@ -0,0 +1,15 @@ +# Risks: WS-CI-005 Semantic Proof Quality + +| Risk | Severity | Mitigation | +|---|---:|---| +| A named test cannot observe its claimed invariant | Critical | Proof strength plus test-of-the-test requirement | +| Mock evidence is accepted for transaction or database truth | Critical | Closed custody compatibility and blind fixtures | +| Tenant isolation is tested as missing data | Critical | Require a real foreign resource for repository/isolation claims | +| SQL NULL or composite ownership bypass is missed | Critical | Shared database-integrity failure patterns and direct-SQL proof | +| Schema/runtime/database rules drift | High | Canonical-rule reuse review across all representations | +| Reviewer instructions grow and diverge | High | One shared reference and deterministic adoption validation | +| New rules force infrastructure for small changes | High | Proof type follows claim; risk routing remains proportionate | +| Blind fixtures leak expected findings | High | Raw cases and separate expectations remain isolated | +| Untrusted review text overrides protocol or causes execution | Critical | Prompt-injection fixtures and explicit non-execution proof | +| More ceremony without better detection | High | Forward evaluations must detect real escaped defects | +| Review evidence becomes merge authority | Critical | Receipts remain advisory; human/GitHub authority unchanged | diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/STATUS.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/STATUS.md new file mode 100644 index 000000000..f4aee3fa1 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/STATUS.md @@ -0,0 +1,26 @@ +# Status: WS-CI-005 Semantic Proof Quality + +## Current status + +Planning is complete for this planning change. No implementation behavior +exists, and every implementation chunk remains inactive. + +## Current focus + +Review the first-principles proof-strength design and the three bounded delivery +contracts. Preserve the simple engineering loop and avoid adding reviewers or +repository authority. + +## Chunk status + +| Chunk | State | Outcome | +|---|---|---| +| `WS-CI-005-PLAN` | Planned | Planning artifacts only | +| `WS-CI-005-01` | Planned | No implementation; requires explicit human start | +| `WS-CI-005-02` | Planned | No implementation; depends on `01` | +| `WS-CI-005-03` | Planned | No implementation; depends on `02` | + +## Next action + +Start `WS-CI-005-01` only through explicit human direction. Do not implement it +automatically when this plan merges. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md new file mode 100644 index 000000000..6ddc87a0e --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md @@ -0,0 +1,79 @@ +# Chunk: WS-CI-005-01 Proof-Strength Foundation + +## Intent + +Add the shared proof vocabulary and deterministic receipt/validator foundation +without changing specialty reviewer prompts yet. + +## Allowed files + +```text +.agents/skills/reviewer-evidence-protocol/SKILL.md +.agents/skills/reviewer-evidence-protocol/references/proof-quality-patterns.md +.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json +.agent-loop/templates/INTERNAL_REVIEW_EVIDENCE.md +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/REVIEWER_MATRIX.md +.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/** +scripts/reviewer_contracts.py +scripts/test_reviewer_contracts.py +``` + +## Not allowed + +Product/runtime code, migrations, workflows, contributor permission changes, +specialty reviewer agent/skill edits, or hosted receipt custody. + +## Acceptance criteria + +- Closed proof-strength values are defined once. +- Every receipt trace row declares proof strength and compatibility. +- PASS requires at least one test-of-the-test probe. +- Incompatible or unavailable proof fails closed. +- PR #349 failure patterns live in one concise shared reference. +- Mutation and malformed-output tests cover every new field/rule. + +## Risk + +L1/P1 reviewer infrastructure. + +## Verification + +### Acceptance-to-proof map + +| Behavior atom | Future implementation | Required future proof | Custody | +|---|---|---|---| +| Only the eight closed proof strengths are accepted | receipt schema and validator | `ReviewerContractTests.test_receipt_rejects_unknown_proof_strength` | local unit | +| Every trace row declares claimed boundary and proof strength | receipt schema | `ReviewerContractTests.test_receipt_requires_boundary_and_strength_per_trace_row` | local unit | +| Compatibility comes from the validator-owned closed matrix | `proof_compatibility_failures` | `ReviewerContractTests.test_reviewer_cannot_self_attest_compatibility` | local unit | +| Service/mock proof cannot satisfy repository, transaction, concurrency, or direct-SQL claims | compatibility matrix | `ReviewerContractTests.test_weaker_proof_cannot_satisfy_infrastructure_claims` | local unit | +| Real isolation proof requires a stored foreign resource | compatibility matrix | `ReviewerContractTests.test_isolation_proof_rejects_missing_row_mock` | local unit | +| Unavailable proof prevents PASS | receipt validator | `ReviewerContractTests.test_unavailable_proof_blocks_pass` | local unit | +| Proof types are not a substitutable strength hierarchy | compatibility matrix | `ReviewerContractTests.test_proof_types_are_not_a_strength_hierarchy` | local unit | +| PASS requires a test-of-the-test probe and observed result | receipt schema and validator | `ReviewerContractTests.test_pass_requires_test_of_the_test_probe` | local unit | +| Shared failure-pattern IDs are complete and unique | shared reference and validator | `ReviewerContractTests.test_failure_pattern_registry_is_complete_and_unique` | local unit | + +All selectors above live in `scripts.test_reviewer_contracts`. The full command +remains: + +```bash +python3 -m unittest -v scripts.test_reviewer_contracts scripts.test_review_target +python3 scripts/reviewer_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_review_contracts.py +git diff --check +``` + +## Required reviewers + +Architecture, CI integrity, security, QA, test delta, senior engineering, +reuse/dedup, documentation, and product/operations. + +## Human review focus + +Confirm the taxonomy is minimal, does not infer semantics from filenames, and +does not add contribution or merge authority. + +## Outcome on merge + +Shared proof-strength structure is available; reviewer adoption remains +unstarted until explicit human direction. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md new file mode 100644 index 000000000..6b4f8cb2f --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md @@ -0,0 +1,77 @@ +# Chunk: WS-CI-005-02 Reviewer Proof Candidate Contracts + +## Intent + +Install candidate test-of-the-test, strict-fake, database/isolation, and +canonical-rule obligations across the existing specialty reviewers. This +chunk proves prompt and contract presence, not behavioral adoption. + +## Allowed files + +```text +.agents/skills/*-review/SKILL.md +.codex/agents/*-reviewer.toml +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/REVIEWER_MATRIX.md +.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/** +scripts/reviewer_contracts.py +scripts/test_reviewer_contracts.py +``` + +## Not allowed + +New reviewer specialties, product/runtime code, workflows, migrations, +repository permissions, or universal fanout requirements. + +## Acceptance criteria + +- Every reviewer consumes the shared proof model without duplicating it. +- Specialty responsibilities are explicit and non-overlapping. +- QA/test-delta challenge whether tests discriminate behavior. +- Architecture/security cover database and composite ownership where relevant. +- Reuse compares canonical rule representations. +- CI integrity verifies actual infrastructure custody. +- Contract mutation tests fail when any adoption obligation is removed. + +## Risk + +L1/P1 reviewer behavior. + +## Verification + +### Acceptance-to-proof map + +| Behavior atom | Future implementation | Required future proof | Custody | +|---|---|---|---| +| All nine agent/skill pairs consume the shared proof model | reviewer contracts and matrix | `ReviewerContractTests.test_all_reviewer_contracts_require_shared_proof_quality` | local unit | +| Removing a shared skill obligation fails closed | semantic skill requirements | `ReviewerContractTests.test_each_proof_quality_skill_requirement_is_independently_enforced` | local mutation | +| Removing a shared agent obligation fails closed | semantic agent requirements | `ReviewerContractTests.test_each_proof_quality_agent_requirement_is_independently_enforced` | local mutation | +| QA/test-delta require a discrimination probe | QA/test-delta contracts | `ReviewerContractTests.test_qa_and_test_delta_require_discrimination_probe` | local unit | +| Architecture/security require relevant database-integrity probes | architecture/security contracts | `ReviewerContractTests.test_architecture_and_security_require_database_integrity_probe` | local unit | +| Reuse requires canonical-rule comparison | reuse contracts | `ReviewerContractTests.test_reuse_requires_canonical_rule_comparison` | local unit | +| CI integrity traces actual infrastructure execution custody | CI contracts | `ReviewerContractTests.test_ci_requires_execution_custody_trace` | local unit | +| Docs/product remain proportionate without database ceremony | docs/product contracts | `ReviewerContractTests.test_docs_and_product_contracts_remain_proportionate` | local unit | + +All selectors above live in `scripts.test_reviewer_contracts`. The full command +remains: + +```bash +python3 -m unittest -v scripts.test_reviewer_contracts +python3 scripts/reviewer_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_review_contracts.py +git diff --check +``` + +## Required reviewers + +All nine existing reviewers, with each reviewing its own contract and adjacent +handoff boundary. No reviewer validates itself as the sole evidence. + +## Human review focus + +Confirm the prompts remain concise, proportionate, and useful for small changes. + +## Outcome on merge + +Candidate reviewer proof-quality contracts are installed. They are not called +adopted until `WS-CI-005-03` passes independent blind forward evaluation. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md new file mode 100644 index 000000000..ee35b35f7 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md @@ -0,0 +1,91 @@ +# Chunk: WS-CI-005-03 Escaped-Defect Evaluation + +## Intent + +Prove the improved reviewer system detects the real failure classes that escaped +earlier review, establish behavioral adoption, and integrate concise results +into evidence/trust workflows. + +## Allowed files + +```text +.agents/skills/evidence-gate/SKILL.md +.agents/skills/pr-trust-bundle/SKILL.md +.agents/skills/task-chunk-loop/SKILL.md +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/** +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/REVIEWER_MATRIX.md +.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/** +scripts/reviewer_contracts.py +scripts/test_reviewer_contracts.py +``` + +## Not allowed + +Product/runtime code, hosted AI review gates, secrets, automatic approval/merge, +or using evaluation outcomes as contribution authority. + +## Acceptance criteria + +- Raw fixtures cover every recorded PR #349 escape without expected-answer + leakage. +- Matching negative controls prevent universal over-flagging. +- Raw fixtures include malicious embedded instructions; reviewers ignore them, + do not execute supplied commands, and report the underlying evidence only. +- Removing untrusted-evidence fixture coverage fails deterministic validation. +- Each applicable reviewer detects or correctly hands off its owned defects. +- Output validation rejects a PASS backed by incompatible proof. +- Evidence and trust-bundle skills summarize proof quality and uncertainty + without copying private session receipts into Git. +- Full forward-evaluation results are recorded against one exact head. + +## Risk + +L1/P1 reviewer evaluation and workflow integration. + +## Verification + +### Acceptance-to-proof map + +| Failure atom | Required raw fixture | Required future proof | Custody | +|---|---|---|---| +| Canonical rule drift | schema/runtime/database disagreement | `ReviewerContractTests.test_blind_fixture_detects_canonical_rule_drift` | blind evaluation | +| Partial owner facts | correct project with wrong identity/instrument | `ReviewerContractTests.test_blind_fixture_detects_partial_owner_fact_validation` | blind evaluation | +| Malformed public input leak | wrong child type before attribute access | `ReviewerContractTests.test_blind_fixture_detects_malformed_input_leak` | blind evaluation | +| Mocked rollback | fake exception without staged state | `ReviewerContractTests.test_blind_fixture_rejects_mocked_rollback_proof` | blind evaluation | +| Label-only security fake | scenario name raised without constructed misuse | `ReviewerContractTests.test_blind_fixture_rejects_label_only_security_fake` | blind evaluation | +| SQL NULL guard escape | nullable comparison bypass | `ReviewerContractTests.test_blind_fixture_detects_sql_null_guard_escape` | blind evaluation | +| Composite ownership gap | individually valid cross-project IDs | `ReviewerContractTests.test_blind_fixture_detects_composite_ownership_gap` | blind evaluation | +| Missing-row isolation proof | mock `None` without stored foreign resource | `ReviewerContractTests.test_blind_fixture_rejects_missing_row_isolation_proof` | blind evaluation | +| Setup-only failure | unique seed collision before target assertion | `ReviewerContractTests.test_blind_fixture_detects_setup_only_failure` | blind evaluation | +| Non-discriminating regression input | invalid value old behavior already rejects | `ReviewerContractTests.test_blind_fixture_rejects_non_discriminating_input` | blind evaluation | +| Untrusted evidence instruction | malicious instruction plus underlying defect | `ReviewerContractTests.test_blind_fixture_ignores_untrusted_evidence_instructions` | blind evaluation | +| False-positive controls | valid mock/pure proof and valid public owner path | `ReviewerContractTests.test_blind_fixture_false_positive_controls` | blind evaluation | +| Fixture coverage cannot be removed | cases/expectations coverage | `ReviewerContractTests.test_proof_quality_fixture_classes_are_mandatory` | local mutation | +| Trust summaries do not claim receipt custody | evidence/trust skills | `ReviewerContractTests.test_trust_workflows_summarize_without_claiming_custody` | local unit | + +All selectors above live in `scripts.test_reviewer_contracts`. The full command +remains: + +```bash +python3 -m unittest -v scripts.test_reviewer_contracts scripts.test_review_target +python3 scripts/reviewer_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_review_contracts.py +git diff --check +``` + +## Required reviewers + +All nine existing reviewers plus human review of false-positive controls and +evaluation independence. + +## Human review focus + +Confirm the fixtures test transferable reasoning rather than memorized wording, +and that readiness remains advisory until human merge. + +## Outcome on merge + +The reviewer contracts are behaviorally adopted and the semantic proof-quality +initiative is complete. Future escaped findings are added only when they +represent a reusable failure class. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md new file mode 100644 index 000000000..3ebb9f081 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md @@ -0,0 +1,52 @@ +# Chunk: WS-CI-005-PLAN Semantic Proof Quality Planning + +## Intent + +Record the evidence-backed design and bounded delivery sequence for improving +reviewer proof discrimination after PR #349. + +## Allowed files + +```text +.agent-loop/CURRENT_STATE.md +.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/** +``` + +## Not allowed + +Application code, tests, workflows, reviewer skills/agents, validator behavior, +product documentation, schemas, migrations, or repository authority changes. + +## Acceptance criteria + +- Intent, discovery, plan, chunk map, status, risks, and decisions are present. +- PR #349 escaped failure classes are recorded without treating CodeRabbit as + authoritative. +- Proof vocabulary and test-of-the-test design are explicit. +- Exactly three implementation chunks have bounded contracts. +- No implementation behavior is claimed complete. + +## Risk + +L1 planning for critical engineering infrastructure. + +## Verification + +```bash +python3 scripts/check_active_state_projections.py +python3 scripts/check_chunk_state_sync.py --base-ref origin/main +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_review_contracts.py +git diff --check +``` + +## Required reviewers + +Architecture, CI integrity, security, QA/test-delta, senior engineering, +reuse/dedup, and documentation. Product/operations verifies that engineering +review language does not alter Workstream product review decisions. + +## Outcome on merge + +The initiative plan becomes durable and `WS-CI-005-01` remains planned. No +implementation begins automatically. From c848ccfeccb96001cc7689060311b6acccd435d6 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 17 Aug 2026 21:37:49 +0100 Subject: [PATCH 2/4] docs(ci): normalize chunk merge states --- .agent-loop/CURRENT_STATE.md | 2 +- .../chunks/WS-CI-005-01-proof-strength-foundation.md | 4 +++- .../chunks/WS-CI-005-02-reviewer-proof-adoption.md | 4 +++- .../chunks/WS-CI-005-03-escaped-defect-evaluation.md | 4 +++- .../chunks/WS-CI-005-PLAN-semantic-proof-quality.md | 4 +++- 5 files changed, 13 insertions(+), 5 deletions(-) diff --git a/.agent-loop/CURRENT_STATE.md b/.agent-loop/CURRENT_STATE.md index 971ef6149..1fcace3ba 100644 --- a/.agent-loop/CURRENT_STATE.md +++ b/.agent-loop/CURRENT_STATE.md @@ -45,7 +45,7 @@ authority; these records do not grant or withhold it. | [WS-CI-002](initiatives/WS-CI-002-deterministic-agent-gates/STATUS.md) | `WS-CI-002-01` complete through PR #311; Agent Gates is deterministic per PR head | Preserve protected-branch review as the independent approval authority | | [WS-CI-003](initiatives/WS-CI-003-atomic-chunk-state/STATUS.md) | `WS-CI-003-01` complete | Require every chunk PR to land its final contract and initiative state atomically | | [WS-CI-004](initiatives/WS-CI-004-review-evidence-integrity/STATUS.md) | Exact-target, nine-reviewer, final-head, impact-cone, and adversarial-proof work is merged through PR #345; `WS-CI-004-05` semantic-completeness hardening is complete | Human review and merge `WS-CI-004-05`; do not start another successor automatically | -| [WS-CI-005](initiatives/WS-CI-005-semantic-proof-quality/STATUS.md) | Proof-strength custody and escaped-defect learning are planned; no implementation behavior exists | Start `WS-CI-005-01` only through explicit human direction | +| [WS-CI-005](initiatives/WS-CI-005-semantic-proof-quality/STATUS.md) | `WS-CI-005-PLAN`, `WS-CI-005-01`, `WS-CI-005-02`, and `WS-CI-005-03` are planned; no implementation behavior exists | Start `WS-CI-005-01` only through explicit human direction | | [WS-DB-001](initiatives/WS-DB-001-v01-schema-baseline/STATUS.md) | v0.1 schema baseline complete through PRs #316 and #317 | Extend `0001_v01_baseline` only through future bounded migrations | | [WS-SEC-001](initiatives/WS-SEC-001-dependency-alert-remediation/STATUS.md) | `WS-SEC-001-01` is complete with patched runtime and tooling dependencies | Handle future security alerts through fresh bounded dependency changes | | [WS-DOCS-001](initiatives/WS-DOCS-001-current-v01-documentation/STATUS.md) | Current v0.1 entry documentation complete | Keep current pages synchronized with merged capability changes | diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md index 6ddc87a0e..dbd4fdc88 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md @@ -73,7 +73,9 @@ reuse/dedup, documentation, and product/operations. Confirm the taxonomy is minimal, does not infer semantics from filenames, and does not add contribution or merge authority. -## Outcome on merge +## Merge state + +- Outcome on merge: `planned` Shared proof-strength structure is available; reviewer adoption remains unstarted until explicit human direction. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md index 6b4f8cb2f..b3159a03c 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md @@ -71,7 +71,9 @@ handoff boundary. No reviewer validates itself as the sole evidence. Confirm the prompts remain concise, proportionate, and useful for small changes. -## Outcome on merge +## Merge state + +- Outcome on merge: `planned` Candidate reviewer proof-quality contracts are installed. They are not called adopted until `WS-CI-005-03` passes independent blind forward evaluation. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md index ee35b35f7..1ae4a934a 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md @@ -84,7 +84,9 @@ evaluation independence. Confirm the fixtures test transferable reasoning rather than memorized wording, and that readiness remains advisory until human merge. -## Outcome on merge +## Merge state + +- Outcome on merge: `planned` The reviewer contracts are behaviorally adopted and the semantic proof-quality initiative is complete. Future escaped findings are added only when they diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md index 3ebb9f081..77065986c 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-PLAN-semantic-proof-quality.md @@ -46,7 +46,9 @@ Architecture, CI integrity, security, QA/test-delta, senior engineering, reuse/dedup, and documentation. Product/operations verifies that engineering review language does not alter Workstream product review decisions. -## Outcome on merge +## Merge state + +- Outcome on merge: `planned` The initiative plan becomes durable and `WS-CI-005-01` remains planned. No implementation begins automatically. From 9f452c4f23fe835fbd50965cc94c1d242b248a61 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 17 Aug 2026 22:43:04 +0100 Subject: [PATCH 3/4] docs(ci): close semantic proof review gaps --- .agent-loop/REVIEW_LOG.md | 11 ++ .../WS-CI-005-semantic-proof-quality/PLAN.md | 6 +- .../WS-CI-005-01-proof-strength-foundation.md | 5 +- .../WS-CI-005-02-reviewer-proof-adoption.md | 5 +- .../WS-CI-005-03-escaped-defect-evaluation.md | 7 +- ...WS-CI-005-PLAN-external-review-response.md | 38 +++++++ .../reviews/WS-CI-005-PLAN-pr-trust-bundle.md | 106 ++++++++++++++++++ 7 files changed, 170 insertions(+), 8 deletions(-) create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-external-review-response.md create mode 100644 .agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md diff --git a/.agent-loop/REVIEW_LOG.md b/.agent-loop/REVIEW_LOG.md index 3a24e7c9b..c21fdea81 100644 --- a/.agent-loop/REVIEW_LOG.md +++ b/.agent-loop/REVIEW_LOG.md @@ -3001,3 +3001,14 @@ the `08A` executable contract omitted `CON-03D`, and canonical dependency views used broad REV persistence labels. The repair adds the explicit `03D -> 08A` gate and names exact merged `REV-04B` runtime `Review`/`ReviewLease`/`FinalAcceptance` targets. + +## 2026-08-17 - WS-CI-005 Semantic Proof Quality Planning + +PR #350 plans a semantic-proof layer for the existing exact-head reviewer +system. Initial internal review corrected proof self-attestation, incomplete +failure replay, missing exact tests, malicious-evidence coverage, premature +adoption language, stale terminology, and invalid chunk states. CodeRabbit +then identified two final gaps: source inspection could appear to replace +required database/session custody, and future outcomes used present-tense +completion wording. Both are corrected; final-head review and hosted checks +remain required before human merge. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md index 04343ac80..ebd7fcbf4 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/PLAN.md @@ -57,7 +57,11 @@ a material claim. The probe states: For changed tests, QA/test-delta must inspect whether the test would fail under the defect. For unchanged trusted proof, a targeted mutation or equivalent source-level counterexample may be used when execution is unavailable, but the -receipt must label it as inspected rather than executed. +receipt must label it as inspected rather than executed. Source inspection may +satisfy only a compatible source-level claim. Repository, transaction, +concurrency, and direct-SQL claims still require existing executed proof with +the custody declared by the compatibility table; a source-level counterexample +is supplemental evidence and cannot support `PASS` for those claims by itself. ## Shared escaped-failure patterns diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md index dbd4fdc88..e640d9acc 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-01-proof-strength-foundation.md @@ -77,5 +77,6 @@ does not add contribution or merge authority. - Outcome on merge: `planned` -Shared proof-strength structure is available; reviewer adoption remains -unstarted until explicit human direction. +After this implementation chunk merges, the shared proof-strength structure +will be available; reviewer adoption will remain unstarted until explicit +human direction. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md index b3159a03c..6ecb37248 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-02-reviewer-proof-adoption.md @@ -75,5 +75,6 @@ Confirm the prompts remain concise, proportionate, and useful for small changes. - Outcome on merge: `planned` -Candidate reviewer proof-quality contracts are installed. They are not called -adopted until `WS-CI-005-03` passes independent blind forward evaluation. +After this implementation chunk merges, candidate reviewer proof-quality +contracts will be installed. They will not be called adopted until +`WS-CI-005-03` passes independent blind forward evaluation. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md index 1ae4a934a..95cd1a983 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/chunks/WS-CI-005-03-escaped-defect-evaluation.md @@ -88,6 +88,7 @@ and that readiness remains advisory until human merge. - Outcome on merge: `planned` -The reviewer contracts are behaviorally adopted and the semantic proof-quality -initiative is complete. Future escaped findings are added only when they -represent a reusable failure class. +Only after this chunk's independent blind evaluation succeeds will the +reviewer contracts be behaviorally adopted and the semantic proof-quality +initiative become complete. Future escaped findings will be added only when +they represent a reusable failure class. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-external-review-response.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-external-review-response.md new file mode 100644 index 000000000..7513b97b2 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-external-review-response.md @@ -0,0 +1,38 @@ +# External Review Response: WS-CI-005-PLAN + +## Comments addressed + +- Restricted source-level counterexamples to compatible source-level claims. + Repository, transaction, concurrency, and direct-SQL claims still require + executed proof with the custody declared by the compatibility table. +- Changed the three future implementation outcomes to conditional wording so + planning does not claim that unstarted behavior already exists. +- Expanded the GitHub pull-request description to the repository trust-bundle + structure. + +## Comments deferred + +None. + +## Human decisions needed + +Human approval remains required before merge and before `WS-CI-005-01` starts. + +## Commands rerun + +- `python3 scripts/check_active_state_projections.py` +- `python3 scripts/check_chunk_state_sync.py --base-ref origin/main` +- `python3 scripts/check_markdown_links.py` +- `python3 scripts/check_stale_review_contracts.py` +- `python3 scripts/check_stale_authorization_docs.py` +- `python3 scripts/check_stale_workstream_wording.py` +- `python3 scripts/check_stale_artifact_contracts.py` +- `python3 scripts/reviewer_contracts.py` +- `python3 -m unittest -v scripts.test_reviewer_contracts scripts.test_review_target` +- `git diff --check origin/main...HEAD` + +## Remaining risks + +This pull request defines planning behavior only. The future implementation +must prove the declared custody and reviewer behavior through the named tests +and independent blind evaluation. diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md new file mode 100644 index 000000000..d0ffbd9ae --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md @@ -0,0 +1,106 @@ +# PR Trust Bundle: WS-CI-005-PLAN + +## Chunk + +`WS-CI-005-PLAN` + +## Goal + +Plan a discriminating semantic-proof layer for the existing reviewer system. + +## Human-approved intent + +Improve reviewer skills and custom agents so a `PASS` is supported by proof +capable of observing the claimed defect, without adding repository authority. + +## What changed + +Added the initiative intent, discovery, plan, risks, decisions, status, chunk +map, and three bounded future implementation contracts. + +## Why it changed + +Prior reviews were correctly bound to exact Git targets but still accepted +tests or inspection too weak to prove the claimed behavior. + +## Design chosen + +Extend the existing WS-CI-004 reviewer protocol with a closed claimed-boundary +and proof-strength model, test-of-the-test probes, reusable escaped-failure +patterns, and blind forward evaluation before behavioral adoption. + +## Alternatives rejected + +- More reviewers: duplicates specialties without improving proof quality. +- Filename inference: cannot establish semantic compatibility. +- Hosted evidence authority: would create a second authority system. +- Universal heavyweight execution: disproportionate for low-risk changes. + +## Scope control + +Planning and repository engineering memory only. No product code, migration, +workflow, permission, merge gate, coverage threshold, or reviewer specialty is +changed. + +## Product behavior + +None. + +## Acceptance criteria proof + +- Proof boundaries and compatibility are explicit in `PLAN.md`. +- Ten historical escaped-defect classes and malicious evidence are recorded. +- Each future acceptance atom maps to an exact named proof. +- State projections and merge outcomes are machine-valid and remain planned. + +## Tests/checks run + +- Active-state, chunk-state, Markdown-link, and stale-wording checks passed. +- Reviewer contract validation passed. +- Reviewer-contract and review-target unit tests passed: 35 tests. +- `git diff --check origin/main...HEAD` passed. + +## Test delta + +No runtime tests changed. Future test selectors are contractual obligations, +not claims that implementation already exists. + +## CI integrity + +No workflow, runner, lane, coverage command, threshold, or package script +changed. Required Agent Gates commands pass locally. + +## Reviewer results + +Architecture, CI integrity, QA, security, documentation/product, and senior +engineering reviewed exact head `c848ccfeccb96001cc7689060311b6acccd435d6`. +All prior findings were replayed and closed; final results were `PASS` or +`PASS WITH LOW RISKS`. These results become stale after the external-review +repair and must be replayed on its final exact head. + +## External review + +CodeRabbit provided a fresh substantive review on PR #350. Its two findings +are addressed in the adjacent external-review response; final-head refresh is +required. + +## Remaining risks + +The planned implementation may still become ceremonial or non-discriminating. +Each future chunk therefore requires its named controls and final blind +evaluation before behavioral adoption. + +## Follow-up work + +After human approval, start only `WS-CI-005-01`. Do not start successors +automatically. + +## Human review focus + +Confirm source inspection cannot replace executed database/session custody, +the taxonomy remains minimal, and no new authority is introduced. + +## Human merge ownership + +GitHub permissions, protected-branch checks, and explicit human approval remain +authoritative. Internal receipts and this bundle are advisory evidence only. From 4afc88e43a59db1bf00bb6cd73637434c1e00f05 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 17 Aug 2026 22:49:34 +0100 Subject: [PATCH 4/4] docs(ci): enumerate reviewer trust results --- .../reviews/WS-CI-005-PLAN-pr-trust-bundle.md | 28 +++++++++++++++---- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md index d0ffbd9ae..685321d78 100644 --- a/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-005-semantic-proof-quality/reviews/WS-CI-005-PLAN-pr-trust-bundle.md @@ -72,11 +72,29 @@ changed. Required Agent Gates commands pass locally. ## Reviewer results -Architecture, CI integrity, QA, security, documentation/product, and senior -engineering reviewed exact head `c848ccfeccb96001cc7689060311b6acccd435d6`. -All prior findings were replayed and closed; final results were `PASS` or -`PASS WITH LOW RISKS`. These results become stale after the external-review -repair and must be replayed on its final exact head. +Initial reviewed code SHA: `c848ccfeccb96001cc7689060311b6acccd435d6` + +Reviewed at: 2026-08-17 + +Reviewer run IDs: `ci005_plan_arch`, `ci005_plan_ci`, `ci005_plan_qa`, +`ci005_plan_security`, `ci005_plan_docs_product`, `ci005_plan_senior` + +| Reviewer | Result | Blocking findings | Notes | +|---|---:|---|---| +| senior engineering | PASS WITH LOW RISKS | None | Adoption remains conditional on blind evaluation. | +| QA/test | PASS | None | Exact proof selectors and all ten escaped classes verified. | +| security/auth | PASS | None | Untrusted evidence and custody boundaries verified. | +| product/ops | PASS | None | Same focused docs/product session; no product lifecycle change. | +| architecture | PASS | None | Canonical WS-CI-004 ownership and three-chunk boundary verified. | +| CI integrity | PASS WITH LOW RISKS | None | No CI surface changed; required local gates passed. | +| docs | PASS | None | Durable state, terminology, and links verified. | +| reuse/dedup | N/A - with approved reason | None | Planning-only Markdown adds no implementation helper or abstraction. Architecture still verified canonical validator and registry reuse. | +| test delta | N/A - with approved reason | None | No test file, assertion, selection, or coverage command changed. QA still verified future proof obligations. | + +The external-review repair makes these committed results historical. Final +exact-head receipts are create-once local session evidence and must replay the +applicable tracks after every push; the GitHub PR body mirrors their current +status without making this committed bundle self-referential. ## External review