diff --git a/.agent-loop/CURRENT_STATE.md b/.agent-loop/CURRENT_STATE.md index 31921fd57..a8bd0f3ea 100644 --- a/.agent-loop/CURRENT_STATE.md +++ b/.agent-loop/CURRENT_STATE.md @@ -44,7 +44,7 @@ authority; these records do not grant or withhold it. | [WS-CI-001](initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md) | Semantic distributed backend lanes complete | Treat further CI optimization as a fresh measured bounded change | | [WS-CI-002](initiatives/WS-CI-002-deterministic-agent-gates/STATUS.md) | `WS-CI-002-01` complete through PR #311; Agent Gates is deterministic per PR head | Preserve protected-branch review as the independent approval authority | | [WS-CI-003](initiatives/WS-CI-003-atomic-chunk-state/STATUS.md) | `WS-CI-003-01` complete | Require every chunk PR to land its final contract and initiative state atomically | -| [WS-CI-004](initiatives/WS-CI-004-review-evidence-integrity/STATUS.md) | Exact-target protocol, nine-reviewer adoption, and final-head closure are merged through PRs #341-#343; `WS-CI-004-04` is complete with explicit impact-cone and adversarial review proof | Human review `WS-CI-004-04`; do not start another successor automatically | +| [WS-CI-004](initiatives/WS-CI-004-review-evidence-integrity/STATUS.md) | Exact-target, nine-reviewer, final-head, impact-cone, and adversarial-proof work is merged through PR #345; `WS-CI-004-05` semantic-completeness hardening is complete | Human review and merge `WS-CI-004-05`; do not start another successor automatically | | [WS-DB-001](initiatives/WS-DB-001-v01-schema-baseline/STATUS.md) | v0.1 schema baseline complete through PRs #316 and #317 | Extend `0001_v01_baseline` only through future bounded migrations | | [WS-SEC-001](initiatives/WS-SEC-001-dependency-alert-remediation/STATUS.md) | `WS-SEC-001-01` is complete with patched runtime and tooling dependencies | Handle future security alerts through fresh bounded dependency changes | | [WS-DOCS-001](initiatives/WS-DOCS-001-current-v01-documentation/STATUS.md) | Current v0.1 entry documentation complete | Keep current pages synchronized with merged capability changes | diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/CHUNK_MAP.md b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/CHUNK_MAP.md index a895a780d..3fa57b490 100644 --- a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/CHUNK_MAP.md @@ -7,6 +7,7 @@ | `WS-CI-004-02` | Adopt the shared protocol across all nine reviewer agents and matching skills, with the shared evaluation harness | L1 | Complete through PR #342 | | `WS-CI-004-03` | Close final-head approval and durable merged-state gaps exposed by PR #340 | L1 | Complete | | `WS-CI-004-04` | Require explicit impact-cone and adversarial proof in exact-head reviews | L1 | Complete | +| `WS-CI-004-05` | Require atomic behavior traceability and residual escape analysis | L1 | Complete | This is sequencing guidance, not an implementation queue. Create one bounded contract only when a human explicitly starts that step. One implementation diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/STATUS.md b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/STATUS.md index 3c47bd4a8..f95cd5d6d 100644 --- a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/STATUS.md +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/STATUS.md @@ -2,14 +2,16 @@ ## Current status -`WS-CI-004-01` through `WS-CI-004-03` are merged. `WS-CI-004-04` is complete. +`WS-CI-004-01` through `WS-CI-004-04` are merged. `WS-CI-004-05` is complete. ## Current focus Preserve exact-final-head review convergence, last-push human approval, resolved review conversations, durable final-state wording, explicit impact-cone inspection, and adversarial probes without changing product behavior or -creating a second permission system. +creating a second permission system. `WS-CI-004-05` closes the semantic- +completeness gap exposed when PR #346 initially mapped a compound criterion only +partially. ## Chunk status @@ -20,13 +22,16 @@ creating a second permission system. | `WS-CI-004-02` | Merged | Nine reviewer skills and agents plus deterministic evaluation harness landed through PR #342 | | `WS-CI-004-03` | Complete | Last-push approval, conversation resolution, and non-temporal final-state enforcement | | `WS-CI-004-04` | Complete | Impact-cone and adversarial proof required in reviewer session receipts | +| `WS-CI-004-05` | Complete | Atomic behavior traceability and residual escape analysis | ## Durable current state The shared protocol, schema, target sensor, reviewer adoption, final-head -closure, and explicit review-depth proof are complete. GitHub remains the sole -contribution and merge authority; repository evidence remains advisory. +closure, explicit review-depth proof, and semantic-completeness hardening are +complete. GitHub remains the sole contribution and merge authority; +repository evidence remains advisory. ## Next action -Human review of `WS-CI-004-04`. Do not start another successor automatically. +Preserve `WS-CI-004-05` behavior during human review and merge. Do not start +another successor automatically. diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/chunks/WS-CI-004-05-semantic-review-completeness.md b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/chunks/WS-CI-004-05-semantic-review-completeness.md new file mode 100644 index 000000000..ad8ba29d3 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/chunks/WS-CI-004-05-semantic-review-completeness.md @@ -0,0 +1,100 @@ +# Chunk Contract: WS-CI-004-05 — Semantic Review Completeness + +## Merge state + +- Outcome on merge: `complete` + +## Goal + +Require reviewers to decompose material requirements into independently +checkable behavior atoms and trace each atom through its owner, implementation, +proof, and execution custody before issuing a passing verdict. + +## Why this chunk exists + +Exact-head review, impact-cone inspection, and adversarial probes prevent stale +or shallow receipts, but PR #346 demonstrated that a reviewer can still accept +a polished criterion-to-test table while one material behavior has no named +proof. Review must establish semantic completeness, not merely the presence of +a matrix. + +## Risk class + +L1 — reviewer protocol, skills, agents, evidence schema, and evaluation harness. + +## Allowed files + +```text +.agents/skills/reviewer-evidence-protocol/SKILL.md +.agents/skills/{architecture-review,security-review,qa-review,test-delta-review,docs-review,product-ops-review,ci-integrity-review,reuse-dedup-review,senior-engineer-review}/SKILL.md +.codex/agents/*-reviewer.toml +.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/{CHUNK_MAP.md,STATUS.md,REVIEWER_MATRIX.md} +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/chunks/WS-CI-004-05-semantic-review-completeness.md +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/{CASES.json,EXPECTATIONS.json} +.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/reviews/WS-CI-004-05-external-review-response.md +.agent-loop/CURRENT_STATE.md +scripts/{reviewer_contracts.py,test_reviewer_contracts.py,test_review_target.py} +``` + +## Not allowed + +```text +backend product/runtime behavior or migrations +hosted or committed exact-head receipt custody +new merge authority, automatic merge, or universal reviewer fanout +coverage, CI, branch-protection, or existing test weakening +``` + +## Acceptance criteria + +- [x] Every material criterion is decomposed into atomic observable behaviors, + including actors, resources/tenants, lifecycle states, failures, and + forbidden side effects where relevant. +- [x] Every behavior atom maps to its owner, implementation source, named proof, + and execution custody; missing or narrative-only mappings block PASS. +- [x] Every final review states a residual escape hypothesis and how the reviewer + attempted to falsify it. +- [x] Architecture, security, QA, test-delta, docs, product/ops, CI, reuse, and + senior-review skills and agents contain specialty-specific completeness + probes instead of generic boilerplate alone. +- [x] The canonical receipt schema rejects missing trace rows or residual escape + analysis, requires all trace rows to be verified for a passing verdict, + and advances the breaking receipt contract to schema version 2. +- [x] Realistic multi-file evaluation fixtures reproduce the semantic omission + observed during PR #346 review without leaking the expected answer. +- [x] Session evidence remains advisory and GitHub remains the sole merge authority. + +## Completion evidence + +| Behavior | Owner | Implementation source | Named proof | Execution custody | Result | +|---|---|---|---|---|---| +| Atomize material criteria | Shared reviewer protocol | `reviewer-evidence-protocol/SKILL.md` and nine specialty agents/skills | `test_all_agent_skill_contracts_compose_with_protocol` plus blind `qa-compound-trace` replay | local deterministic tests and exact-head QA review | verified | +| Trace every atom through owner, implementation, proof and custody | Shared protocol and receipt schema | protocol steps 6-7 and receipt `traceability` | receipt schema tests and reviewer-contract mutation tests | local deterministic tests | verified | +| Require residual escape falsification | Shared protocol and receipt schema | protocol step 8 and receipt `residual_escape` | `test_final_verdict_requires_verified_trace_and_closed_escape` | local deterministic tests | verified | +| Preserve specialty-specific depth | Nine specialty reviewer owners | matching skills and `.codex/agents/*-reviewer.toml` | `scripts/reviewer_contracts.py` and exact-head reviewer fanout | local validation and advisory review sessions | verified | +| Reproduce the PR #346 omission | Evaluation harness | `CASES.json` / `EXPECTATIONS.json` | blind `qa-compound-trace` classification plus fixture validation | isolated QA review and local tests | verified | +| Keep evidence advisory | GitHub authority boundary | receipt `custody`, schema closed fields, contract non-goals | authority-field adversarial schema probe and security review | local deterministic tests and advisory review session | verified | + +## Verification commands + +```bash +python3 -m unittest -v scripts.test_reviewer_contracts scripts.test_review_target +python3 scripts/reviewer_contracts.py +python3 scripts/check_chunk_state_sync.py --base-ref origin/main +python3 scripts/check_active_state_projections.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/check_stale_review_contracts.py +git diff --check +``` + +## Required reviewers + +Architecture, security, QA, test-delta, CI integrity, senior engineering, +reuse/dedup, and documentation. + +## Human review focus + +Confirm the change forces semantic proof without turning reviewer prose into a +second permission system or adding unnecessary process ceremony. diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/CASES.json b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/CASES.json index 8866bd14b..79c047535 100644 --- a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/CASES.json +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/CASES.json @@ -26,6 +26,7 @@ {"id":"product_ops-handoff","reviewer":"product_ops","class":"handoff","task":"Review the supplied change within your specialty.","evidence":"The contributor lifecycle is correct, but a CI workflow skips failed tests."}, {"id":"qa-positive","reviewer":"qa","class":"positive","task":"Review the supplied change within your specialty.","evidence":"Acceptance requires denial for an expired lease, but tests cover only the successful active-lease path."}, + {"id":"qa-compound-trace","reviewer":"qa","class":"positive","task":"Review this planning diff and its claimed criterion-to-test coverage. The supplied excerpts are from separate changed files.","evidence":"Contract acceptance criterion: child mutation, binding suspension, unit retirement, competing publication, revocation, replay, cross-project use, wrong session/transaction, close failure, and rollback all fail closed without mutation. Verification table: PREP denial/session/close failure maps to test_policy_publication_authorization.py; child mutation/binding suspension/unit retirement/competing publication maps to test_policy_publication_concurrency.py; duplicate replay maps to test_policy_publication_recovery.py. Trust bundle claims the matrices cover every material acceptance criterion."}, {"id":"qa-negative","reviewer":"qa","class":"negative","task":"Review the supplied change within your specialty.","evidence":"A private refactor preserves observable behavior and all boundary and negative assertions remain unchanged."}, {"id":"qa-stale","reviewer":"qa","class":"stale_replay","task":"Re-evaluate after the supplied update.","evidence":"Prior finding QA-8 reported no expired-lease regression. The new head adds it; replay QA-8 and verify the behavior."}, {"id":"qa-output","reviewer":"qa","class":"output_contract","task":"Assess whether a final review result is possible.","evidence":"Tests ran on head aaa, but the requested final head is bbb."}, diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/EXPECTATIONS.json b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/EXPECTATIONS.json index 208f5ffae..ef77dbbbc 100644 --- a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/EXPECTATIONS.json +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/evaluations/EXPECTATIONS.json @@ -27,6 +27,7 @@ {"case_id":"product_ops-handoff","outcome":"handoff","required_finding_ids":[],"handoff_specialty":"ci_integrity"}, {"case_id":"qa-positive","outcome":"finding","required_finding_ids":[],"handoff_specialty":null}, + {"case_id":"qa-compound-trace","outcome":"finding","required_finding_ids":[],"handoff_specialty":null}, {"case_id":"qa-negative","outcome":"clear","required_finding_ids":[],"handoff_specialty":null}, {"case_id":"qa-stale","outcome":"replayed","required_finding_ids":["QA-8"],"handoff_specialty":null}, {"case_id":"qa-output","outcome":"provisional","required_finding_ids":[],"handoff_specialty":null}, diff --git a/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/reviews/WS-CI-004-05-external-review-response.md b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/reviews/WS-CI-004-05-external-review-response.md new file mode 100644 index 000000000..2853910d7 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-004-review-evidence-integrity/reviews/WS-CI-004-05-external-review-response.md @@ -0,0 +1,43 @@ +# WS-CI-004-05 External Review Response + +## Comments addressed + +- `CR-347-002` Major: acceptance criteria were implemented but remained + unchecked. Valid. Every criterion is now checked and mapped through owner, + implementation source, named proof, execution custody, and result. +- `CR-347-003` Minor: the contract did not list the stale-wording checks or + record reviewer-session closure. Valid. Verification now names both stale + checks, the docs reviewer requires link/stale evidence, and all requested + internal reviewer sessions were closed before this response. +- `CR-347-004` Major: specialty-skill adoption checks did not enforce the full + shared semantic contract. Valid. Stable requirement IDs now cover + atomization, owner, implementation source, named proof, execution custody, + result, residual escape, and fail-closed PASS behavior for both skills and + custom agent TOMLs, with independent mutation proof. +- `CR-347-005` Minor: both receipt test suites needed unavailable and multi-row + PASS rejection. Valid. Both suites now exercise those boundaries while + preserving PROVISIONAL acceptance for unavailable proof. + +## Comments not applied + +- `CR-347-001` Major requested that durable state use `in review` until merge. + Not valid under the repository's atomic chunk-state contract. `AGENTS.md` + requires the same PR to land its declared final outcome and prohibits + `in review`, `pending review`, or `ready for review` as state that can reach + `main`. GitHub open PRs remain the transient-work view; the branch projection + describes the outcome that this PR will atomically land. + +## Human decisions needed + +None beyond normal approval and merge ownership. + +## Commands rerun + +Reviewer contract/schema tests, skill validation, reviewer contract validation, +chunk-state sync, active-state projections, Markdown links, stale wording, +stale review contracts, and diff checks. + +## Remaining risks + +Natural-language semantic completeness still requires reviewer judgment beyond +the structural guarantees of JSON Schema and deterministic contract checks. diff --git a/.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json b/.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json index 2a3a43fa5..b10f42bc5 100644 --- a/.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json +++ b/.agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json @@ -1,12 +1,12 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://workstream.local/schemas/internal-review-receipt-v1.json", + "$id": "https://workstream.local/schemas/internal-review-receipt-v2.json", "title": "Workstream Internal Review Receipt", "type": "object", "additionalProperties": false, - "required": ["schema_version", "custody", "target", "reviewer", "inspections", "evidence", "impact_cone", "adversarial_probes", "findings", "uncertainty", "freshness", "verdict"], + "required": ["schema_version", "custody", "target", "reviewer", "inspections", "evidence", "impact_cone", "adversarial_probes", "traceability", "residual_escape", "findings", "uncertainty", "freshness", "verdict"], "properties": { - "schema_version": {"const": 1}, + "schema_version": {"const": 2}, "custody": {"const": "advisory_session"}, "target": {"$ref": "#/$defs/target"}, "reviewer": { @@ -67,6 +67,34 @@ } } }, + "traceability": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["criterion", "behavior", "owner", "implementation_source", "proof_source", "execution_custody", "result"], + "properties": { + "criterion": {"type": "string", "minLength": 1}, + "behavior": {"type": "string", "minLength": 1}, + "owner": {"type": "string", "minLength": 1}, + "implementation_source": {"type": "string", "minLength": 1}, + "proof_source": {"type": "string", "minLength": 1}, + "execution_custody": {"type": "string", "minLength": 1}, + "result": {"enum": ["verified", "missing", "unavailable"]} + } + } + }, + "residual_escape": { + "type": "object", + "additionalProperties": false, + "required": ["hypothesis", "method", "result"], + "properties": { + "hypothesis": {"type": "string", "minLength": 1}, + "method": {"type": "string", "minLength": 1}, + "result": {"enum": ["falsified", "survives", "unavailable"]} + } + }, "findings": { "type": "array", "items": {"$ref": "#/$defs/finding"} @@ -111,6 +139,16 @@ "properties": {"result": {"const": "pass"}} }, "minContains": 1 + }, + "traceability": { + "items": { + "type": "object", + "required": ["result"], + "properties": {"result": {"const": "verified"}} + } + }, + "residual_escape": { + "properties": {"result": {"const": "falsified"}} } } } diff --git a/.agents/skills/architecture-review/SKILL.md b/.agents/skills/architecture-review/SKILL.md index 0924c225e..3d259447e 100644 --- a/.agents/skills/architecture-review/SKILL.md +++ b/.agents/skills/architecture-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged owners and consumers, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-architecture findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -32,6 +34,13 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. Architecture drift is blocking even when tests pass. +## Completeness probe + +Build an owner/consumer/public-port matrix for every changed boundary. Trace +construction, runtime composition, persistence ownership, and every known +consumer. Search for private imports and parallel paths. Treat an uninspected +consumer or unmapped ownership transition as missing proof. + ## Output ```text @@ -40,6 +49,7 @@ Protocol envelope: target / run / evidence / findings / uncertainty / freshness Boundary violations: Abstraction risks: Coupling risks: +Atomic traceability and residual escape: Simpler alternative: Required fixes: ``` diff --git a/.agents/skills/ci-integrity-review/SKILL.md b/.agents/skills/ci-integrity-review/SKILL.md index 478e29058..1dec02aa6 100644 --- a/.agents/skills/ci-integrity-review/SKILL.md +++ b/.agents/skills/ci-integrity-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged workflows and commands, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-CI findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Inspect @@ -35,6 +37,13 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Removed lint/typecheck/test gate. - Package script changed to hide errors. +## Completeness probe + +Trace each required gate from trigger and path filter through job dependency, +command, exit propagation, artifact/coverage aggregation, and protected-branch +status. Probe cancellation, skipped-job, empty-selection, and rerun behavior; +green status alone is not proof that the intended command ran. + ## Output ```text @@ -42,6 +51,7 @@ Result: PASS / PASS WITH LOW RISKS / FAIL Protocol envelope: target / run / evidence / findings / uncertainty / freshness CI files changed: Integrity concerns: +Gate traceability and residual escape: Required fixes: ``` diff --git a/.agents/skills/docs-review/SKILL.md b/.agents/skills/docs-review/SKILL.md index ddd75d81e..235d4e094 100644 --- a/.agents/skills/docs-review/SKILL.md +++ b/.agents/skills/docs-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged entry and authority pages, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-documentation findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -27,6 +29,17 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - README or runbook updates - Architecture decision records +## Completeness probe + +Map each changed fact to every current entry page, canonical specification, +status projection, example, and historical record that could contradict it. +Verify lifecycle state and tense independently; a generally correct document +does not excuse one stale authority claim. +Run the repository Markdown-link and stale-wording checks when those scripts +exist, and record their executed results. Confirm all requested sub-agent +sessions are closed as review evidence; this is a state observation, not a +shell command. + ## Output ```text @@ -35,6 +48,7 @@ Protocol envelope: target / run / evidence / findings / uncertainty / freshness Docs required: yes/no Missing docs: Suggested doc locations: +Fact-to-document traceability and residual escape: Required fixes: ``` diff --git a/.agents/skills/product-ops-review/SKILL.md b/.agents/skills/product-ops-review/SKILL.md index a5e08b262..898ccac42 100644 --- a/.agents/skills/product-ops-review/SKILL.md +++ b/.agents/skills/product-ops-review/SKILL.md @@ -15,6 +15,8 @@ findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off engineering-only findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -36,6 +38,13 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Confirm user-facing review decisions remain `accept`, `needs_revision`, and `reject`. - Confirm out-of-band guidance becomes a guide, policy, template, checker, or ADR before it is enforceable. +## Completeness probe + +Trace every lifecycle atom through actor, prerequisite, state transition, +failure/retry/revision path, evidence produced, and downstream consumer. Include +separation-of-duty and cross-project variants. Missing operator recovery or +user-visible failure behavior blocks a passing verdict. + ## Output ```text @@ -45,6 +54,7 @@ Operator workflow risks: Contributor/reviewer workflow risks: Payment/reputation risks: Naming or wording drift: +Lifecycle traceability and residual escape: Required fixes: ``` diff --git a/.agents/skills/qa-review/SKILL.md b/.agents/skills/qa-review/SKILL.md index 9dc91cde2..7ec9561f6 100644 --- a/.agents/skills/qa-review/SKILL.md +++ b/.agents/skills/qa-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged behavior and tests, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-QA findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -33,6 +35,9 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Do not approve because code looks clean. - Do not request broad rewrites unless necessary. - Tie findings to acceptance criteria or observable behavior. +- Split compound criteria into behavior atoms. For each atom name the owner, + implementation source, exact test, execution custody, and observed result. +- A test module name, broad command, or prose promise is not proof of every atom. ## Output @@ -41,6 +46,8 @@ Result: PASS / PASS WITH LOW RISKS / FAIL Protocol envelope: target / run / evidence / findings / uncertainty / freshness Covered criteria: Missing criteria: +Atomic criterion-to-proof traceability: +residual escape hypothesis: Findings: Required fixes: Suggested tests: diff --git a/.agents/skills/reuse-dedup-review/SKILL.md b/.agents/skills/reuse-dedup-review/SKILL.md index 2d87e5fcc..36e9a415c 100644 --- a/.agents/skills/reuse-dedup-review/SKILL.md +++ b/.agents/skills/reuse-dedup-review/SKILL.md @@ -15,6 +15,8 @@ findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-reuse findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -25,6 +27,13 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Similar abstractions now exist in multiple places. - Shared behavior belongs in existing module. +## Completeness probe + +For each new helper, schema, policy path, or abstraction, search by behavior, +types, call sites, and ownership—not name alone. Map the new responsibility to +the closest existing public owner and explain why reuse or extension is or is +not valid. Unsearched likely owners are missing proof. + ## Output ```text @@ -32,6 +41,7 @@ Result: PASS / PASS WITH LOW RISKS / FAIL Protocol envelope: target / run / evidence / findings / uncertainty / freshness Possible duplicates: Existing code to reuse: +Responsibility traceability and residual escape: Required fixes: ``` diff --git a/.agents/skills/reviewer-evidence-protocol/SKILL.md b/.agents/skills/reviewer-evidence-protocol/SKILL.md index 044f44b2c..b84d4c6de 100644 --- a/.agents/skills/reviewer-evidence-protocol/SKILL.md +++ b/.agents/skills/reviewer-evidence-protocol/SKILL.md @@ -22,9 +22,22 @@ their own questions; they do not replace or duplicate this protocol. verdict. State the failure or bypass hypothesis, the inspection or command used to test it, and the observed result. Passing tests alone are not an adversarial probe. -6. Run the same `python3 scripts/review_target.py` command again immediately +6. Atomize every material acceptance criterion or claimed invariant into + independently observable behaviors. Include relevant actor/context, action, + resource or tenant, lifecycle state, failure mode, and forbidden side effect; + do not preserve a compound sentence as one row when its parts can fail + independently. +7. Build a traceability row for every behavior atom: criterion, behavior, + owner, implementation source, proof source, execution custody, and + verification result. For planning-only changes, name the future symbol/path + and future test. Narrative coverage, a module name without a test, or one + test mapped to an unexamined compound criterion is incomplete. +8. State a residual escape hypothesis: the most plausible material defect that + could still pass the named proof. Attempt to falsify it through a concrete + inspection or command and record the result. +9. Run the same `python3 scripts/review_target.py` command again immediately before the verdict. -7. Compare both snapshots before constructing the receipt. The receipt stores +10. Compare both snapshots before constructing the receipt. The receipt stores their matching target triple once; its start/end inspections cannot redefine that target. A final verdict is valid only when the snapshots match and both worktrees are clean. Dirty state permits provisional findings only. @@ -38,6 +51,7 @@ their own questions; they do not replace or duplicate this protocol. - Replay every prior finding on the final target. Record its disposition and verification; never silently drop it. - State uncertainty and unavailable proof explicitly. +- A missing, unverified, or merely narrative trace row blocks a passing verdict. - Route another specialty's issue to that reviewer; do not invent its verdict. ## Verdict @@ -51,8 +65,9 @@ implementation, merge, or Workstream product lifecycle decisions. ## Output Return the exact target, reviewer/run identity, start/end inspection, evidence, -impact cone, adversarial probes, findings and replay dispositions, uncertainty, -freshness, and verdict. Use the canonical schema and templates. Receipts remain +impact cone, adversarial probes, atomic traceability rows, residual escape +analysis, findings and replay dispositions, uncertainty, freshness, and verdict. +Use the canonical schema and templates. Receipts remain private out-of-tree session evidence written only by the orchestrator; a PR summary may mirror a verdict but is neither receipt custody nor an attestation. Do not write receipt custody from a reviewer. diff --git a/.agents/skills/security-review/SKILL.md b/.agents/skills/security-review/SKILL.md index f17418014..a84f01d85 100644 --- a/.agents/skills/security-review/SKILL.md +++ b/.agents/skills/security-review/SKILL.md @@ -15,6 +15,8 @@ findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-security findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -39,6 +41,14 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Findings must be concrete. - Critical/High findings block PR. +## Completeness probe + +Decompose each protected behavior across actor/context, action, project or +resource, lifecycle state, failure mode, and forbidden evidence/side effect. +Trace both allow and deny paths, including cross-tenant/resource substitution, +replay, revocation, and transaction failure. Missing negative-path proof blocks +a passing verdict. + ## Output For each finding: @@ -52,6 +62,8 @@ Suggested fix: Blocks PR: yes/no ``` +Include atomic traceability and the residual escape hypothesis for authorization. + End with PASS / PASS WITH LOW RISKS / FAIL. Protocol envelope: target / run / evidence / findings / uncertainty / freshness. diff --git a/.agents/skills/senior-engineer-review/SKILL.md b/.agents/skills/senior-engineer-review/SKILL.md index 37a0d5bd0..1ac8810e6 100644 --- a/.agents/skills/senior-engineer-review/SKILL.md +++ b/.agents/skills/senior-engineer-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged ownership and operational context, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off specialty findings without inventing their verdicts. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -28,6 +30,13 @@ Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. - Ownership in 3-6 months - Fit with existing conventions +## Completeness probe + +Trace each responsibility to one owner, one failure boundary, one operational +signal, and one maintenance path. Inspect size, branching, transaction/error +handling, and rollback independently. State the most plausible maintenance or +operational defect still hidden by otherwise green proof. + ## Output ```text @@ -36,6 +45,7 @@ Protocol envelope: target / run / evidence / findings / uncertainty / freshness Maintainability risks: Simplicity improvements: Operational concerns: +Responsibility traceability and residual escape: Required fixes: ``` diff --git a/.agents/skills/test-delta-review/SKILL.md b/.agents/skills/test-delta-review/SKILL.md index 7e1f4594e..71982393f 100644 --- a/.agents/skills/test-delta-review/SKILL.md +++ b/.agents/skills/test-delta-review/SKILL.md @@ -14,6 +14,8 @@ inspect relevant unchanged tests and behavior owners, replay prior findings, separate executed from inspected evidence, state uncertainty and freshness, and hand off non-test findings without inventing another specialty's verdict. Use canonical reviewer IDs from the initiative `REVIEWER_MATRIX.md` in handoffs. +Atomize every material criterion. For every behavior atom, record its owner, implementation source, named proof, +execution custody, and result. Missing or narrative-only rows block PASS. ## Focus @@ -34,6 +36,13 @@ For every changed test: - Coverage lowered without approval. - Bug fix without regression coverage when feasible. +## Completeness probe + +Map each changed or claimed behavior atom to the exact assertion that would fail +if that atom regressed. Check actor/resource/state/failure variants separately, +and compare with the old behavior. A test name without a discriminating +assertion is not verified traceability. + ## Output ```text @@ -43,6 +52,8 @@ Tests added: Tests modified: Tests removed/skipped: Weakening concerns: +Atomic behavior/assertion traceability: +residual escape hypothesis: Required fixes: ``` diff --git a/.codex/agents/architecture-reviewer.toml b/.codex/agents/architecture-reviewer.toml index 5915ebb5b..8d409a585 100644 --- a/.codex/agents/architecture-reviewer.toml +++ b/.codex/agents/architecture-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-architecture findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For architecture, build an owner/consumer/public-port matrix and inspect every known consumer. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/ci-integrity-reviewer.toml b/.codex/agents/ci-integrity-reviewer.toml index cb11adab5..22b9d9bf9 100644 --- a/.codex/agents/ci-integrity-reviewer.toml +++ b/.codex/agents/ci-integrity-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-CI findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For CI, trace trigger through command, exit propagation, aggregation, and required status. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/docs-reviewer.toml b/.codex/agents/docs-reviewer.toml index ea08d770e..45cd72613 100644 --- a/.codex/agents/docs-reviewer.toml +++ b/.codex/agents/docs-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-documentation findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For docs, map each changed fact across current entry pages, specifications, and state projections. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/product-ops-reviewer.toml b/.codex/agents/product-ops-reviewer.toml index 9aee5658d..90a32e7c1 100644 --- a/.codex/agents/product-ops-reviewer.toml +++ b/.codex/agents/product-ops-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off engineering-only findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For product/ops, trace actor, prerequisite, transition, failure/revision, evidence, and consumer. You must review the current branch against the base branch, any applicable plan or chunk contract, Workstream product docs/templates, and repository engineering guidance. Treat .agent-loop/policies/ as engineering process policy. Do not rename or replace Workstream product contracts such as project guide, submission artifact policy, checker policy, review policy, revision policy, or payment policy. diff --git a/.codex/agents/qa-reviewer.toml b/.codex/agents/qa-reviewer.toml index 5040701d2..a64495e20 100644 --- a/.codex/agents/qa-reviewer.toml +++ b/.codex/agents/qa-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-QA findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For QA, split compound acceptance criteria and require a discriminating exact test per atom. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/reuse-dedup-reviewer.toml b/.codex/agents/reuse-dedup-reviewer.toml index ab02baf81..ec72090d9 100644 --- a/.codex/agents/reuse-dedup-reviewer.toml +++ b/.codex/agents/reuse-dedup-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-reuse findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For reuse, search likely owners by behavior, type, and call site rather than name alone. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/security-reviewer.toml b/.codex/agents/security-reviewer.toml index 6afb9cc56..dfaba37ae 100644 --- a/.codex/agents/security-reviewer.toml +++ b/.codex/agents/security-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-security findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For security, enumerate actor/context, action, resource/tenant, state, failure, and side effect. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/senior-engineer-reviewer.toml b/.codex/agents/senior-engineer-reviewer.toml index d963a5e41..09a390487 100644 --- a/.codex/agents/senior-engineer-reviewer.toml +++ b/.codex/agents/senior-engineer-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off specialty findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For engineering, trace each responsibility to one owner, failure boundary, signal, and maintenance path. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/.codex/agents/test-delta-reviewer.toml b/.codex/agents/test-delta-reviewer.toml index 2679bb04e..ce5060497 100644 --- a/.codex/agents/test-delta-reviewer.toml +++ b/.codex/agents/test-delta-reviewer.toml @@ -16,6 +16,10 @@ Run `python3 scripts/review_target.py` at start and end. Do not issue a final ve unless the exact target matches and both snapshots are clean. Replay every prior finding, distinguish executed from inspected evidence, state uncertainty and freshness, and hand off non-test findings without deciding them. +Atomize material criteria and invariants. Produce traceability from each behavior +atom to owner, implementation source, named proof, execution custody, and result. +State and test a residual escape hypothesis. Missing or narrative-only rows block PASS. +For tests, name the exact assertion that fails for every behavior atom and compare old behavior. You must review the current branch against the base branch, any applicable plan or chunk contract, and repository engineering guidance. Focus areas: diff --git a/scripts/reviewer_contracts.py b/scripts/reviewer_contracts.py index d43c13fb6..7758dd9f3 100644 --- a/scripts/reviewer_contracts.py +++ b/scripts/reviewer_contracts.py @@ -21,6 +21,22 @@ RECEIPT_SCHEMA_PATH = ROOT / ".agent-loop/templates/INTERNAL_REVIEW_RECEIPT.schema.json" CASE_CLASSES = {"positive", "negative", "stale_replay", "output_contract", "handoff"} OUTCOMES = {"finding", "clear", "replayed", "provisional", "handoff"} +SEMANTIC_SKILL_REQUIREMENTS = { + "semantic.atomization": "Atomize every material criterion", + "semantic.ownership": "record its owner", + "semantic.implementation_source": "implementation source", + "semantic.named_proof": "named proof", + "semantic.execution_custody": "execution custody", + "semantic.result": "and result", + "semantic.traceability": "traceability", + "semantic.residual_escape": "residual escape", + "semantic.fail_closed": "Missing or narrative-only rows block PASS", +} +SEMANTIC_AGENT_REQUIREMENTS = { + **SEMANTIC_SKILL_REQUIREMENTS, + "semantic.atomization": "Atomize material criteria", + "semantic.ownership": "to owner", +} MATRIX_ROW = re.compile( r"^\|[^|]+\|\s*`([^`]+)`\s*\|\s*`\.codex/agents/([^`/]+\.toml)`\s*\|" r"\s*`\.agents/skills/([^`/]+)/SKILL\.md`\s*\|$", @@ -86,9 +102,16 @@ def contract_failures(root: Path = ROOT) -> list[str]: "freshness", "hand off", "python3 scripts/review_target.py", + "Atomize material criteria", + "traceability", + "residual escape hypothesis", + "Missing or narrative-only rows block PASS", ): if token not in normalized_agent: failures.append(f"{reviewer}: agent missing {token!r}") + for requirement_id, token in SEMANTIC_AGENT_REQUIREMENTS.items(): + if token not in normalized_agent: + failures.append(f"{reviewer}: agent missing {requirement_id} ({token!r})") for token in ( "reviewer-evidence-protocol", "exact target", @@ -103,6 +126,9 @@ def contract_failures(root: Path = ROOT) -> list[str]: ): if token not in normalized_skill: failures.append(f"{reviewer}: skill missing {token!r}") + for requirement_id, token in SEMANTIC_SKILL_REQUIREMENTS.items(): + if token not in normalized_skill: + failures.append(f"{reviewer}: skill missing {requirement_id} ({token!r})") if agent_path.as_posix().replace(f"{root.as_posix()}/", "") not in matrix: failures.append(f"{reviewer}: agent absent from matrix") if skill_path.as_posix().replace(f"{root.as_posix()}/", "") not in matrix: diff --git a/scripts/test_review_target.py b/scripts/test_review_target.py index f733c10a6..76402b203 100644 --- a/scripts/test_review_target.py +++ b/scripts/test_review_target.py @@ -172,7 +172,7 @@ def setUpClass(cls) -> None: sha = "a" * 40 target = {"base_sha": sha, "merge_base_sha": sha, "head_sha": sha} cls.receipt = { - "schema_version": 1, + "schema_version": 2, "custody": "advisory_session", "target": target, "reviewer": {"specialty": "security", "run_id": "run-1"}, @@ -191,6 +191,22 @@ def setUpClass(cls) -> None: "result": "pass", } ], + "traceability": [ + { + "criterion": "deny invalid input", + "behavior": "invalid input cannot bypass denial", + "owner": "security", + "implementation_source": "owner.py:Owner", + "proof_source": "negative test", + "execution_custody": "unit test", + "result": "verified", + } + ], + "residual_escape": { + "hypothesis": "another invalid form bypasses denial", + "method": "negative test inventory", + "result": "falsified", + }, "findings": [], "uncertainty": [], "freshness": "current", @@ -218,6 +234,8 @@ def test_required_identity_target_and_inspection_fields(self) -> None: "inspections", "impact_cone", "adversarial_probes", + "traceability", + "residual_escape", ): with self.subTest(key=key): self.assert_invalid(lambda receipt, key=key: receipt.pop(key)) @@ -226,6 +244,36 @@ def test_required_identity_target_and_inspection_fields(self) -> None: self.assert_invalid(lambda receipt: receipt["inspections"].pop("end")) self.assert_invalid(lambda receipt: receipt.__setitem__("impact_cone", [])) self.assert_invalid(lambda receipt: receipt.__setitem__("adversarial_probes", [])) + self.assert_invalid(lambda receipt: receipt.__setitem__("traceability", [])) + + def test_final_verdict_requires_verified_trace_and_closed_escape(self) -> None: + for result in ("missing", "unavailable"): + with self.subTest(trace_result=result): + self.assert_invalid( + lambda receipt, result=result: receipt["traceability"][0].__setitem__( + "result", result + ) + ) + for result in ("survives", "unavailable"): + with self.subTest(escape_result=result): + self.assert_invalid( + lambda receipt, result=result: receipt["residual_escape"].__setitem__( + "result", result + ) + ) + + def append_unverified_row(receipt): + row = copy.deepcopy(receipt["traceability"][0]) + row["behavior"] = "second independent behavior" + row["result"] = "missing" + receipt["traceability"].append(row) + + self.assert_invalid(append_unverified_row) + receipt = copy.deepcopy(self.receipt) + receipt["verdict"] = "PROVISIONAL" + receipt["traceability"][0]["result"] = "unavailable" + receipt["residual_escape"]["result"] = "unavailable" + jsonschema.validate(receipt, self.schema) def test_closed_tokens_and_unknown_claims_are_rejected(self) -> None: self.assert_invalid(lambda receipt: receipt.__setitem__("verdict", "APPROVED")) diff --git a/scripts/test_reviewer_contracts.py b/scripts/test_reviewer_contracts.py index 32aea19f8..81c303049 100644 --- a/scripts/test_reviewer_contracts.py +++ b/scripts/test_reviewer_contracts.py @@ -8,7 +8,12 @@ import unittest from pathlib import Path -from scripts.reviewer_contracts import CASE_CLASSES, REVIEWERS +from scripts.reviewer_contracts import ( + CASE_CLASSES, + REVIEWERS, + SEMANTIC_AGENT_REQUIREMENTS, + SEMANTIC_SKILL_REQUIREMENTS, +) from scripts.reviewer_contracts import contract_failures, fixture_failures, load_json, main from scripts.reviewer_contracts import output_failures, output_set_failures from scripts.reviewer_contracts import CASES_PATH, EXPECTATIONS_PATH @@ -58,6 +63,38 @@ def test_missing_protocol_output_or_handoff_contract_fails(self) -> None: finally: temporary.cleanup() + def test_each_semantic_skill_requirement_is_independently_enforced(self) -> None: + for requirement_id, token in SEMANTIC_SKILL_REQUIREMENTS.items(): + with self.subTest(requirement_id=requirement_id): + temporary, root = self.copied_contract_root() + try: + skill = root / ".agents/skills/architecture-review/SKILL.md" + skill.write_text( + skill.read_text(encoding="utf-8").replace(token, "removed"), + encoding="utf-8", + ) + self.assertTrue( + any(requirement_id in failure for failure in contract_failures(root)) + ) + finally: + temporary.cleanup() + + def test_each_semantic_agent_requirement_is_independently_enforced(self) -> None: + for requirement_id, token in SEMANTIC_AGENT_REQUIREMENTS.items(): + with self.subTest(requirement_id=requirement_id): + temporary, root = self.copied_contract_root() + try: + agent = root / ".codex/agents/architecture-reviewer.toml" + agent.write_text( + agent.read_text(encoding="utf-8").replace(token, "removed"), + encoding="utf-8", + ) + self.assertTrue( + any(requirement_id in failure for failure in contract_failures(root)) + ) + finally: + temporary.cleanup() + def test_agent_handoff_contract_and_matrix_ids_are_enforced(self) -> None: temporary, root = self.copied_contract_root() try: @@ -100,7 +137,7 @@ def test_matrix_agent_and_skill_pairs_are_one_to_one(self) -> None: def test_every_reviewer_has_every_blind_case_class(self) -> None: self.assertEqual(fixture_failures(self.cases, None), []) rows = self.cases["cases"] - self.assertEqual(len(rows), len(REVIEWERS) * len(CASE_CLASSES)) + self.assertGreaterEqual(len(rows), len(REVIEWERS) * len(CASE_CLASSES)) def test_missing_reviewer_or_case_class_fails(self) -> None: cases = copy.deepcopy(self.cases) @@ -202,7 +239,7 @@ def test_output_requires_envelope_replay_and_exact_handoff(self) -> None: "handoff_specialty": "ci_integrity", } receipt = { - "schema_version": 1, + "schema_version": 2, "custody": "advisory_session", "target": {"base_sha": "a" * 40, "merge_base_sha": "a" * 40, "head_sha": "a" * 40}, "reviewer": {"specialty": "architecture", "run_id": "eval-1"}, @@ -217,6 +254,10 @@ def test_output_requires_envelope_replay_and_exact_handoff(self) -> None: "adversarial_probes": [ {"hypothesis": "case bypass", "method": "inspect raw case", "result": "pass"} ], + "traceability": [ + {"criterion": "routing", "behavior": "route finding", "owner": "architecture", "implementation_source": "raw case", "proof_source": "inspection", "execution_custody": "review session", "result": "verified"} + ], + "residual_escape": {"hypothesis": "a second route is hidden", "method": "inspect supplied evidence", "result": "falsified"}, "findings": [ {"id": "ARCH-7", "severity": "Medium", "location": "case", "blocks_pr": False, "disposition": "fixed", "verification": "replayed"} ], @@ -252,6 +293,8 @@ def test_output_requires_envelope_replay_and_exact_handoff(self) -> None: ("target", "base_sha"), ("reviewer", "run_id"), ("inspections", "end"), + ("traceability",), + ("residual_escape",), ("uncertainty",), ("freshness",), ("verdict",), @@ -269,6 +312,22 @@ def test_output_requires_envelope_replay_and_exact_handoff(self) -> None: "output: receipt must separate executed and inspected evidence", output_failures(output, expectation, broken_receipt), ) + passing_receipt = copy.deepcopy(receipt) + passing_receipt["inspections"] = {"start": {"cleanliness": "clean"}, "end": {"cleanliness": "clean"}} + passing_receipt["verdict"] = "PASS" + for result in ("missing", "unavailable"): + passing_receipt["traceability"][0]["result"] = result + self.assertTrue(output_failures(output, expectation, passing_receipt)) + passing_receipt["traceability"][0]["result"] = "verified" + second_row = copy.deepcopy(passing_receipt["traceability"][0]) + second_row["behavior"] = "second independent behavior" + second_row["result"] = "missing" + passing_receipt["traceability"].append(second_row) + self.assertTrue(output_failures(output, expectation, passing_receipt)) + passing_receipt["traceability"].pop() + for result in ("survives", "unavailable"): + passing_receipt["residual_escape"]["result"] = result + self.assertTrue(output_failures(output, expectation, passing_receipt)) def test_positive_finding_requires_stable_receipt_finding(self) -> None: expectation = { @@ -287,7 +346,7 @@ def test_positive_finding_requires_stable_receipt_finding(self) -> None: "handoff_specialty": None, } receipt = { - "schema_version": 1, + "schema_version": 2, "custody": "advisory_session", "target": {"base_sha": "a" * 40, "merge_base_sha": "a" * 40, "head_sha": "a" * 40}, "reviewer": {"specialty": "architecture", "run_id": "eval-positive"}, @@ -302,6 +361,10 @@ def test_positive_finding_requires_stable_receipt_finding(self) -> None: "adversarial_probes": [ {"hypothesis": "case bypass", "method": "inspect raw case", "result": "pass"} ], + "traceability": [ + {"criterion": "finding", "behavior": "detect defect", "owner": "architecture", "implementation_source": "raw case", "proof_source": "inspection", "execution_custody": "review session", "result": "verified"} + ], + "residual_escape": {"hypothesis": "defect is concealed", "method": "inspect supplied evidence", "result": "falsified"}, "findings": [], "uncertainty": [], "freshness": "current",