Repository navigation
Harden findings-report producer constraints - #218
Conversation
Reject fragment-bearing finding IDs and cap uncited confidence and severity in the shared schema. Require final-source verification before emission and cover leaf/root compatibility and fail-closed source bounds. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Jesper Schulz-Wedde (JesperSchulz)
left a comment
There was a problem hiding this comment.
Reviewed 6d304ab8e9060625ac94a3a4ed66751ec926b878. The change is sound: the shared finding schema rejects fragment-bearing IDs and applies the existing medium/low confidence and minor/info severity caps when references are empty, including nested, root-owned and rolled-up agent findings. Knowledge-backed findings retain their existing values, and role-specific agent prefixes remain compatible. Exact citation equality and authoritative source bounds remain semantic checks; no consumer repair or widened normalization is introduced. The pre-emission checklist and 1-based example agree with that contract. The regression matrix covers the schema boundaries and preserves existing acceptance/composition behavior, and exact-head contract, fixture and skill/schema integration pass. All GitHub checks are green; no merge-critical issue found. Because this touches the stable shared schema/DO protocol, the required second maintainer approval remains necessary before merge.
Summary
Prevent report-contract failures observed in BC-Bench run 37293252219 without adding product review heuristics or consumer repair paths.
^[^#]+$, rejecting citation paths with appended#locationfragments.referencesis empty, allow onlyconfidence: medium|lowandseverity: minor|info. Knowledge-backed findings retain all existing confidence and severity values.Compatibility and boundaries
The shared finding definition applies to top-level and nested reports. No agent-prefix regex is added: leaf and root-owned
agent:<slug>IDs and rolled-up<leaf-id>:agent:<slug>IDs remain compatible. Existing provenance and composition checks remain unchanged.JSON Schema cannot compare
idwithreferences[0].pathor inspect source files. The existingValidate-FindingsReport.ps1semantic gate remains responsible for exact citation equality, reference existence/retrieval, source scope, and final-file line/range bounds. BC-ALAgents must continue supplying the authoritative source snapshot and enforcing the complete acceptance gate. Invalid uncited confidence now fails at the schema gate rather than the laterAGENT_CONFIDENCE_INVALIDcheck.No silent ID rewriting, confidence/severity downgrades, location clamping, or widened bounded normalization is introduced. Stable protocol changes require both maintainers' approval.
Validation
The new schema regression failed against the original schema and passes with these constraints. Tests cover every severity/confidence combination for knowledge-backed and uncited findings across leaf, root-owned, rolled-up, and nested shapes; fragment IDs at leaf/root/nested positions; and a 22-line source that accepts line 22 but rejects line 44 and an out-of-bounds range, even with bounded normalization enabled. Non-fragment citation mismatches remain semantic failures, and rejected raw payloads remain unchanged.
Passed locally:
pwsh .\tools\Test-ReviewContract.ps1 -Root .pwsh .\tools\Test-ReviewFixtures.ps1 -Root .— 228 cases, 114/334 paired articles, 20 leaf domains.pwsh .\.github\scripts\Test-SkillIndex.ps1 -Root .— schema examples and 19 declared review leaves.pwsh .\.github\scripts\Test-KnowledgeIndex.ps1 -Root .— 402 articles; bounded retrieval round-tripped 691 samples.python -X utf8 .github\scripts\validate_frontmatter.py --root .— 0 errors, 2 existing keyword-count warnings.git diff --check.No pre-commit configuration or dedicated formatter is defined in this repository. Generated indexes were exercised only in test-managed temporary directories; none is committed. The diff is limited to the shared schema, DO, and the existing contract test script.