Skip to content

Harden findings-report producer constraints - #218

Merged
Jesper Schulz-Wedde (JesperSchulz) merged 1 commit into
mainfrom
gggdttt-harden-findings-semantics
Oct 5, 2026
Merged

Jesper Schulz-Wedde (JesperSchulz) merged 1 commit into
mainfrom
gggdttt-harden-findings-semantics

Conversation

@gggdttt

Copy link
Copy Markdown
Collaborator

Summary

Prevent report-contract failures observed in BC-Bench run 37293252219 without adding product review heuristics or consumer repair paths.

  • Require every finding ID to match ^[^#]+$, rejecting citation paths with appended #location fragments.
  • When references is empty, allow only confidence: medium|low and severity: minor|info. Knowledge-backed findings retain all existing confidence and severity values.
  • Add a concise DO pre-emission checklist: copy citation IDs verbatim without suffixes, respect uncited caps, and verify 1-based locations against the final source snapshot and its line count, never diff/patch-relative numbering. Use valid 1-based placeholders in the output example.

Compatibility and boundaries

The shared finding definition applies to top-level and nested reports. No agent-prefix regex is added: leaf and root-owned agent:<slug> IDs and rolled-up <leaf-id>:agent:<slug> IDs remain compatible. Existing provenance and composition checks remain unchanged.

JSON Schema cannot compare id with references[0].path or inspect source files. The existing Validate-FindingsReport.ps1 semantic gate remains responsible for exact citation equality, reference existence/retrieval, source scope, and final-file line/range bounds. BC-ALAgents must continue supplying the authoritative source snapshot and enforcing the complete acceptance gate. Invalid uncited confidence now fails at the schema gate rather than the later AGENT_CONFIDENCE_INVALID check.

No silent ID rewriting, confidence/severity downgrades, location clamping, or widened bounded normalization is introduced. Stable protocol changes require both maintainers' approval.

Validation

The new schema regression failed against the original schema and passes with these constraints. Tests cover every severity/confidence combination for knowledge-backed and uncited findings across leaf, root-owned, rolled-up, and nested shapes; fragment IDs at leaf/root/nested positions; and a 22-line source that accepts line 22 but rejects line 44 and an out-of-bounds range, even with bounded normalization enabled. Non-fragment citation mismatches remain semantic failures, and rejected raw payloads remain unchanged.

Passed locally:

  • pwsh .\tools\Test-ReviewContract.ps1 -Root .
  • pwsh .\tools\Test-ReviewFixtures.ps1 -Root . — 228 cases, 114/334 paired articles, 20 leaf domains.
  • pwsh .\.github\scripts\Test-SkillIndex.ps1 -Root . — schema examples and 19 declared review leaves.
  • pwsh .\.github\scripts\Test-KnowledgeIndex.ps1 -Root . — 402 articles; bounded retrieval round-tripped 691 samples.
  • python -X utf8 .github\scripts\validate_frontmatter.py --root . — 0 errors, 2 existing keyword-count warnings.
  • PowerShell AST parsing and git diff --check.

No pre-commit configuration or dedicated formatter is defined in this repository. Generated indexes were exercised only in test-managed temporary directories; none is committed. The diff is limited to the shared schema, DO, and the existing contract test script.

Reject fragment-bearing finding IDs and cap uncited confidence and severity in the shared schema. Require final-source verification before emission and cover leaf/root compatibility and fail-closed source bounds.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6d304ab8e9060625ac94a3a4ed66751ec926b878. The change is sound: the shared finding schema rejects fragment-bearing IDs and applies the existing medium/low confidence and minor/info severity caps when references are empty, including nested, root-owned and rolled-up agent findings. Knowledge-backed findings retain their existing values, and role-specific agent prefixes remain compatible. Exact citation equality and authoritative source bounds remain semantic checks; no consumer repair or widened normalization is introduced. The pre-emission checklist and 1-based example agree with that contract. The regression matrix covers the schema boundaries and preserves existing acceptance/composition behavior, and exact-head contract, fixture and skill/schema integration pass. All GitHub checks are green; no merge-critical issue found. Because this touches the stable shared schema/DO protocol, the required second maintainer approval remains necessary before merge.

@JesperSchulz
Jesper Schulz-Wedde (JesperSchulz) merged commit 7726d5d into main Oct 5, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants