Skip to content

Summarize risk findings for review#18

Merged
heznpc merged 1 commit into
mainfrom
feat/risk-summary
Jun 26, 2026
Merged

Summarize risk findings for review#18
heznpc merged 1 commit into
mainfrom
feat/risk-summary

Conversation

@heznpc

@heznpc heznpc commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • add riskSummaries so human-facing reports group keyword risk findings by label
  • keep full riskFindings in JSON for automation and --fail-on risk
  • make corpus validation label risk_summary findings instead of every line-level risk match
  • update validation/readme copy around risk-summary usefulness and underlying risk load

Real corpus check

  • corpus: 64 real instruction documents, 51 roots, 13 holdout documents, 3,359 lines
  • before risk summaries: 22 surface overlaps + 65 risk findings = 87 findings
  • after risk summaries: 22 surface overlaps + 7 risk summaries = 29 findings
  • with --experimental-similar: 52 total findings (adds 23 similar candidates)

Verification

  • npm test
  • npm run dogfood
  • npm run validate:corpus
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.risk-summary.final.no-text.json
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.risk-summary-similar.final.no-text.json --experimental-similar
  • npm run pack:check
  • npm run smoke:install
  • npm audit --audit-level=high

@heznpc heznpc merged commit 8579bc8 into main Jun 26, 2026
1 check passed
@heznpc heznpc deleted the feat/risk-summary branch June 26, 2026 21:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant