Skip to content

Add review load validation metric#19

Merged
heznpc merged 1 commit into
mainfrom
feat/review-load-metric
Jun 26, 2026
Merged

Add review load validation metric#19
heznpc merged 1 commit into
mainfrom
feat/review-load-metric

Conversation

@heznpc

@heznpc heznpc commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • add review item load and underlying risk finding load metrics to corpus validation
  • warn when report findings exceed the review-item budget per 1,000 instruction lines
  • document the review-load target alongside surface-overlap and risk-summary usefulness targets

Real corpus check

  • corpus: 64 real instruction documents, 51 roots, 13 holdout documents, 3,359 lines
  • default run: 29 report findings, review item load 8.6 per 1,000 lines, risk finding load 19.4 per 1,000 lines
  • with --experimental-similar: 52 report findings, review item load 15.5 per 1,000 lines, risk finding load 19.4 per 1,000 lines

Verification

  • npm test
  • npm run validate:corpus
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.review-load.no-text.json
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.review-load-similar.no-text.json --experimental-similar
  • npm run dogfood
  • npm run pack:check
  • npm run smoke:install
  • npm audit --audit-level=high

@heznpc heznpc merged commit e86e08c into main Jun 26, 2026
1 check passed
@heznpc heznpc deleted the feat/review-load-metric branch June 26, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant