Skip to content

Require label coverage for strict corpus validation#20

Merged
heznpc merged 1 commit into
mainfrom
feat/strict-label-coverage
Jun 26, 2026
Merged

Require label coverage for strict corpus validation#20
heznpc merged 1 commit into
mainfrom
feat/strict-label-coverage

Conversation

@heznpc

@heznpc heznpc commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • add label coverage metrics to corpus validation output
  • make strict release validation fail when any report findings remain unreviewed
  • warn when duplicate, surface-overlap, or risk-summary usefulness cannot be measured from reviewed labels

Real corpus check

  • corpus: 64 real instruction documents, 51 roots, 13 holdout documents, 3,359 lines
  • current private manifest: 29 report findings, 0 reviewed, 29 unreviewed
  • review item load remains 8.6 per 1,000 lines and risk finding load remains 19.4 per 1,000 lines
  • strict mode now exits 1 until findings are manually labeled and usefulness targets are measurable

Verification

  • npm test
  • npm run validate:corpus
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.strict-label-coverage.no-text.json
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --strict --out /tmp/rulemeter-real-corpus/result.strict-label-coverage.strict.json exits 1 as expected
  • npm run dogfood
  • npm run pack:check
  • npm run smoke:install
  • npm audit --audit-level=high

@heznpc heznpc merged commit 9ad3de4 into main Jun 26, 2026
1 check passed
@heznpc heznpc deleted the feat/strict-label-coverage branch June 26, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant