Skip to content

Report stale corpus labels#22

Merged
heznpc merged 1 commit into
mainfrom
feat/stale-label-detection
Jun 26, 2026
Merged

Report stale corpus labels#22
heznpc merged 1 commit into
mainfrom
feat/stale-label-detection

Conversation

@heznpc

@heznpc heznpc commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • report stale corpus labels when manifest fingerprints no longer match current findings
  • expose stale label counts and fingerprint IDs in validation JSON and Markdown
  • make strict validation fail on stale labels so private release evidence cannot silently rely on old reviews

Real corpus check

  • corpus: 64 real instruction documents, 51 roots, 13 holdout documents, 3,359 lines
  • current private manifest: 29 report findings, 0 reviewed, 29 unreviewed, 0 stale labels
  • review item load remains 8.6 per 1,000 lines and risk finding load remains 19.4 per 1,000 lines
  • strict mode exits 1 as expected because usefulness remains unlabeled/unmeasured

Verification

  • npm test
  • npm run validate:corpus
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.stale-label-detection.no-text.json
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --strict --out /tmp/rulemeter-real-corpus/result.stale-label-detection.strict.json exits 1 as expected
  • npm run dogfood
  • npm run pack:check
  • npm run smoke:install
  • npm audit --audit-level=high

@heznpc heznpc merged commit ed0ce55 into main Jun 26, 2026
1 check passed
@heznpc heznpc deleted the feat/stale-label-detection branch June 26, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant