Skip to content

Count mixed findings toward holdout metrics#23

Merged
heznpc merged 1 commit into
mainfrom
fix/holdout-mixed-split-counts
Jun 26, 2026
Merged

Count mixed findings toward holdout metrics#23
heznpc merged 1 commit into
mainfrom
fix/holdout-mixed-split-counts

Conversation

@heznpc

@heznpc heznpc commented Jun 26, 2026

Copy link
Copy Markdown
Member

Summary

  • preserve the concrete split membership for each validation finding
  • count mixed calibration/holdout findings toward each included split instead of only a synthetic mixed bucket
  • add regression coverage so holdout usefulness cannot be bypassed by mixed risk summaries

Debug evidence

  • before: real corpus holdout risk decisions were absent because mixed risk summaries only counted under mixed
  • after: real corpus holdout decisions include 12 surface-overlap findings and 6 risk-summary findings
  • report load is unchanged: 29 report findings, 8.6 review items per 1,000 lines, 19.4 underlying risk findings per 1,000 lines

Verification

  • npm test
  • npm run validate:corpus
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --out /tmp/rulemeter-real-corpus/result.mixed-split-counts.no-text.json
  • node scripts/validate-corpus.mjs --manifest /tmp/rulemeter-real-corpus/manifest.json --format json --strict --out /tmp/rulemeter-real-corpus/result.mixed-split-counts.strict.json exits 1 as expected
  • npm run dogfood
  • npm run pack:check
  • npm run smoke:install
  • npm audit --audit-level=high

@heznpc heznpc merged commit 5a8b35f into main Jun 26, 2026
1 check passed
@heznpc heznpc deleted the fix/holdout-mixed-split-counts branch June 26, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant