Repository navigation
Holdout run, guards for the gaps it exposed, README with benchmark - #9
Merged
Merged
Conversation
…ark (SUB-01) The holdout run (2555b8d, reported unchanged) and the dev sets showed values in valid outputs that were not on the page as such: a scheme note kept in an electronic address, a country read as address line or city, a VAT ID used as seller identifier, a party name or e-mail address used as buyer reference, and repairs that took the other party's once-printed address. Each now gets voided and asked for instead. The eval scorer derived ground-truth totals only from complete models, so mutations that remove a fact or print a currency symbol were wrongly counted as wrong totals. README rewritten for the jury: what it does, how Nemotron is used, benchmark with holdout, limits, privacy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub renders it natively; colours follow the app's stamps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2555b8dand reported unchanged: clean 8/12 valid, noisy 6/8, totals correct in all 14 valid outputs.docs/benchmark/2026-10-05/.Final dev sets: clean 42/45 valid, noisy 25/30, mutated 27/28 as expected; no valid output with a wrong amount in any set. Typecheck and 90 tests green.
🤖 Generated with Claude Code