The problem: our claim verifier (styxx.certify, the OATH implementation) only obligates a number to be grounded when its line contains recognized trigger vocabulary (AUROC, margin, rate, accuracy...). Numbers on lines without those words land in ABSTAIN - honestly reported, but unchecked.
We measured the size of this gap: 0.5227 of full-precision decimals in our own corpus sit on unbound lines (b35-c G2b). That is the largest known hole in the verifier.
Why it is hard (and interesting): naive fixes over-trigger. Our own history is instructive - a blunt trigger add produced 6 false accusations (cycle 23), and a bare value-range guard let an ordinal collide with a correlation boundary (cycle 24). The v0.4 decimal+range guard was the shipped compromise. See the comments in styxx/certify.py around _TRIGGERS and _TRIGGERS_CORR.
What a contribution looks like:
- A proposal for broader obligation (dependency parse? sentence-level classifier? proximity to a receipt-name mention?)
- Run it against the existing mutant battery:
python papers/closed-model-frontier/validate_oath_v0.py
- Report catch rate and false-accusation rate - a fix that raises catches while adding a single false accusation does not ship (that rule is why the verifier is trustworthy)
The rule of this repo: any change to certify.py needs its own preregistration with frozen bars before the battery runs. See PREREG_oath_v062_signed_extraction for the shape.
The problem: our claim verifier (styxx.certify, the OATH implementation) only obligates a number to be grounded when its line contains recognized trigger vocabulary (AUROC, margin, rate, accuracy...). Numbers on lines without those words land in ABSTAIN - honestly reported, but unchecked.
We measured the size of this gap: 0.5227 of full-precision decimals in our own corpus sit on unbound lines (b35-c G2b). That is the largest known hole in the verifier.
Why it is hard (and interesting): naive fixes over-trigger. Our own history is instructive - a blunt trigger add produced 6 false accusations (cycle 23), and a bare value-range guard let an ordinal collide with a correlation boundary (cycle 24). The v0.4 decimal+range guard was the shipped compromise. See the comments in styxx/certify.py around
_TRIGGERSand_TRIGGERS_CORR.What a contribution looks like:
python papers/closed-model-frontier/validate_oath_v0.pyThe rule of this repo: any change to certify.py needs its own preregistration with frozen bars before the battery runs. See PREREG_oath_v062_signed_extraction for the shape.