Repository navigation
Conformance fixtures cover a minority of the machine-checkable rules #157
Description
coverage.rules reports, on every run, how many machine-checkable rules have no fixture behind them. That number is currently the majority of them, and it has grown with each batch of rules added by the normativity audit, because minting a rule is cheap and writing a fixture is not.
The live list is the check's own output, so it is not repeated here.
Why it matters more than a coverage percentage
A rule with no fixture is a rule no implementation is held to. Both validators can pass every test and still disagree about it, and there is no run that would say so. OO-LD/oold-js#1 is the worked example: a real divergence between the two implementations, sitting there green, because no case in the corpus has a schema without @context.
So the gap is not tidiness. It is the difference between "the two implementations agree" and "the two implementations are both correct", and only the second is worth having before v1.0.
Shape of the work
Per area rather than per rule: a fixture usually establishes a pattern that the next several rules in the same area reuse. examples/compliance/ already holds the machinery (valid: true/false cases checked against the meta-schema); what the uncovered rules mostly need is the tiers beyond meta-schema validity, which is #158.
Treat the coverage.rules warning as the backlog and work it down; do not add a rule without asking whether a fixture can express it.
Activity
simontaurus commentedon Sep 27, 2026
Numbers for the record, measured against
oold 2.1.0on 1.0.0-rc.5: 28 of 47 machine-checkable rules have a check, so the premise in the title and the body, "a minority", no longer holds.Two of them landed because of this issue.
OOLD-SCH-96a3, the root@contextrequirement rc.4 led with, andOOLD-EXT-c77awere both machine-checkable with nothing behind them; they now haverule.contextandrule.range-formin OO-LD/oold-python#171.What remains is the harder half, and it is the half this issue is really about. The 19 uncovered rules mostly need what #158 describes: a case format that can say "this document must fail, citing this rule".
OOLD-EXT-68fais the worked example, and rc.4's own release notes record why the corpus cannot fail for it.Staying open through 1.0.0 deliberately. The conformance programme is Second Stage work (AP 5), and a 1.0 that claims a conformance suite it does not have would be the worse outcome. The specification's status section now says as much: the gap between the rules an implementation is held to and what any implementation demonstrably does is stated rather than implied.