Skip to content

Compliance cases declare an expected outcome only for meta-schema validity #158

Description

@simontaurus

A case in examples/compliance/ says valid: true or valid: false, and that expectation is checked against the meta-schema only. The other tiers - generated-instance satisfiability, RDF round-trip, pattern lint, predicate attribution - have no per-case expectation at all. They assert "nothing failed" over the corpus as a whole.

What that costs

  • A rule cannot be pinned. There is no way to write "this document must fail, citing OOLD-RT-d9bd". A fixture can only say the document is well-formed or is not, so most of the catalogue is unexpressible as a test. That is the mechanism behind Conformance fixtures cover a minority of the machine-checkable rules #157.
  • A wrong pass is invisible. "No failures" is satisfied by a checker that never fires. Several checks in both implementations passed for exactly that reason until a fixture forced them to run.
  • Agreement substitutes for correctness. With no declared outcome, the only available signal is that two implementations did the same thing, which is also what happens when both are wrong.

Shape

Give a case an expected outcome per tier, keyed to a rule id where one applies: which check, what status, which rule cited. The catalogue already provides stable ids, and coverage.rules already walks the mapping from check to rule, so the vocabulary exists; what is missing is the case format that uses it.

This is the prerequisite for OO-LD/oold-python#148, which replaces verdict comparison with a per-case comparison, and it is what turns the corpus into something a future port can be held to without anyone writing a pairwise harness for it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions