docs(conformance): document the v2.3 report schema (refs TailorAU/tailor-app#5567) - #67
Merged
Merged
Conversation
…nsuming-implementation contract (refs TailorAU/tailor-app#5567) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_40889bb8-80dc-4157-b88f-14fc74061aa8) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lane Results part 1 of 3 for TailorAU/tailor-app#5567: the v2.3 conformance report format existed only as code. This documents it in the runner README so the two Tailor implementations can publish results docs "in the same schema" faithfully.
What's documented
spec/v2.3/conformance/runner/README.md:counts,results,runner_disclaimer,http_coverage(emitted by the runner atsrc/index.tsmain()), plusspec_versionandvector_set_ref(stamped by thejqprovenance step in.github/workflows/conformance.yml, after the gate runs — with the rationale for stamping at the CI seam rather than in the runner).path,id,kind,outcome(discriminated union:pass/fail+reason/skip+reason), and the mirrored top-levelverification_mode(cryptographicvsstructural, per the Honesty disclosure).implementationidentifier object), exclusions recorded asoutcome.status: "excluded"+ reason — and the exact contract oftools/check-conformance-expected-failures.mjs(reads only.results[].id+.results[].outcome.status+.counts, recomputes counts, accepts onlypass/fail/skip, rejects skips), so downstream docs can't break the gate.Verified against reality
The tables were written from the actual emission code, the workflow's
jqstep, and the realv23-conformance-reportartifact from the latest green main run (run 33471758084,vector_set_ref902b810): top-level keys, per-result key sets, and outcome shapes in the artifact match the code exactly — no deltas found.Docs-only: nothing but the README changed; no runner code touched.
🤖 Generated with Claude Code
Note
Low Risk
README-only documentation with no runtime, CI, or gate behavior changes.
Overview
Adds a Report schema section to the v2.3 conformance runner README so the
--jsonoutput and publishedv23-conformance-reportartifact are specified in one place (for TailorAU/tailor-app#5567).The new docs define top-level fields (
counts,results,runner_disclaimer,http_coverage, plus CI-stampedspec_versionandvector_set_ref), per-result shape (path,id,kind,outcome, optionalverification_mode), and thepass/fail/skipoutcome union. It explains that provenance is merged in.github/workflows/conformance.ymlafter the gate, and why that stays out of the runner.A Consuming implementations subsection spells out how third parties should publish in the same format (additive keys OK, exclusions as
excludedfor publication only) versus whatcheck-conformance-expected-failures.mjsactually enforces (id,outcome.status, matchingcounts, no skips).The usage example now points
--jsonat this section. Docs-only — no runner or workflow code changes.Reviewed by Cursor Bugbot for commit 9ca183a. Bugbot is set up for automated code reviews on this repo. Configure here.