Skip to content

docs(conformance): document the v2.3 report schema (refs TailorAU/tailor-app#5567) - #67

Merged
Tailor-AUS merged 1 commit into
mainfrom
docs/conformance-report-schema
Sep 1, 2026
Merged

Tailor-AUS merged 1 commit into
mainfrom
docs/conformance-report-schema

Conversation

@Tailor-AUS

@Tailor-AUS Tailor-AUS commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Lane Results part 1 of 3 for TailorAU/tailor-app#5567: the v2.3 conformance report format existed only as code. This documents it in the runner README so the two Tailor implementations can publish results docs "in the same schema" faithfully.

What's documented

  • Report schema section in spec/v2.3/conformance/runner/README.md:
    • Top-level field table — counts, results, runner_disclaimer, http_coverage (emitted by the runner at src/index.ts main()), plus spec_version and vector_set_ref (stamped by the jq provenance step in .github/workflows/conformance.yml, after the gate runs — with the rationale for stamping at the CI seam rather than in the runner).
    • Per-result field table — path, id, kind, outcome (discriminated union: pass / fail + reason / skip + reason), and the mirrored top-level verification_mode (cryptographic vs structural, per the Honesty disclosure).
  • Consuming implementations note: same top-level shape, additive keys allowed (e.g. an implementation identifier object), exclusions recorded as outcome.status: "excluded" + reason — and the exact contract of tools/check-conformance-expected-failures.mjs (reads only .results[].id + .results[].outcome.status + .counts, recomputes counts, accepts only pass/fail/skip, rejects skips), so downstream docs can't break the gate.

Verified against reality

The tables were written from the actual emission code, the workflow's jq step, and the real v23-conformance-report artifact from the latest green main run (run 33471758084, vector_set_ref 902b810): top-level keys, per-result key sets, and outcome shapes in the artifact match the code exactly — no deltas found.

Docs-only: nothing but the README changed; no runner code touched.

🤖 Generated with Claude Code


Note

Low Risk
README-only documentation with no runtime, CI, or gate behavior changes.

Overview
Adds a Report schema section to the v2.3 conformance runner README so the --json output and published v23-conformance-report artifact are specified in one place (for TailorAU/tailor-app#5567).

The new docs define top-level fields (counts, results, runner_disclaimer, http_coverage, plus CI-stamped spec_version and vector_set_ref), per-result shape (path, id, kind, outcome, optional verification_mode), and the pass / fail / skip outcome union. It explains that provenance is merged in .github/workflows/conformance.yml after the gate, and why that stays out of the runner.

A Consuming implementations subsection spells out how third parties should publish in the same format (additive keys OK, exclusions as excluded for publication only) versus what check-conformance-expected-failures.mjs actually enforces (id, outcome.status, matching counts, no skips).

The usage example now points --json at this section. Docs-only — no runner or workflow code changes.

Reviewed by Cursor Bugbot for commit 9ca183a. Bugbot is set up for automated code reviews on this repo. Configure here.

…nsuming-implementation contract (refs TailorAU/tailor-app#5567)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Tailor-AUS
Tailor-AUS merged commit 8d0f305 into main Sep 1, 2026
10 checks passed
@cursor

cursor Bot commented Sep 1, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_40889bb8-80dc-4157-b88f-14fc74061aa8)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants