chore(compat): refactor compatibility test to use oasdiff - #460
Merged
jpower432 merged 8 commits intoAug 27, 2026
Conversation
We have a CLI that already transforms CUE to OpenAPI. oasdiff is a tool that tests for API diffs. It tests from the producer and consumer side and does not require as many workaround as cue.Subsume. Assisted-by: Claude (Anthropic, Claude Opus 4.8) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> Co-authored-by: Eddie Knight <21176439+eddie-knight@users.noreply.github.com> Signed-off-by: Jennifer Power <jpower@redhat.com>
- Makefile: fail breaking-check on gh query/download errors instead of
going green; skip only when the release list is genuinely empty.
- Makefile: auto-wire --allow .oasdiff-allow when the file exists, and
restore the GEMARA_COMPAT_PRERELEASE knob for prerelease baselines.
- breaking.go: exempt experimental schemas from the gate by mapping
x-status experimental to oasdiff x-stability-level alpha, restoring the
old compat test's non-stable exemption.
- breaking.go/breakingcheck.go: carry the oasdiff path through Change so
multi-schema output is attributable and the allowlist is keyed per
schema ("<id> <path>").
- release.yml: pin the release toolchain via go-version-file: cmd/go.mod.
- Make the integration test deterministic and target a stable schema.
- CONTRIBUTING: document the .oasdiff-allow path and scoped-entry format.
Assisted-by: Claude Code <noreply@anthropic.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
- Disjunctions of string literals now emit OpenAPI enum values (handles nested chains and *"default" markers). - Hidden definitions (#_Name) are omitted from output; refs to them are rewritten to the visible definition they constrain. - Hidden struct fields (_name) no longer leak as properties. - make test now also runs the cmd/ test suite in CI. Signed-off-by: Eddie Knight <knight@linux.com>
…er tests - Error out in the pre-pass when a #_ definition does not unify with a visible definition, instead of silently leaving it in the output where it only surfaced as a confusing leak-check test failure. - Document that ref rewriting intentionally drops the hidden definition's extra constraints, that findBaseIdent ignores attribute-only structs, and that CUE default markers are deliberately not emitted as OpenAPI defaults. - Treat only unquoted _name labels as hidden fields; a quoted "_name" is an exported CUE field. - Assert exact enum values (citing their CUE sources), assert the rewritten $ref structurally instead of via a loose substring match, and add a table test for collectEnumStrings' non-enum fallback. - Run the test suites with -count=1: they read *.cue schemas at runtime, which Go's test cache cannot track. Signed-off-by: Eddie Knight <knight@linux.com>
- Remove the GEMARA_COMPAT_PRERELEASE baseline-selection knob: it was a pre-v1 affordance for when only prereleases existed; past v1 the gate always compares against the latest stable v1.x release. - Expose the schema name (not the internal /_schema/ wrapper path) in breaking-change output and allowlist keys, so entries read as the user-friendly "<check-id> <schema>" (e.g. "request-property-enum-value-removed ControlEvaluation"). Rename Change.Path to Change.Schema accordingly. - Add an end-to-end test exercising --allow: a real detected break is suppressed only by its matching schema-scoped entry. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The v1.5.0 baseline openapi.yaml asset was generated by the earlier, lossy CUE projection. Making the projection faithful (emitting enum constraints CUE always enforced, rewriting hidden strict-variant definitions to their visible base schemas) causes oasdiff to report the gap against the stale baseline as breaking, though the underlying contract is unchanged (0 ERR at the pre-converter-fix commit). Add .oasdiff-allow covering only that transition so make breaking-check passes. Once this lands and the next v1 release publishes a faithful openapi.yaml, the baseline matches the projection and a follow-up PR must delete the file. Also fix the Makefile allowlist path: the -f test runs at the repo root but the flag is consumed inside `cd cmd`, so pass ../.oasdiff-allow. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The integration test previously exercised only a dropped required property. Add a case that narrows an enum on a real converter-generated projection and asserts the gate flags it (exit 1). This guards the seam Eddie flagged: if the converter stops emitting enum constraints, the baseline carries none, the diff sees no enum to narrow, and the gate silently misses the change. Assisted-by: Claude Code <noreply@anthropic.com> Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Contributor
Author
|
@eddie-knight Ready for re-review, but I am going to test the converter changes with the EDITED Verified and the hidden defs work fixed the broken link between Control Evaluations and Assessment Logs on the website schema docs. |
eddie-knight
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR swaps the breaking changes testing approach to use
oasdiff.oasdiffis looking at the main types of breaking changes we need to capture - field adds, removals, adds/loosening existing constraints.Rationale
We have a CLI that already transforms CUE to OpenAPI.
oasdiffis a tool that tests for OpenAPI diffs and breaking changes. It tests from the producer and consumer side and does not require as many workaround ascue.Subsume. It also has clearer error messages.Schema Changes
Schema Changes Made
auditlog.cue) changescapabilitycatalog.cue) changescontrolcatalog.cue) changesenforcementlog.cue) changesevaluationlog.cue) changesguidancecatalog.cue) changesmappingdocument.cue) changespolicy.cue) changesprinciplecatalog.cue) changesriskcatalog.cue) changesthreatcatalog.cue) changesvectorcatalog.cue) changesSchema Change Details
Testing
Related Issues
Refs: #459
Unblocks this and other in-flight PRs
Reviewer Hints
Self-review checklist