feat: add spec traceability via t.Specification() and JUnit XML output#345
Open
aaron-zeisler wants to merge 22 commits into
Open
feat: add spec traceability via t.Specification() and JUnit XML output#345aaron-zeisler wants to merge 22 commits into
aaron-zeisler wants to merge 22 commits into
Conversation
Introduce a SpecReference struct for linking tests to formal specification requirements, inspired by the OpenFeature @Specification (Java) / [Specification] (.NET) convention. Tests call t.Specification(specID, number, summary) to record which spec requirements they exercise. Co-authored-by: Cursor <cursoragent@cursor.com>
Add Specifications field to TestResult so spec references recorded via t.Specification() are available in test results for reporting. Co-authored-by: Cursor <cursoragent@cursor.com>
Add spec references as <properties> on JUnit <testcase> elements, making specification coverage visible in CI dashboards and test reports without additional tooling. Co-authored-by: Cursor <cursoragent@cursor.com>
Add TestSpecReference type to testmodel and a specifications field to EvalTestSuite so YAML-driven eval tests can declare spec references in data. The parameterized test runner propagates them to t.Specification() automatically. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Two PS comments on helper functions without *ldtest.T are retained as comments since they cannot call t.Specification(). Co-authored-by: Cursor <cursoragent@cursor.com>
Applies renumbering: 1.2.1.2->1.2.1.1, 1.2.2.2->1.2.2.1, 1.2.2.3->1.2.2.2. Co-authored-by: Cursor <cursoragent@cursor.com>
Applies renumbering for sections 1.2.2-1.2.7 shifted down by one. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…() calls DATASYSTEM applies renumbering: 1.2.5-1.2.10 shifted to 1.2.4-1.2.9. Co-authored-by: Cursor <cursoragent@cursor.com>
…) calls Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…) calls FLGEA applies renumbering for section 1.6 subsections. Co-authored-by: Cursor <cursoragent@cursor.com>
…calls CONTEXT applies complete renumbering from spec rewrite. Co-authored-by: Cursor <cursoragent@cursor.com>
1b1ed3b to
88236a8
Compare
…ations fields Replace # SPEC_ID X.Y.Z: comments with structured specifications fields in YAML test data files. FLGEA applies renumbering for section 1.6 subsections. Co-authored-by: Cursor <cursoragent@cursor.com>
t.LongRunning() may skip the test via panic, making any t.Specification() calls after it unreachable. Move all spec references to the top of each test function so they are always recorded in TestResult regardless of skip state. Co-authored-by: Cursor <cursoragent@cursor.com>
88236a8 to
0313b38
Compare
Co-authored-by: Cursor <cursoragent@cursor.com>
gofmt rejects a blank `//` line immediately before a func declaration. These were pre-existing on feat/fdv2 but surfaced by the linter. Co-authored-by: Cursor <cursoragent@cursor.com>
- IgnoresHeartBeat: was DATASYSTEM 1.2.2 (fallback condition config), should be FDV2PL 4.3.9 (heartbeat events silently ignored) - DisconnectsOnGoodbye: was DATASYSTEM 1.2.2, should be FDV2PL 4.3.5 (goodbye event logged and triggers disconnection) - InitializeFromTwoPollingInitializers: both 1.1.2 and 1.1.3 had identical summaries; corrected to reflect distinct spec requirements (1.1.2 = ordering, 1.1.3 = termination/fallthrough) Co-authored-by: Cursor <cursoragent@cursor.com>
Ensures AI agents (Claude Code, Cursor, Codex) working in this repo know to use t.Specification() when writing contract tests. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 55149b3. Configure here.
… runner - Remove t.Specification() from eventuallyRequireDataStoreInit, eventuallyValidateFlagData, and neverValidateFlagData since these are utility functions called multiple times per test, causing duplicate spec entries. Retain as comments instead. - Move t.Specification() loop before t.RequireCapability() in the parameterized eval runner so specs are recorded even when the capability check skips the test. Co-authored-by: Cursor <cursoragent@cursor.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Add first-class specification traceability to the test harness, inspired by the OpenFeature
@Specification(Java) /[Specification](.NET) pattern adapted for Go.Framework changes (4 commits)
SpecReferencetype andT.Specification(specID, number, summary)method onldtest.TTestResult.Specificationsfield, propagated fromTon test completion<properties>on<testcase>elementsTestSpecReferencetype andspecificationsfield onEvalTestSuite, automatically propagated tot.Specification()by the parameterized test runnerTest annotations (13 commits)
t.Specification()calls (Go) or structuredspecificationsfields (YAML)sdk-specsmain:*ldtest.TFiles changed
framework/ldtest/: 4 files (newspec.go, modifiedtest_scope.go,result.go,junit_test_logger.go)data/testmodel/eval.go: YAML schema extensionsdktests/: 16 Go filesdata/data-files/: 30 YAML filesTest plan
go build ./...passesgofmtclean<properties>with spec references (verified locally)specificationsfields parse correctly (verified locally)via LD Research 🤖
Note
Medium Risk
Mostly additive test-harness metadata, but it changes
TestResult/JUnit XML output shape (adds<testcase><properties>), which could affect downstream CI/report parsers and any code assuming the previous structs.Overview
Adds first-class specification traceability to the harness via
t.Specification(specID, number, summary)and a newSpecReferencetype, capturing these references onTestResultat completion.Surfaces recorded spec references in JUnit XML by emitting them as
<property name="specification" ...>entries under each<testcase>.Extends parameterized eval YAML suites with a
specificationsfield (EvalTestSuite.Specifications) and propagates those references into the runtimet.Specification()calls; updates many Go/YAML tests to include these spec annotations and adds anAGENTS.mdguide describing the conventions.Reviewed by Cursor Bugbot for commit f57b4d1. Bugbot is set up for automated code reviews on this repo. Configure here.