Skip to content

feat: add spec traceability via t.Specification() and JUnit XML output#345

Open
aaron-zeisler wants to merge 22 commits into
feat/fdv2from
aaronz/spec-tagging
Open

feat: add spec traceability via t.Specification() and JUnit XML output#345
aaron-zeisler wants to merge 22 commits into
feat/fdv2from
aaronz/spec-tagging

Conversation

@aaron-zeisler
Copy link
Copy Markdown

@aaron-zeisler aaron-zeisler commented May 21, 2026

Summary

Add first-class specification traceability to the test harness, inspired by the OpenFeature @Specification (Java) / [Specification] (.NET) pattern adapted for Go.

Framework changes (4 commits)

  • SpecReference type and T.Specification(specID, number, summary) method on ldtest.T
  • TestResult.Specifications field, propagated from T on test completion
  • JUnit XML output: spec references surfaced as <properties> on <testcase> elements
  • YAML schema: TestSpecReference type and specifications field on EvalTestSuite, automatically propagated to t.Specification() by the parameterized test runner

Test annotations (13 commits)

  • All 17 specs annotated with t.Specification() calls (Go) or structured specifications fields (YAML)
  • 5 specs renumbered to match latest sdk-specs main:
    • HOOK: gap-closing (3 reqs)
    • AUTOENVATTR: section shift starting at 1.2.3
    • DATASYSTEM v2: gap at old 1.2.4 closed
    • FLGEA: major renumbering in section 1.6
    • CONTEXT: complete rewrite with new numbering
  • 2 PS annotations retained as comments on helper functions that don't accept *ldtest.T

Files changed

  • framework/ldtest/: 4 files (new spec.go, modified test_scope.go, result.go, junit_test_logger.go)
  • data/testmodel/eval.go: YAML schema extension
  • sdktests/: 16 Go files
  • data/data-files/: 30 YAML files

Test plan

  • go build ./... passes
  • gofmt clean
  • CI lint passes
  • Spot-check JUnit XML output contains <properties> with spec references (verified locally)
  • Verify YAML specifications fields parse correctly (verified locally)

via LD Research 🤖


Note

Medium Risk
Mostly additive test-harness metadata, but it changes TestResult/JUnit XML output shape (adds <testcase><properties>), which could affect downstream CI/report parsers and any code assuming the previous structs.

Overview
Adds first-class specification traceability to the harness via t.Specification(specID, number, summary) and a new SpecReference type, capturing these references on TestResult at completion.

Surfaces recorded spec references in JUnit XML by emitting them as <property name="specification" ...> entries under each <testcase>.

Extends parameterized eval YAML suites with a specifications field (EvalTestSuite.Specifications) and propagates those references into the runtime t.Specification() calls; updates many Go/YAML tests to include these spec annotations and adds an AGENTS.md guide describing the conventions.

Reviewed by Cursor Bugbot for commit f57b4d1. Bugbot is set up for automated code reviews on this repo. Configure here.

@aaron-zeisler aaron-zeisler requested a review from a team as a code owner May 21, 2026 18:30
Comment thread sdktests/common_tests_stream_fdv2.go
aaron-zeisler and others added 15 commits May 21, 2026 11:54
Introduce a SpecReference struct for linking tests to formal
specification requirements, inspired by the OpenFeature
@Specification (Java) / [Specification] (.NET) convention.
Tests call t.Specification(specID, number, summary) to record
which spec requirements they exercise.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add Specifications field to TestResult so spec references recorded
via t.Specification() are available in test results for reporting.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add spec references as <properties> on JUnit <testcase> elements,
making specification coverage visible in CI dashboards and test
reports without additional tooling.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add TestSpecReference type to testmodel and a specifications field
to EvalTestSuite so YAML-driven eval tests can declare spec
references in data. The parameterized test runner propagates them
to t.Specification() automatically.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Two PS comments on helper functions without *ldtest.T are retained
as comments since they cannot call t.Specification().

Co-authored-by: Cursor <cursoragent@cursor.com>
Applies renumbering: 1.2.1.2->1.2.1.1, 1.2.2.2->1.2.2.1,
1.2.2.3->1.2.2.2.

Co-authored-by: Cursor <cursoragent@cursor.com>
Applies renumbering for sections 1.2.2-1.2.7 shifted down by one.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…() calls

DATASYSTEM applies renumbering: 1.2.5-1.2.10 shifted to 1.2.4-1.2.9.

Co-authored-by: Cursor <cursoragent@cursor.com>
…) calls

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…) calls

FLGEA applies renumbering for section 1.6 subsections.

Co-authored-by: Cursor <cursoragent@cursor.com>
…calls

CONTEXT applies complete renumbering from spec rewrite.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aaron-zeisler aaron-zeisler force-pushed the aaronz/spec-tagging branch from 1b1ed3b to 88236a8 Compare May 21, 2026 18:57
@aaron-zeisler aaron-zeisler changed the base branch from aaronz/spec-traceability-comments to feat/fdv2 May 21, 2026 18:57
aaron-zeisler and others added 2 commits May 21, 2026 11:59
…ations fields

Replace # SPEC_ID X.Y.Z: comments with structured specifications
fields in YAML test data files. FLGEA applies renumbering for
section 1.6 subsections.

Co-authored-by: Cursor <cursoragent@cursor.com>
t.LongRunning() may skip the test via panic, making any
t.Specification() calls after it unreachable. Move all spec
references to the top of each test function so they are always
recorded in TestResult regardless of skip state.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aaron-zeisler aaron-zeisler force-pushed the aaronz/spec-tagging branch from 88236a8 to 0313b38 Compare May 21, 2026 19:00
aaron-zeisler and others added 2 commits May 21, 2026 12:04
Co-authored-by: Cursor <cursoragent@cursor.com>
gofmt rejects a blank `//` line immediately before a func declaration.
These were pre-existing on feat/fdv2 but surfaced by the linter.

Co-authored-by: Cursor <cursoragent@cursor.com>
@aaron-zeisler aaron-zeisler changed the title feat: replace spec comments with t.Specification() calls and JUnit output feat: add spec traceability via t.Specification() and JUnit XML output May 21, 2026
Comment thread sdktests/common_tests_stream_fdv2.go Outdated
Comment thread sdktests/common_tests_stream_fdv2.go Outdated
- IgnoresHeartBeat: was DATASYSTEM 1.2.2 (fallback condition config),
  should be FDV2PL 4.3.9 (heartbeat events silently ignored)
- DisconnectsOnGoodbye: was DATASYSTEM 1.2.2, should be FDV2PL 4.3.5
  (goodbye event logged and triggers disconnection)
- InitializeFromTwoPollingInitializers: both 1.1.2 and 1.1.3 had
  identical summaries; corrected to reflect distinct spec requirements
  (1.1.2 = ordering, 1.1.3 = termination/fallthrough)

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread sdktests/server_side_persistence_base.go Outdated
Ensures AI agents (Claude Code, Cursor, Codex) working in this repo
know to use t.Specification() when writing contract tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 55149b3. Configure here.

Comment thread sdktests/common_tests_eval.go
… runner

- Remove t.Specification() from eventuallyRequireDataStoreInit,
  eventuallyValidateFlagData, and neverValidateFlagData since these
  are utility functions called multiple times per test, causing
  duplicate spec entries. Retain as comments instead.
- Move t.Specification() loop before t.RequireCapability() in the
  parameterized eval runner so specs are recorded even when the
  capability check skips the test.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant