test(snapshot): Add structured JSON fixture targets#349
Merged
cameroncooke merged 35 commits intomainfrom Apr 24, 2026
Merged
Conversation
…velope Introduce a new type layer that separates domain results from presentation, enabling tools to produce structured JSON output alongside existing text/JSONL rendering. Migrates simulator-list as the pilot tool to validate the end-to-end flow. - Add ToolDomainResultBase and 26 schema-derived domain result types - Add ProgressEvent, StructuredOutputEnvelope, ToolExecutionContext types - Add PipelineEvent adapter for backward-compatible rendering - Add DefaultToolExecutionContext bridging new and existing execution models - Wire CLI --output json to produce StructuredOutputEnvelope when available - Migrate simulator list tool to new ToolExecutor pattern as proof-of-concept - Replace old STRUCTURED_JSON_OUTPUT_PLAN with comprehensive architecture doc
…ttern Apply the domain result architecture to all non-build tool families so each tool produces a typed domain result and emits progress events, while preserving existing PipelineEvent rendering via the adapter. Migrated tool families: - device: list, app-path, install, launch, stop - macos: app-path, launch, stop - project-discovery: discover, schemes, bundle-id (sim+mac), build-settings - session-management: clear, set, show, use-profile - simulator: boot, open, app-path, install, launch, stop - simulator-management: erase, location, appearance, statusbar - swift-package: list, run, stop Each tool now exposes a createXxxExecutor() returning a ToolExecutor and sets structuredOutput on the handler context for CLI JSON envelope output. Existing text/MCP behavior is preserved through the DefaultToolExecutionContext + PipelineEvent adapter bridge.
…erage, scaffolding, and capture tools Complete Phase 2 of the domain result architecture by migrating all remaining tool families to the ToolExecutor pattern, covering build/test streaming, debugger integration, UI automation, coverage, scaffolding, and video capture. Migrated tool families: - build/build-run/test: simulator, device, macos, swift-package (11 tools) - debugging: attach, detach, continue, breakpoints, lldb, stack, variables (8) - ui-automation: tap, swipe, button, gesture, key, type, screenshot, snapshot (11) - coverage: report, file-coverage (2) - project-scaffolding: iOS, macOS (2) - capture: record-video (1) - swift-package: clean (1) New shared infrastructure: - xcodebuild-domain-results.ts: streaming xcodebuild to domain result mapping - tool-execution-compat.ts: handler-backed execution context helper - ui-automation/shared/domain-result.ts: UI action result helpers - project-scaffolding/domain-result.ts: scaffold result helpers Only intentionally skipped: xcode-ide bridge tools, doctor, clean, manage-workflows, active-processes (no matching domain result types).
…ut schemas Complete Phase 2 by migrating all remaining tool handlers: doctor, clean, manage-workflows, sync-xcode-defaults, and the 5 xcode-ide bridge tools. Adds 6 new structured output schemas and aligns clean and sync-xcode-defaults with their existing JSON fixture expectations. New domain result types and schemas: - doctor-report, workflow-selection (new schemas) - xcode-bridge-status, xcode-bridge-sync, xcode-bridge-tool-list, xcode-bridge-call-result (new schemas) - clean reuses build-result schema (per fixture) - sync-xcode-defaults reuses session-defaults schema (per fixture) Bridge layer updated to carry typed payloads so tools translate real data instead of parsing rendered text. Every tool handler in the codebase now uses the ToolExecutor pattern with structured output. Only 2 internal utilities remain without it (device/build-settings, swift-package/active-processes).
…on streaming Implement the final output routing layer so each CLI mode draws from the correct canonical source: json from domain results, jsonl from streamed ProgressEvents, and text from PipelineEvent-adapted progress. Phase 3 changes: - Add --output jsonl CLI mode streaming ProgressEvents as NDJSON - Add ToolHandlerContext.emitProgress for canonical progress delivery - Add InvokeOptions.onProgress/onStructuredOutput hooks - Wire direct invocation path to forward progress hooks - Upgrade daemon tool.invoke to v3 streaming protocol: progress frames + terminal result frame replacing buffered PipelineEvent[] - Update daemon server, client, and tool-invoker for streaming - Add DefaultToolExecutionContext.progressSink and getResult() - Remove dead cli-json render strategy - Remove unused DaemonToolMessage type - Forward progressSink in compat helpers for stateful tools Daemon protocol bumped to v3 for tool.invoke only; xcode-ide and other unary methods remain unchanged.
Fully remove PipelineEvent as the canonical data model, completing Phase 5 of the domain result output architecture. ProgressEvent is now the sole event contract across rendering, daemon transport, and tool execution. Core changes: - Expand ProgressEvent with rendering-capable variants (header, section, detail-tree, file-ref, next-steps, build-stage, compiler diagnostics, test progress/failure/discovery) - Build progress-native text renderer replacing PipelineEvent-based rendering while preserving byte-identical text output - Migrate xcodebuild parser pipeline to emit ProgressEvent directly - Migrate all tool handlers, test helpers, bridge integrations, and daemon transport off PipelineEvent - Remove RenderSession PipelineEvent dependency Deleted: - src/types/pipeline-events.ts - src/utils/domain-result-adapter.ts (adapter no longer needed) Added: - src/utils/renderers/domain-result-text.ts (structured result to text rendering) All text snapshot fixtures pass unchanged. Zero PipelineEvent references remain in src/.
…y tests Fix text and structured output regressions exposed by running the full snapshot test suite end-to-end after the PipelineEvent removal. No fixture files were modified — all fixes are in tool and renderer code to match the existing hand-crafted fixtures. Text rendering fixes: - Eliminate duplicate header/error emission in tool handlers that were emitting both legacy events and structured result text - Fix spacing, error prefix, and section ordering regressions across device, simulator, project-discovery, scaffolding, and session tools - Restore canonical transcript shape for build/test/clean tools JSON fixture parity infrastructure: - Add json-fixture-parity.snapshot.test.ts validating structured output against all hand-crafted JSON fixtures - Add json-harness.ts for running tools and capturing structured output - Add json-normalize.ts for deterministic comparison of tool output against fixture expectations Validated: 1453 unit tests pass, 523 snapshot tests pass (37 skipped for environment reasons), 141 JSON schema validations pass.
… result only Eliminate dual-path rendering for non-streaming tools. The domain result is now the sole source of text output — no progress events are emitted during execution. This removes the architectural issue where both the live progress stream and renderDomainResultTextItems could produce the same text, causing duplication bugs. Changes: - Simplify DefaultToolExecutionContext.emitResult() to store only - Expand renderDomainResultTextItems with complete rendering for all 20+ non-streaming result kinds - Update shouldRenderStructuredOutput to render structured output for all non-streaming results by default - Remove ctx.emit()/ctx.emitProgress() from all non-streaming tool handlers (~65 tools across all families) - Update architecture doc to reflect the two-category model: non-streaming (domain result only) vs streaming (live progress) - Update unit tests for new rendering contract Streaming tools (build, build-run, test) are unchanged — they still emit progress events for CLI text/jsonl modes.
…treaming Restore durable headers via ctx.emit() for non-streaming tools that need frontmatter in their text output. Implement explicit liveProgressEnabled gating so streaming only activates for CLI text/jsonl. Make xcodebuild tools emit complete live tails (summary, build log path). Remove filterStructuredOutputItems and dual-path coordination logic from cli-text-renderer.
Fix every non-streaming tool snapshot failure caused by Phase 5a migration. Domain result text rendering now matches all CLI and MCP fixtures byte-for-byte. Fixes: - Restore durable ctx.emit(header) for tools needing frontmatter - Fix app-path error summary to use "Query failed." not full message - Fix app-path success to include duration in simulator variant - Add MCP build-and-run intermediate status lines to domain renderer - Fix discover-projs path normalization for workspace root - Fix swift-package run error suppressing extra build log tail - Fix snapshot-ui header title, JSON alignment, tips formatting - Fix snapshot-ui error details section rendering - Fix build settings empty-value formatting preserving "key =" - Update cli-text-renderer to skip structured replay when live header was already emitted All 544 snapshot tests pass (0 failures, 16 skipped). All 1467 unit tests pass.
Remove all it.skip and conditional skip patterns added during the refactor. Fix JSON fixture parity for simulator, device, session, and resource tests. Add deterministic simulator state setup for list snapshot tests. Normalize JSON device state strings. Unit tests: 1467/1467 passed Snapshot tests: 0 skipped (was 16)
Stop the daemon before swift-package list tests to clear stale process records from prior run tests. Also kill any system-level spm processes not tracked by the daemon.
…t isolation Remove ensureNamedSimulatorStates and ensureSimulatorState functions that were added during the refactor. These booted up to 5 real named simulators with no teardown, causing resource exhaustion, Launch Services errors, device test stalls, and fixture mismatches. Restore pre-refactor approach: - Only ensureSimulatorBooted for primary test simulator - createTemporarySimulator for throwaway tests - Environment-dependent list/resource tests marked as it.skip - Suite-level daemon/process cleanup for swift-package isolation
Fix simulator-management open test to expect success (no longer fails with Launch Services error now that simulators aren't exhausted). Update simulator test failure JSON fixture to match real intentional test failures instead of system launch denial that only occurred under simulator resource exhaustion.
…e 17 Pro - Remove all it.skip patterns from snapshot suites - Boot iPhone 17 Pro as primary test simulator (not iPhone 17) - Add normalizer for simulator state markers ([STATUS], SIM_STATE) - Add normalizer for floating-point frame coordinates in UI hierarchy - Add JSON normalizer for simulator state fields - Update list/resource fixtures for single-booted-sim environment - Update device list fixtures with STATUS placeholders - Add test execution rules to CLAUDE.md and AGENTS.md - Add investigation report for snapshot failures Pre-existing failures (verified on pre-refactor base): - snapshot-ui success: accessibility hierarchy content drift - smoke tests: 4 pre-existing failures
- Normalize accessibility hierarchy JSON block to stable placeholder (content varies with simulator state/app foreground) - Normalize simulator state in JSON fixtures (Booted/Shutdown -> <SIM_STATE>) - Update snapshot-ui fixtures with correct tips text and normalized hierarchy - Fix simulator build fixture sim name (iPhone 17, not iPhone 17 Pro) - Boot iPhone 17 Pro as primary sim, pass iPhone 17 as build param name
Pass `-collect-test-diagnostics never` to xcodebuild for all build and test actions. After a failing physical-device `xcodebuild test`, Xcode invokes `devicectl diagnose`, which can prompt for a macOS password on a TTY and wedge automated runs until they time out. Opting out of post-failure diagnostics collection makes the real error surface immediately instead of hanging. Co-Authored-By: Claude <noreply@anthropic.com>
Replace the hand-rolled command construction and error extraction in the clean tool with a call to the shared `executeXcodeBuildCommand` helper. This consolidates xcodebuild invocation behind one path, so clean automatically inherits project/workspace resolution, destination handling, and any shared flags (including the new `-collect-test-diagnostics never`) without duplicating logic. Stderr is captured and filtered for a minimal set of diagnostic noise patterns so failure messages remain useful without dragging in xcodebuild banner lines. Co-Authored-By: Claude <noreply@anthropic.com>
Drop CLI and MCP snapshot command timeouts from 300s back to 120s now that the device diagnostics hang is prevented upstream. Longer timeouts only masked the wedge and slowed the suite. Also normalize the doctor resource's "Xcode running" line so the fixture no longer depends on whether Xcode happens to be open during a run, and add a `test:snapshot:device` script for targeted reruns of the device snapshot file. Co-Authored-By: Claude <noreply@anthropic.com>
Capture the root-cause analysis for the physical-device snapshot hang triggered by `devicectl diagnose` interactive auth, along with two broader investigation logs covering device snapshot regressions and CLI/MCP snapshot parity. Add a pointer in AGENTS.md so future runs that hit the same symptom can jump straight to the RCA. Co-Authored-By: Claude <noreply@anthropic.com>
Retire the ProgressEvent union in favor of DomainFragment types that share the `kind` discriminator with their corresponding final domain result, so streaming partial updates and final results are part of the same model family and no longer require a separate UI event model at the tool layer. Introduce `src/types/domain-fragments.ts` for tool-level fragments (invocation, build/test/run stages, compiler diagnostics, etc.) and `src/types/runtime-status.ts` for infrastructure-level status fragments. Renderers translate fragments to presentation output, keeping tools free of rendering concerns. Add a `raw` render strategy plus `src/rendering/render-items.ts` to stream transcript-kind fragments (process-command, process-line) straight to stderr while still rendering the final domain result on stdout. Extract app lifecycle and app-query domain shapes into dedicated modules (`utils/app-lifecycle-results.ts`, `utils/app-query-results.ts`) and add `utils/transcript-context.ts` for shared transcript hooks. Migrate every MCP tool, daemon path, CLI renderer, and test fixture to the fragment model, rename `getEvents`/`renderEvents` to `getFragments`/`renderFragments`, and drop the now-unused `tool-error-handling.ts`, `tool-event-builders.ts`, and `types/progress-events.ts`. Remove the outdated `DOMAIN_RESULT_OUTPUT_ARCHITECTURE.md` doc and the two smoke tests (`e2e-mcp-doctor`, `e2e-mcp-simulator`) superseded by snapshot coverage.
Update the rebased test suite to match the current fragment and runtime-status APIs. Replace stale progress-event expectations with fragment-based assertions, move infrastructure status helpers onto the runtime-only status type, and repair the small broken test helpers surfaced by typecheck after the rebase. This keeps the branch working on top of main without reintroducing old ProgressEvent assumptions. Co-Authored-By: OpenAI Codex <noreply@openai.com>
cameroncooke
force-pushed
the
cameroncooke/test/structured-json-fixtures
branch
from
91d1ce0 to
70997d3
Compare
commit: |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Fix the daemon-backed JSONL path so streamed fragments are only recorded once in the render session while still being serialized once to NDJSON. Also preserve screenshot dimensions when the base64 fallback keeps the original PNG after optimization fails, and remove the dead imports and helper parameter surfaced in review. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Move the shared xcode bridge status schema fragment into the common structured-output defs file so the status and sync schemas cannot drift independently. Also replace the hardcoded physical device UUID in the repo-maintained agent docs with a portable placeholder for other contributors. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Keep --output json machine-readable when a tool fails to populate structured output. Instead of throwing and printing a plain stderr message, write a parseable JSON error envelope and exit with a non-zero code. Update the CLI test to assert the envelope contract so this failure mode stays stable for automation consumers. Co-Authored-By: OpenAI Codex <noreply@openai.com>
Unify structured tool results around canonical envelope fields, diagnostics, and rendering paths. Update tool manifests, schemas, documentation, and tests so streaming tools use streaming-only context while unary tools return structured results. Co-Authored-By: Codex <noreply@openai.com>
Refresh CLI, MCP, and JSON snapshot fixtures for the standardized output model and diagnostics rendering. Keep fixture churn separate from the source changes for review. Co-Authored-By: Codex <noreply@openai.com>
Render fixture mismatches with a real line diff so consecutive changes are grouped as removals followed by additions. Keep the output plain ASCII so Vitest error messages remain aligned and readable. Co-Authored-By: Codex <noreply@openai.com>
Replace volatile PATH build setting values with a stable placeholder while keeping the build settings entry in the fixture output. Co-Authored-By: Codex <noreply@openai.com>
Update recorded snapshot fixture values that changed during the latest focused runs. Keep these separate from harness and normalisation logic for easier review. Co-Authored-By: Codex <noreply@openai.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a4baa6f. Configure here.
| configuration: Debug | ||
| simulatorName: iPhone 17 Pro | ||
| simulatorId: B38FE93D-578B-454B-BE9A-C6FA0CE5F096 | ||
| simulatorId: A2C64636-37E9-4B68-B872-E7F0A82A5670 |
Contributor
There was a problem hiding this comment.
Example project config pins developer-specific simulator UUID
Low Severity
The example project's sessionDefaults.simulatorId was swapped to a new hardcoded UUID (A2C64636-...), and simulatorName was relocated to the bottom of the file. Simulator UUIDs are machine-specific, so any contributor cloning the example will not have that simulator and the defaults will not resolve. This looks like local developer state leaking into the committed example rather than an intentional example change.
Reviewed by Cursor Bugbot for commit a4baa6f. Configure here.
Collaborator
Author
There was a problem hiding this comment.
Not addressing in this PR. Committing a simulatorId to .xcodebuildmcp/config.yaml is intentional: simulatorName is the stable, canonical identity, and the runtime is designed to reconcile simulatorId in memory from the name on each contributor's machine so the committed ID is effectively a per-developer cache. Writing back to disk is deliberately avoided to prevent config churn when different contributors each have different local simulator UUIDs.
The swap from B38FE93D-... to A2C64636-... here is a lateral move (the prior value was equally machine-local) and is routinely overwritten by setup / session_set_defaults / the bootstrap reconciliation, so it does not regress contributor onboarding.
Separately, there's a real gap in src/utils/simulator-defaults-refresh.ts that I'm tracking outside this PR: the current reconciliation only fires when exactly one of simulatorId / simulatorName is present, so when both are set (as in this example config) a stale simulatorId is used as-is instead of being reconciled in memory from the name. Tracked in #357.
Project domain fragments directly onto the public `--output jsonl`
wire shape instead of wrapping them in `{type:"fragment", fragment}`.
The event discriminator is derived mechanically from the fragment's
`kind` and `fragment` fields (lowercased `<kind>.<fragment>`) and all
remaining fields pass through unchanged, so new fragments surface
automatically without mapping work.
Export infrastructureStatus from src/types/runtime-status.ts and import it in cli-tool-catalog and typed-tool-factory instead of redefining the same helper locally in each module. Prevents drift if RuntimeStatusFragment's shape or level set ever changes.
…t format The output standardization refactor (b6bde24) strips known prefixes (e.g. 'Install app in simulator operation failed:') inside buildInstall/ Launch/StopFailure and renders the header as 'Failed to install/launch/ stop app.' with the raw diagnostic under 'Errors (N):'. Update the stale assertions in install/launch/stop_app_sim and debug_attach_sim tests to match the rendered text, keeping the diagnostic-bullet assertions intact.
buildSchemeList{Result,ErrorResult} previously always stored the input
path under artifacts.workspacePath, so a projectPath input surfaced in
the rendered output as 'Workspace: /.../Foo.xcodeproj'. Union the
scheme-list artifacts shape, its JSON schema, and the domain-result
renderer so projects populate artifacts.projectPath and render with the
'Project' label, while workspaces continue to populate workspacePath.
Adds coverage for both input kinds in list_schemes.test.ts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete the move from event-driven tool output to a domain-result/domain-fragment architecture.
At the end of this PR:
DomainFragments plus a final resultjson/jsonloutput is backed by the same canonical data modelWhat changed
Replace the old event model with domain results/fragments
This removes the old
PipelineEvent/ProgressEvent-style active path and replaces it with:ToolDomainResultDomainFragmentKey outcomes:
Move rendering responsibility downstream
Rendering is now driven from the structured result/fragment model rather than from tool-owned presentation events.
This includes:
Make JSON output a first-class contract
This PR establishes and validates structured output as a stable product surface.
Included:
Make raw/streaming output follow the same model
CLI output modes now route through the same canonical data model.
Included:
Migrate tools onto the final architecture
This includes the remaining tool families and shared helpers so the branch ends in one consistent model rather than mixed patterns.
Covered areas include:
Strengthen snapshot and normalization coverage
This PR also finishes the test contract needed to safely support the refactor.
Included:
Why
The previous architecture mixed together:
That made it harder to:
This PR leaves the system with one canonical model for tool output and pushes presentation decisions to the correct downstream layers.
End state
After this PR:
Validation
Ran successfully:
npm run typechecknpm run lintnpm run testnpm run buildnpm run format:checknpm run test:snapshotSnapshot result:
15/15files passed561/561tests passed